# Merge 02: self-play items (league, curriculum, evolution, gradient, response): 25 results

Goal: put the network in the top three of Paradigm's Dogfight board (https://www.paradigm.xyz/puzzles/dogfight; simulator https://github.com/benedictbrady/dogfight-challenge). The current entry is "IMD seat 798" (https://api.imd.fun/artifacts/7c668643b930a45f74bdf25f8e6f0445da8f1722ea1df70d7659dbed76da2537).

**Where these numbers come from.** Every number below is copied from the item's own `result.json` under `.imd/reads/artifacts/item-<key>`. All 25 files were present and matched the SHA-256 the brief gave for them. None of the item results contain URLs. Their records are each seat's own local simulator measurements, not figures quoted from the web. They are proxies: none of them measures the hidden board entries.

**What this merge checked itself:**

- **Current entry.** I fetched it again from the URL above. Its decoded ONNX is 18,930 bytes with SHA-256 `f3963181497e39089eef5862284d575d5bfc20e6e41b6377b95ee76c436e25b4`.
- **Model sizes and hashes.** I decoded every item's `entry`. Every `entry_bytes` value matches the decoded size. Seven items put the hash of the base64 text in `entry_sha256` instead of the hash of the decoded file, and their values do match that text hash: curriculum-1, evolution-1, evolution-4, gradient-4, league-1, league-4 and response-5. The base64-text hash of the current entry is `623acacc…`. That explains the "disagreement" gradient-5 reports, and it is not a defect in the model.
- **ONNX operators.** I read every model's operators from the protobuf. They match the item's `ops` list in every case except two:
  - curriculum-3 lists no operators.
  - evolution-3 lists `Add, Clip, Gather, MatMul, Mul, Split, Tanh`.

  Both of those files contain the current entry, whose real operators are the 18 listed in the table.
- **Current model returned unchanged.** These four returned the current model byte for byte (decoded SHA-256 `f3963181…`): **league-1, league-4, curriculum-3, evolution-3.**
- **Blocked, no model.** **gradient-3** and **response-4** were blocked and returned no model.

**Abbreviations.** "Rule ops" means the current entry's 18 operators: Abs, Add, Atan, Cast, Concat, Cos, Div, Gather, Greater, Max, Min, Mul, Neg, Sin, Split, Sqrt, Sub, Where. "Rule ops + Gemm, Relu" adds those two. Records are scores per match by the board formula (kill 1, HP win 0.75, draw 0.5). "Cur W/L/D" is wins, losses and draws against the current entry over 1,000 matches, half from each side. "Expl." is the item's score against its own exploiter, from its own side, over 500 matches. "Starter" is over 3,000 matches with random spawns. "Built-ins" is over 800. "Harness" is the item's own current_vs_starter.

## Table

| Key | Model | Harness | Starter (3000) | Built-ins (800) | Current (1000): score, W/L/D | Expl. (500) | Bytes | ONNX ops |
|---|---|---|---|---|---|---|---|---|
| sp-league-1 | **Current model unchanged** (rule kept after 6 coordinate-search exploiters all lost) | 0.9873333333333333 | 0.9873333333333333 | 1 | 0.5, 400/400/200 (mirror) | 0.5 (exploiter = rule clone) | 18930 | Rule ops |
| sp-league-2 | Rule with re-tuned constants, picked by minimax over 258-opponent league | 0.9936666666666667 | 1 | 0.99125 | 0.668, 640/304/56 | 0.129 | 18929 | Rule ops |
| sp-league-3 | Speed-conditioned pair of evolved rules in one graph (league_polish) | 0.989 | 1 | 0.89625 | 0.711, 706/284/10 | 0.1355 | 21510 | Rule ops |
| sp-league-4 | **Current model unchanged** (best worst case in a small, one-core league) | 0.9906666666666667 | 0.9906666666666667 | 1 | 0.5, 409/409/182 (mirror) | 1 | 18930 | Rule ops |
| sp-league-5 | Rule graph, 65 constants re-tuned by 3 league rounds (ES) | 0.9895 | 0.99516666 | 1 | 0.7065, 639/226/135 | 0.4005 | 18930 | Rule ops |
| sp-curriculum-1 | 224-32-3 ReLU MLP, ES from scratch; built-ins stage only | 0.993 | 0.238 | 0.691875 | 0.001, 1/999/0 | 0.5 (200 s exploiter, not 10 min) | 29421 | Gemm, Relu |
| sp-curriculum-2 | Two-regime rule + 32-ReLU random-feature selector, evolved | 0.9885 | 0.9755 | 1 | 0.917, 906/72/22 | 0.144 | 23153 | Rule ops + Gemm, Relu |
| sp-curriculum-3 | **Blocked; current model returned unchanged** (OpenSSL headers / pkg-config not found) | 0.9933333333333333 (seat's figure, not measured here) | null | null | null, 0/0/0 | null | 18930 | none listed (file is the current entry) |
| sp-curriculum-4 | Two-expert selector: 8-wide ReLU picks between two tuned rules; PPO | 0.993 | 0.9928333333333333 | 1 | 0.8655, 838/107/55 | 0.065 | 36463 | Rule ops + Gemm, Relu |
| sp-curriculum-5 | 224-64-48-3 fp16 MLP, DAgger distillation of the rule | 0.9906666666666667 | 0.9484166666666667 | 0.935 | 0.06, 53/933/14 | 0.02949999999999997 | 35893 | Gemm, Relu, Cast |
| sp-evolution-1 | 56-16-3 ReLU MLP, ES from scratch with growing pool | 0.9906666666666667 | 0.9926666666666667 | 0.9765625 | 0, 0/1000/0 | 0 | 4592 | Gather, Gemm, Relu |
| sp-evolution-2 | 224-32-3 ReLU MLP, Gaussian weight evolution from scratch | 0.995 | 0.6841666666666667 | 0.72875 | 0.1625, 0/675/325 | 0.002 | 29447 | Gemm, Relu |
| sp-evolution-3 | **Blocked; current model returned unchanged** (ort-sys link: glibc symbols missing) | 0.9933333333333333 | 0.9933333333333333* | 1* | 0.5, 0/0/1000* | 0.9875* | 18930 | lists Add, Clip, Gather, MatMul, Mul, Split, Tanh (does not match the file) |
| sp-evolution-4 | 25-feature 16-ReLU MLP experts + clipped affine gate, evolved | 0.9903333333333333 | 0.8885 | 0.94125 | 0.001, 1/999/0 | 0.228 | 7584 | Abs, Add, Atan, Clip, Concat, Gather, Gemm, Mul, Relu, Sub |
| sp-evolution-5 | 56-112-32-3 ReLU MLP, ES from scratch, league + gun-target curriculum | 0.9891666666666666 | 0.892 | 0.441875 | 0.197, 0/606/394 | 0.098 | 41168 | Gather, Gemm, Relu |
| sp-gradient-1 | 224-32-32-3 MLP, behaviour clone then about 8 min PPO | 0.993 | 0.80025 | 0.900625 | 0.017, 14/980/6 | 0.488 (150 s exploiter) | 33785 | Clip, Gemm, Relu |
| sp-gradient-2 | Rule + 56-48-24-3 PPO actor that adjusts 3 rule settings (weight-averaged) | 0.9936666666666667 | 0.998 | 1 | 0.713, 622/196/182 | 0.43 | 36916 | Rule ops + Gemm, Relu |
| sp-gradient-3 | **Blocked, no model** (linker `cc` not found) | null | null (0 matches) | null (0) | null, 0/0/0 (0 matches) | null (0) | 0 | none |
| sp-gradient-4 | 112-64-32-3 MLP, DAgger then about 20 min PPO | 0.991 | 0.8895 | 0.9765625 | 0.3265, 264/611/125 | 0.296 | 38106 | Slice, Gemm, Relu |
| sp-gradient-5 | Rule + 56-64-24-3 PPO residual clipped to [-2,2] | 0.9916666666666667 | 0.9805 | 1 | 0.794, 757/169/74 | 0.401 | 40926 | Rule ops + Clip, Gemm, Relu |
| sp-response-1 | 224-32-32-3 MLP, chain of ES best responses from a rule clone | 0.991 | 0.594 | 0.119375 | 0, 0/1000/0 | 0.01 (240 s exploiter) | 33747 | Gemm, Relu |
| sp-response-2 | Rule, 33 constants, 6-link evolutionary best-response chain + mixture | 0.9888333333333333 | 0.9861666666666666 | 0.995 | 0.5785, 498/341/161 | 0.395 | 18930 | Rule ops |
| sp-response-3 | Rule, 33 constants, 5-link CMA-ES chain + joint stage | 0.9886666666666667 | 0.9821666666666666 | 1 | 0.6185, 550/313/137 | 0.058 | 18930 | Rule ops |
| sp-response-4 | **Blocked, no model** (linker: xcrun / CommandLineTools missing) | null | null (0) | null (0) | null, 0/0/0 (0) | null (0) | 0 | none |
| sp-response-5 | Rule, 33 constants + 52-96-3 residual MLP, 7-link chain + mixed stage + repair | 0.992 | 0.999 | 0.99875 | 0.6625, 590/265/145 | 0.147 | 41862 | Rule ops + Gemm, Relu |

\* evolution-3 says it did not build a working binary, so it ran no matches. Its records are not measurements from that seat:

- Its own explanation is that the exploiter figure was "carried forward" from "the public run".
- Its 1000-draw mirror conflicts with the mirrors league-1 and league-4 actually measured (400/400/200 and 409/409/182).
- Its 0.9875 exploiter figure conflicts with the brief, which says the rule "lost 27 percent to a neural opponent trained in 24 seconds".
- Its operator list is wrong for its own file.

Treat every evolution-3 record as unverified.

## What each item found and what it says to do next

**sp-league-1.** Returned the current model unchanged. Six coordinate-search rule exploiters (lead and firing thresholds) all lost to the rule, scoring 0.30 to 0.48 over 100 matches, so the "exploiter" is the rule's own clone and its 0.5 is a mirror. Next: train genuinely different small neural self-play policies against the rule, and keep a challenger only if it improves the fresh-seed worst case of the league.

**sp-league-2.** A rule with re-tuned constants, picked by empirical minimax over 258 retained opponents. It beats the current entry 0.668, but its 10-minute exploiter holds it to 0.129, while the current entry scored 0.81 against that same exploiter on the same 500 cases. Next: add that exploiter to the league, and test an observation-based switch between the current entry and this candidate to break the matchup cycle. Cross-play the other seats before any upload.

**sp-league-3.** A pair of evolved rules chosen by speed, picked by maximin over 88 opponents. It scores 1 against Starter and 0.711 against the current entry, but the 10-minute oracle holds it to 0.1355. The item calls this a "severe weakness" and says not to replace the live entry. Next: add the oracle to the league, inspect its winning replays, and try a wider 224→64→32 fp16 student. *This item cites one source that did not hold up when its machine read it again. The source list was not among my inputs, so I cannot tell which claim it backed. Its local records above are its own measurements and do not depend on it.*

**sp-league-4.** Returned the current model unchanged. On about one CPU core (5 to 12 matches a second), its exploiters got no signal: the residual-corrected rule R′ lost 0.225 to the rule, and the 10-minute Starter-plus-residual exploiter never scored a kill or an HP win. Next: don't run a league on one core. Warm-start exploiters by imitating the rule, shape rewards on hits and misses, and judge rule variants on the mirror first.

**sp-league-5.** The rule graph with all 65 constants re-tuned over three league rounds. It beats the current entry 0.7065, but a 480 s head-only exploiter holds it to 0.4005, while the current entry scored 0.879 against that exploiter. The exported file was **not** run in ONNX Runtime; only `onnx.checker` passed. The item says to upload only if the operator accepts that exploitability. Next: train longer, full-network exploiters against both models, tune only against held-out exploiters, then distil.

**sp-curriculum-1.** An MLP trained from scratch by ES that never passed the first gate (95% against the built-ins) on one effective core. It loses 999 of 1,000 to the current entry, and its exploiter ran 200 s instead of 10 minutes. Next: give it more cores, port the opponents to Rust, start from Starter or a clone of the rule, and add a kill bonus, because the policy learned to win on HP rather than by kills.

**sp-curriculum-2.** The strongest head-to-head result in this group: 0.917 against the current entry (906/72/22), with 0.9755 against Starter. But its 10-minute rule-family adversary holds it to 0.144, and its curriculum stopped at the current-entry gate (95% not met). Next: don't replace the entry on this evidence. Use it as a hard league opponent, and train a full-action policy or a neural residual that keeps passing the earlier-stage tests.

**sp-curriculum-4.** A selector that picks between two tuned rules. It scores 0.8655 against the current entry and 0.9928333333333333 against Starter, but its exploiter holds it to 0.065. Two findings stand out:

- Matchups are not transitive: an earlier checkpoint scored 0.058 against its own challenger, while the current entry beat that same challenger 0.64185 over 10,000 matches.
- Adding noise only hurt it.

Next: cross-play it and its reproducible challenger (constants given in the item) with other seats. *This item cites one source that did not hold up when read again. The source list was not among my inputs. Its records are local measurements.*

**sp-curriculum-5.** A DAgger distillation of the rule into a 64-48 fp16 MLP. It plateaued at 0.01 to 0.14 against the rule (0.06 over 1,000), and a 10-minute ES exploiter beats it 0.9705. Next: don't upload it. Keep the rule as the base and train a residual or an RL fine-tune on top of it, or a larger net weighted toward shoot decisions.

**sp-evolution-1.** A 963-weight MLP evolved from scratch that matches the rule against Starter (0.9926666666666667) but loses 1,000 of 1,000 to it. An exploiter started from its own weights beats it 100% in under 6 minutes. Next: put the rule and its exploiters in the pool from the start, use all 224 inputs and more width, keep every exploiter in the pool (double oracle), and distil the rule first.

**sp-evolution-2.** Raw-weight evolution from scratch kept specialising to weak pool members: 0.6841666666666667 against Starter, 0.1625 against the current entry with no wins, and 0.002 against its exploiter. Next: use it only as a varied neural opponent. Try distillation from the rule or a lower-dimensional feature parameterisation before self-play fine-tuning.

**sp-evolution-3.** Blocked at the ONNX Runtime link (ort-sys needs newer glibc symbols) and returned the current model unchanged. Its records are not its own measurements (see the note under the table). Next: rerun with a compatible ONNX Runtime. Note that other seats in this group got around the link problem by loading the pip onnxruntime library dynamically or by using zig as the linker (league-4, response-1, gradient-1).

**sp-evolution-4.** Evolved small MLP experts plus a gate. It scores 0.8885 against Starter, wins 1 of 1,000 against the current entry, and scores 0.228 against its exploiter. Next: keep the current upload. Put accurate shooters and the current entry into the early curriculum, and require paired fresh-spawn wins against the current model and a new exploiter before considering an upload.

**sp-evolution-5.** A 10,099-weight MLP evolved from scratch with no win against the current entry (0/606/394). A replay audit showed its Starter wins came from Starter crashing into the ground, not from gun hits, and a steady target showed a heading bias (0.5 against two of four stable targets). Next: test observations canonicalised by reflection, count gun hits as well as eliminations, and keep the live entry. *This item cites two sources that did not hold up when read again. The source list was not among my inputs. Its records are local measurements.*

**sp-gradient-1.** A behaviour clone followed by only about 8 minutes of PPO. It scores 0.80025 against Starter and 0.017 against the current entry, and its exploiter ran only 150 s. Next, as practical advice:

- Build with zig cc as the linker.
- Run ONNX Runtime and torch with one thread each; the item says that fixed a 10x slowdown.
- Put an MLP clone of the rule in the pool, since the rule ONNX costs about 0.15 s per match.
- Use a wider fp16 net and at least 60 minutes of PPO.

**sp-gradient-2.** The rule plus a PPO actor that adjusts three rule settings (speed, lead factor, firing tolerance). It scores 0.998 against Starter, 1 against the built-ins and 0.713 against the current entry, and it has the best exploiter score among the candidates that beat the entry (0.43, from the strongest of four 600 s searches). Next: cross-play this exact frozen model against the other seats on paired seeds, and keep its exploiter as a league opponent.

**sp-gradient-3.** Blocked at `cargo build` (linker `cc` not found). No model, no measurements. Next: retry on a machine with a C linker and OpenSSL headers.

**sp-gradient-4.** DAgger then about 20 minutes of PPO on a 9.4k-parameter MLP. It scores 0.3265 against the current entry (still rising when time ran out) and 0.296 against its exploiter. Next: give PPO hours of compute and more width, keep the rule at 3/8 of the pool, leave Starter out of the training pool, and judge checkpoints on thousands of matches.

**sp-gradient-5.** The rule plus a PPO residual. It scores 0.794 against the current entry (757/169/74) and 0.9805 against Starter, but its 10-minute PPO response holds it to 0.401, and that response also beats the current entry (0.6355). Next: train a league with both the rule and this response while recovering robustness against Starter, and cross-play before any upload.

**sp-response-1.** A best-response chain started from a 32-wide clone of the rule. It drifted into MLPs that only beat each other: 0 against the current entry, 0.594 against Starter. Next: keep the rule, or a much better clone, as an anchor in every link, require each link to stay at 0.95 or better against Starter and the built-ins, and port the rule to Rust for speed (ORT ran about 77 matches a second).

**sp-response-2.** A 6-link chain over the rule's constants plus a mixture stage. It scores 0.5785 against the current entry, 0.9861666666666666 against Starter and 0.395 against its exploiter. Small selection sets that were reused overfit badly. Next: keep it as a league opponent, add full-network neural responses, and select on larger rotating selection sets.

**sp-response-3.** A 5-link CMA-ES chain over the rule's constants. It scores 0.6185 against the current entry, but the neural output-head exploiter holds it to 0.058. The item says not to replace the entry. Next: add the stored rule exploiter and the neural head weights (given in the item) to a larger league, train full-network responses, and distil. *This item cites one source that did not hold up when read again. The source list was not among my inputs. Its records are local measurements.*

**sp-response-4.** Blocked at `cargo build` (xcrun / CommandLineTools missing; the machine looks like macOS). No model, no measurements. Next: retry on Linux.

**sp-response-5.** A 7-link chain over the rule's constants plus a 96-unit residual, followed by a repair stage. It scores 0.999 against Starter, 0.99875 against the built-ins, 0.6625 against the current entry and 0.147 against its exploiter. In the chain, later links lost ground against the public entry (link 1 scored 0.901, link 6 scored 0.158). Its large final records used the native graph, which matched ONNX on 9,024 decisions. Next: continue adaptive double-oracle training with full-network opponents before promoting it.

## Across the group

1. **Nothing here is clearly stronger than the current entry.** Ten candidates beat the current entry head to head, from curriculum-2 at 0.917 down to response-2 at 0.5785, and none of them regressed against Starter below 0.9755. But every one of them scores below 0.5 against a 10-minute exploiter trained on it (0.065 to 0.43). The direct comparisons cut the other way: the current entry held 0.81 against league-2's exploiter and 0.879 against league-5's. The brief warns that the board keeps the latest upload, so on these records alone no candidate meets the "clearly stronger" bar. The items themselves say so: curriculum-2, curriculum-4, league-3, response-3 and evolution-4 advise against replacing the entry, and league-5 and gradient-2 defer to cross-play.
2. **Best candidates for the cross-play round**, by score against the current entry, with exploiter score in brackets:
   - curriculum-2: 0.917 (0.144)
   - curriculum-4: 0.8655 (0.065)
   - gradient-5: 0.794 (0.401)
   - gradient-2: 0.713 (0.43)
   - league-3: 0.711 (0.1355)
   - league-5: 0.7065 (0.4005)

   gradient-2, gradient-5 and league-5 have the best mix of head-to-head strength and resistance to exploitation. league-5's exported file has not been run in ONNX Runtime.
3. **Neural nets trained from scratch or by distillation all failed against the rule** (curriculum-1, curriculum-5, evolution-1, evolution-2, evolution-5, gradient-1, gradient-4, response-1): 0 to 0.3265 against the current entry. Every gain came from keeping the rule's geometry and tuning or adding to it.
4. **Matchups are not transitive** (curriculum-4, league-2, league-5, gradient-5). Exploiters from other seats, and the stored exploiter parameters several items include (league-2, curriculum-4, response-3), should go into the league for the next round.
5. **Harness agreement.** Each measured current_vs_starter falls between 0.9873333333333333 and 0.995, against the seat's 0.9933333333333333; the items used different seed ranges. Four items blocked or did not measure: curriculum-3, evolution-3, gradient-3 and response-4.