# setup-repair: installer deadline/watchdog repair, env cleanup, no host inspection

Credit: Taylor (gtmcknight), solved-policy research,
github.com/gtmcknight/amm-sim @ 4812a6382311624e1506aca131ca35c378f930d9 (research/solved).

Item of batch e03-recovery. Starting point: the goal trunk at `57ce0df` (wave 2), whose
installer is the one audited at foundation `4a02cd145745a9d1c45d2ca8d669b650dce619f8`.
Linux x86_64 only was executed; macOS execution remains untested (see Limitations).

## Files changed (all inside the item's paths)

| Path | Change | sha256 |
|---|---|---|
| `setup_fast.sh` | repaired (modified) | `48e150d165ea3b385fb92e0e3a7aefbbe0d97c38c53efa737a587ed62c308d20` |
| `bootstrap/deadline.sh` | new: deadline + owned-child watchdog library and CLI | `2ceb9adf5f1b2b492002dfc371f94dbb3c196ec439a05b914aa8887f7954a8af` |
| `tests/test_setup_fast.py` | modified: no host probes, explicit platform input, exit-3 and D2 tests | `44668751eb1ed768cc606ab2a66a551fc32d2561177c1286e0a1dbd3935184f8` |
| `tests/test_setup_deadline.py` | new: deadline/watchdog/resume suite on fabricated artifacts | `9cd163537af7c1f952fe58b172a064f2bbcb79b850b25fa19cea447f9acc821c` |
| `docs/setup.md` | updated: exit codes, deadline/resume section, audit reproductions, test totals | `923841f81135b99caccd4629d0c3b0fd18ab45c85a110ab1ada9d6a5f97b5f91` |

Not changed, byte for byte: `bootstrap/pins.sh` (sha256
`cc5af39de3626b919659b5bc8ae9b5536363037a8310453a65784c3463127bcb`) and `bootstrap/manifest.tsv`
(sha256 `f9a0c49310aa6f501eac9a760e537af444d21f556239e3f00a9ed6f1d9628dad`), confirmed with
`git diff --quiet` against the wave-2 commit. No source, data, compiler or wheel pin moved; download verification
(size bound, sha256, atomic rename into `<cache>/sha256/<sha>/<name>`, pip `--require-hashes`)
is unchanged. The result file is untracked and not force-added.

## Audit reproductions (before the fix, on the wave-2 installer logic)

- **D1** `--offline --deadline 1` on a warm engine cache exited 0 (the audit measured 1.573 s);
  `--offline --fetch-only --deadline 0` exited 0 (0.609 s). Cause in the code: `check_deadline`
  ran only before downloads and before each install step; the cache-hash loop, the fetch-only
  summary and the final summary never checked; tar, pip, verify_env and the smokes had no bound.
- **D2** `write_env` emitted the three engine exports only in engine mode and nothing in replica
  mode, so a shell that had sourced an engine `env.sh` kept `IMD_AMM_ENGINE`, `IMD_AMM_HARNESS`
  and `SOLCX_BINARY_PATH` after sourcing a replica one.
- **Host probe**: `HOST_HAVE=$(uname -sm)` ran unconditionally on every invocation (exit 3 when it
  disagreed with `--platform`), and the tests used `platform.system()/machine()` to decide what
  to run. Per the brief it was removed before any installer execution in this item.

## Fixes

**D1, one clock for everything.** `bootstrap/deadline.sh` is sourced by the installer:

- `dl_run LABEL CMD...` runs every child (each cache `sha256sum`, each `curl`, `tar`, the
  interpreter probe, `venv`, `pip`, `verify_env.py`, both smokes) in its own process group via
  bash job control, with a watchdog subshell that polls a per-run token file every 0.05 s and,
  when the absolute deadline passes, writes an expiry flag and sends TERM, then KILL after a
  grace period, to that group only. No `ps`, `pgrep`, `/proc`, `setsid` or GNU `timeout`; only
  `kill`, `wait`, `sleep` and `set -m` (bash 3.2 compatible; `EPOCHREALTIME`/`date +%s%N` with
  whole-second fallbacks). The watchdog closes its stdio so it can never hold a caller's pipe.
- `dl_check` also guards clock-free points: start (so `--deadline 0` exits 75 at once), every
  cache check, the fetch-only summary, the install stamp and the final summary. No
  `setup_fast: OK` line, no `timings.log` entry and no stamp after expiry; exit 75 with
  `resume: rerun the same command.`
- Atomic resumable state: `python/<hash>/.ok` after extraction, `.setup_fast/progress` lines
  `<key> wheels` / `<key> engine` written via temp file + rename after each finished step,
  `installed.key` only after verification. A rerun skips finished steps (`install=resumed`),
  `--force` clears them, a step cut off mid-way is redone because it was never marked.
- INT/TERM/HUP to the installer run `dl_abort`: owned group killed and reaped, exit 130.

**D2.** A replica-mode `env.sh` now contains
`unset IMD_AMM_ENGINE IMD_AMM_HARNESS SOLCX_BINARY_PATH` where the engine exports would be.

**No host inspection.** `uname` is gone. `--platform` only selects artifacts. An incompatible
platform surfaces when the extracted interpreter fails to execute (exit 3, "the interpreter
selected by --platform P does not run on this machine ... Nothing was installed"); the unusable
extraction is removed and no venv, env.sh or stamp exists. `--fetch-only` works anywhere. macOS
engine mode is still refused (exit 2), the default cache is still `$HOME/.cache/imd-amm`, no Rust.

**Tests.** `import platform` and the host-derived skips are gone. `IMD_TEST_PLATFORM`
(`linux-x86_64` or `macos-arm64`) is the explicit input naming the runtime that may be executed
in `scratch/`; unset, the integration groups skip. The old host-mismatch test became a fabricated
non-runnable interpreter archive (Mach-O header bytes) expecting exit 3. The deadline suite uses
a fake interpreter tarball whose `python3` answers the installer's probes and stalls once on
request, spawning a grandchild that writes `killed` into a scratch marker on TERM.

## Commands run and what they reported

Benchmarks first (cold fetch-only, warm cache, fresh scratch):

| Command (all `--cache scratch/...`, `--scratch scratch/...`) | Result |
|---|---|
| `--platform linux-x86_64 --fetch-only` (empty cache) | OK, 16 downloads, 135,768,787 bytes, total 6.47 s |
| `--platform macos-arm64 --replica-only --fetch-only` (empty cache) | OK, 4 downloads, 80,347,821 bytes, total 2.74 s |
| `--platform linux-x86_64 --smoke` (warm cache, fresh scratch) | OK install=fresh fetch 0.80 install 6.44 verify 2.20 smoke 2.65 total 12.13 s; verify_env ok, smoke_replica ok reproducible 256 runs 702 runs/s/core, smoke_engine ok seed 9700000 afterSwap_peak_gas 22592 |
| same, `--offline` rerun | OK install=reused total 1.58 s (1.62 s after final watchdog change) |
| same, `--offline --deadline 1` (D1) | exit 75 "during: verification (after 1.022s)" in 1.05 s |
| same, `--offline --fetch-only --deadline 0` (D1) | exit 75 "during: start" in 0.05 s |
| fresh scratch `--offline --deadline 4` | exit 75 "during: wheel installation (after 4.017s)", then rerun OK install=fresh total 6.05 s (extraction reused) |
| `--replica-only --offline` fresh scratch | OK install=fresh total 6.99 s |
| source engine env.sh then replica env.sh | `ENGINE=[] HARNESS=[] SOLCX=[] MODE=replica` |
| `--platform linux-x86_64`, empty cache, one command (cold engine setup) | OK setup=cold downloads=16 install=fresh fetch 5.97 install 4.87 verify 2.27 total 13.14 s |
| `--platform linux-x86_64 --replica-only`, empty cache, one command | OK setup=cold downloads=4 (114,819,511 bytes) install=fresh fetch 2.72 install 4.70 verify 2.04 total 9.48 s |

Suites (final code), from the repository root:

| Command | Result |
|---|---|
| `python3 tests/test_setup_deadline.py -v` | Ran 13 tests, OK, 14 s |
| `IMD_TEST_PLATFORM=linux-x86_64 python3 tests/test_setup_fast.py -v` | Ran 28 tests, OK (skipped=1: MacIntegration), 42 s |
| `python3 tests/test_setup_fast.py` (no platform input) | Ran 28 tests, OK (skipped=8), 8 s |
| `python3 -B -m unittest discover -s tests -p test_gatep.py` (unchanged suite, bounded 300 s) | Ran 24 tests, OK, 14 s |
| `python3 -B reference/materialize.py --out scratch/reference` | 35 verified originals, 0.22 s |
| `python3 -B gatep.py --reference scratch/reference/taylor --cache scratch/cache/gatep` fresh | `gatep: PASS (executed gates.py)`, 4.46 s |
| same, second run | `gatep: PASS (verified cache hit)`, 0.13 s, stdout identical |
| stress: 2000 `dl_run` calls under a Python pipe capture | 28.96 s, no hang (see Failures) |

Integration timings from `scratch/test_setup_fast/*/.setup_fast/timings.log` (warm cache): engine
fresh 6.8 to 8.1 s, engine fresh with smokes 8.6 s, engine reused 1.6 s, replica fresh 7.4 s,
deadline-interrupted install resumed to OK in 6.8 s.

## Failures met while working, and what was done

1. **Watchdog orphan (found by the suite, fixed).** The first helper killed its sleeping watchdog
   with TERM. If TERM arrived between the watchdog's fork of `sleep` and recording its PID, the
   handler failed under `set -u`, the subshell slept out the whole budget and, worse, the orphan
   `sleep` kept the caller's stdout/stderr pipe open: `test_5_replica_only_has_no_engine` hung
   until the harness was killed at 470 s, and a 400-iteration `dl_run` loop reproduced it.
   Replaced by a polling watchdog with closed stdio, a per-run live token and no synchronous
   kill; the 2000-iteration loop then completed under capture and all suites passed.
2. `test_5` asserted `IMD_AMM_ENGINE` absent from the replica env; it is now present in the
   `unset` line. Test changed to assert no `export IMD_AMM_ENGINE` and the `unset` line present.
3. The new exit-3 test first ran with the failing fake `curl` on PATH, which its own file://
   manifest needs (download exit 6). Removed `no_network=True` from that test.

No failure remains in the delivered files; the final totals above are from the final code.

## Coverage: complete and missing

Complete (executed here): zero deadline (warm offline, fetch-only); delayed cached hashing killed
at the deadline; fetch cut off keeping the verified file, then resumed; install interrupted in
probe/pip/verify with descendant cleanup proven by scratch markers, then resumed (`fresh`,
`fresh`, `resumed`, then `reused`); unexpired success; SIGTERM to the installer (130); sequential
env sourcing with fabricated and with real env files; fabricated non-runnable interpreter (exit 3);
macOS engine-mode refusal; SHA corruption and missing-cache offline; real Linux cold/cached/offline
engine and replica setups with imports and both smokes (engine seed 9700000 only); real install cut
off by `--deadline 2` and resumed; macOS artifacts fetched, hash-verified and inspected; GateP fresh
and cache hit.

Missing (not executed here): any execution on macOS arm64 (`MacIntegration` skipped; bash 3.2,
BSD `sleep`/`date` fallbacks of `deadline.sh` untested); a sleep without fractional-second
support; the host-cache default `$HOME/.cache/imd-amm` was not written (every run used scratch
caches, as the brief requires); no engine seed other than 9700000 was run, and only by the smoke.

## Limitations and resume

- The deadline's granularity is one watchdog poll (0.05 s) plus the child's reaction to TERM;
  a child ignoring TERM is killed after a 2 s grace period. A process that escapes its group
  (its own `setsid`) is not chased: no process discovery by design.
- Hashing is a child like any other, so a 60 MB cache check cut off at the deadline is simply
  redone on resume. Resume command is always the same command line.
- README.md is outside this item's paths and still names only `tests/test_setup_fast.py`; the
  deadline suite is documented in `docs/setup.md`.
- Nothing is claimed about other items' concurrent changes to this trunk.