# Score item result Implemented the paired sequential scorer within this item's source paths. No banks generated, foundation/configuration changes, dependency installs, tuning, or engine simulations. Work completed within the 55-minute item cap; every shell command was bounded by `timeout` at 60 seconds or less. Outputs remain untracked; no staging or commits were performed. ## Delivered paths - `score.py`: stdlib offline row scorer, lazy replica/engine/custom adapters, cumulative rounds of 128, exact-seed pairing, sample SE, deterministic worst five, strict worse/optional above-bar stops, full-precision differences, separate source provenance, validation, and clear CLI exits. - `tests/test_score.py`: 35 deterministic unittest tests with contract-shaped fixtures and backend spies. These are NOT real simulator integration tests. - `docs/score.md`: schemas, statistical limitations, source compatibility limits, API, CLI examples for all modes, exits, checkpoint/rerun instructions. - `artifacts/goal/batches/e02-evaluation/items/score/result.md`: this handoff. All four headers credit Taylor (gtmcknight), the full pinned commit, and `research/solved`. Temporary test data and check logs are confined to `scratch/score-tests` and `scratch/score`. No files under scratch are deliverables. ## Behavior and acceptance checks Baseline banks obey `docs/contracts.md`: first `_credit`, backend, supported baseline policy, nonempty provenance, ascending unique integer seeds, finite six-decimal strings, and engine gas/range validation. Candidate row documents allow shuffled order, arbitrary nonempty policy identity, and unrounded edges. All requested seeds must match exactly; no silent intersection. Backend and explicit metric/source metadata must agree. Policy identity may differ and is recorded separately. Because the foundation contract does not define a universal configuration fingerprint, semantic equivalence of market configuration and runtime versions still requires caller review; provenance is not certified by this scorer. Live calls exactly match `ammfast.run_batch(seeds, policy, head_params=None)` and `engine_eval.run_batch(seeds, strategy_path)`. Custom `module:function` adapters retain the chosen mode's signature. Imports and calls occur only for the next requested round. Results are validated before emitting that round; errors prevent subsequent calls. All engine bank seeds are validated before any call, including seeds in later rounds. Empty input emits `{n:0, mean_diff:null, se:null, worst5:[]}` without a live import. Singleton input has null SE and no statistical stop. Partial final rounds are cumulative. Equality at either threshold does not stop. Worst five sort by `(diff, seed)`. The worse condition takes precedence if a negative bar allows both conditions. Gas remains metadata. These stopping rules are screening heuristics, not sequentially calibrated confidence guarantees. ## Checks, counts, timings - Verified 35 pinned reference originals using `reference/materialize.py`'s `originals()` integrity path: 0.108986 seconds. Nothing was extracted or changed. - Initial synthetic scoring workload: 128 observations, 1,000 summaries, 0.373670 seconds. This was an initial sizing check, not a simulation benchmark. - Before implementation, real archived T-minus-HA workload from the verified `taylor/calib_out_2048.json`: 128 paired seeds, 1,000 summaries, 0.489795 seconds; mean -0.986402028402785, sample SE 0.08995544337225576. No simulation was run. - First unittest run: **35 tests, 0 failures, 0 errors, 0.078 seconds** reported by unittest (shell wall time 0.399 seconds). - Traced unittest run: **35 tests, 0 failures, 0 errors, 0.746852 seconds**. Function-body executable line coverage: **186/190 = 97.9%**. Uncovered lines in `score.py`: 27, 28 (invalid float conversion handler), 112 (invalid policy identity in file loader), 215 (defensive custom-source result guard). This is stdlib trace line coverage, not branch coverage or module-entry coverage. - Implemented scorer benchmark against real archived T and six-decimal HA, in memory only: **128 paired seeds x 1,000 evaluations in 0.681927 seconds**, about 1,466 evaluations/second. Each returned `worse`, n=128. This measures scoring overhead, not simulator runs/second/core or replica parity. - CLI subprocess with `python3 -B -S` (site packages disabled): passed using 300 deterministic row pairs, emitted exactly one round at n=128 with mean=-1, SE=0, status=worse; baseline COMBO and different candidate identity appeared separately on stderr. Execution shell wall time: 0.061683 seconds. - `git diff --check -- score.py tests/test_score.py docs/score.md`: exit 0, no output (0.092 seconds). New file bytes additionally checked by reading them back, parsing both Python sources with AST, checking Markdown fences, headers, UTF-8, sizes, and SHA256 (0.057 seconds). Coverage includes analytical mean/SE, zero variance, strict equality at both thresholds (including nonzero SE), n=0/n=1, cumulative partial rounds, tied worst five, shuffled rows, precision below 1e-6, overflow, invalid banks/provenance/ metrics/backends/gas/edges/seeds, missing/extra/duplicate rows, duplicate JSON keys, differing policies, head forwarding, lazy imports, generator adapters, CLI failures and interruption. Replica and engine spies both prove exactly one 128-seed call and zero later-round calls for worse and above-bar stops. An invalid later engine seed blocks every call. No out-of-range engine seed was simulated. ## Failures and limitations All scorer commands and tests passed. The first final-artifact validator failed at ``, line 7: its naive triple-backtick substring count included a quoted Python string inside a code block. This was a validator defect, not malformed Markdown. The check was repaired to count fence lines and rerun successfully. Expected negative tests intentionally exercise validation errors, missing optional backend imports, and interruption; those passed. No real backend integration, Numba/CPython parity, gas measurement, engine wheel, Linux x86_64 deployment, or macOS arm64 deployment was tested. The requested setup reuse was inspected but **not executed**: `setup_fast.sh` contains `HOST_HAVE=$(uname -sm)` and reports the host. The higher-priority operator rule forbids host inspection. Thus the answer to whether host-inspection commands were run is **false**. The pinned setup and all foundation files remain unchanged. Offline scoring needs no installation. Reference verification reused `reference/materialize.py` directly, without extraction or downloading. No credentials, host configuration, environment inspection, or host command output are included in these artifacts. Live provenance records the strategy path and adapter identity, not a verified strategy/code hash or runtime manifest. Users should retain those separately. Trusted custom adapters must honor requested-only execution; the scorer cannot prevent arbitrary internal work or stdout logging within third-party code. A failed or timed-out run can have partial round output without final status. There is no automatic live-state resume; rerun the command or score a retained complete candidate row file offline. ## Source hashes | Path | Bytes | SHA256 | |---|---:|---| | `score.py` | 12278 | `7b9fa5e37695a5d1b048efe91a8b6c45d31bd4245a733261774c871fe9105a43` | | `tests/test_score.py` | 17421 | `9301f8deb1b842a1b981dbe4e4a1b6ba2c57e4009f9dec085e1ad89266e23dcc` | | `docs/score.md` | 10346 | `088147b0ee58da957761bf2dfd84ea12af45fdc820d1ae5cbf354817b1682473` | The result file is checked separately for existence, UTF-8, header, and the 16,000-byte limit; its own digest is deliberately not embedded in itself. ## Exact validation and benchmark commands Read-only interface review: ```sh timeout 20s cat docs/contracts.md setup_fast.sh reference/materialize.py ``` Initial benchmark (before implementation): ```sh timeout 30s python3 -B - <<'PY' import importlib.util, time, statistics spec=importlib.util.spec_from_file_location('materialize','reference/materialize.py') m=importlib.util.module_from_spec(spec); spec.loader.exec_module(m) t=time.perf_counter(); originals=m.originals(); dt=time.perf_counter()-t print('verified reference originals:',len(originals),'seconds:',format(dt,'.6f')) for name in originals: if 'calib' in name: print('calibration source:',name) print('small scoring benchmark: 128 paired observations, 1000 cumulative summary computations') diffs=[(i%17)/1000 for i in range(128)] t=time.perf_counter() for _ in range(1000): statistics.mean(diffs); statistics.stdev(diffs)/len(diffs)**.5; sorted(zip(diffs,range(128)))[:5] print('seconds:',format(time.perf_counter()-t,'.6f')) PY ``` Inspect only the verified calibration fixture's shape: ```sh timeout 20s python3 -B - <<'PY' import importlib.util,json s=importlib.util.spec_from_file_location('m','reference/materialize.py');m=importlib.util.module_from_spec(s);s.loader.exec_module(m) x=json.loads(m.originals()['taylor/calib_out_2048.json']) print(type(x).__name__) print(str(x)[:1800]) PY ``` Real archived-data sizing benchmark, before implementation: ```sh timeout 20s python3 -B - <<'PY' import importlib.util,json,time,statistics s=importlib.util.spec_from_file_location('m','reference/materialize.py');m=importlib.util.module_from_spec(s);s.loader.exec_module(m) x=json.loads(m.originals()['taylor/calib_out_2048.json']) a={r['seed']:r['edge'] for r in x if r['kind']=='HA'} b={r['seed']:r['edge'] for r in x if r['kind']=='T'} seeds=sorted(a.keys() & b.keys())[:128] t=time.perf_counter() for _ in range(1000): d=[b[s]-a[s] for s in seeds];mean=statistics.mean(d);se=statistics.stdev(d)/len(d)**.5;worst=sorted(zip(d,seeds))[:5] print('real archived T-minus-HA scoring: n=',len(d),'iterations=1000 seconds=',format(time.perf_counter()-t,'.6f'),'mean=',mean,'se=',se) PY ``` Tests and whitespace check: ```sh timeout 60s python3 -B -m unittest discover -s tests -p test_score.py -v timeout 20s git diff --check -- score.py tests/test_score.py docs/score.md ``` Traced checks, saved checkpoints, and implemented-scorer benchmark: ```sh timeout 60s python3 -B - <<'PY' import importlib.util, json, time, trace, unittest from pathlib import Path p=Path('scratch/score');p.mkdir(parents=True,exist_ok=True) t=trace.Trace(count=True, trace=False) suite=unittest.defaultTestLoader.discover('tests',pattern='test_score.py') start=time.perf_counter() with (p/'tests.log').open('w') as log: result=t.runfunc(unittest.TextTestRunner(stream=log,verbosity=2).run,suite) elapsed=time.perf_counter()-start source=str(Path('score.py').resolve()) executable=trace._find_executable_linenos(source) counts=t.results().counts covered={line for (filename,line),count in counts.items() if str(Path(filename).absolute())==source and count} # Definitions/imports executed during discovery were not traced: report function-body coverage separately. import ast module=ast.parse(Path('score.py').read_text()) body_lines=set() for node in ast.walk(module): if isinstance(node,(ast.FunctionDef,ast.AsyncFunctionDef)): for statement in node.body: body_lines.update(range(statement.lineno,statement.end_lineno+1)) lines=set(executable)&body_lines report={'tests':result.testsRun,'failures':len(result.failures),'errors':len(result.errors), 'seconds':elapsed,'function_executable_lines':len(lines),'covered':len(lines&covered), 'missing_lines':sorted(lines-covered),'method':'stdlib trace; function-body executable lines only, not branch coverage'} (p/'checks.json').write_text(json.dumps(report,indent=2)+'\n') print(json.dumps(report)) if not result.wasSuccessful(): raise SystemExit(1) spec=importlib.util.spec_from_file_location('materialize','reference/materialize.py') m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m) originals=m.originals(); archived=json.loads(originals['taylor/calib_out_2048.json']) a={r['seed']:r['edge'] for r in archived if r['kind']=='HA'} b={r['seed']:r['edge'] for r in archived if r['kind']=='T'} import score seeds=sorted(a.keys() & b.keys())[:128] baseline=score.Dataset('replica','HA',{s:float(format(a[s],'.6f')) for s in seeds},{'source':'verified archived HA'}) candidate=score.RowCandidate(score.Dataset('replica','T',{s:b[s] for s in seeds},{'source':'verified archived T'})) start=time.perf_counter() for _ in range(1000): status=score.evaluate(baseline,candidate) benchmark={'iterations':1000,'paired_seeds':len(seeds),'seconds':time.perf_counter()-start,'status':status,'source':'archived T minus six-decimal HA; no simulations'} (p/'benchmark.json').write_text(json.dumps(benchmark,indent=2)+'\n') print(json.dumps(benchmark)) PY ``` CLI fixture creation and execution: ```sh timeout 20s python3 -B - <<'PY' import json from pathlib import Path root=Path('scratch/score');root.mkdir(parents=True,exist_ok=True) credit='Taylor (gtmcknight), amm-sim@4812a6382311624e1506aca131ca35c378f930d9 research/solved' for filename,policy,edge in [('cli-bank.json','COMBO','1.000000'),('cli-candidate.json','different-candidate',0.0)]: doc={'_credit':credit,'backend':'replica','policy':policy,'provenance':{'source':'deterministic CLI fixture, not simulation'},'rows':[{'seed':10000+i,'edge':edge} for i in range(300)]} (root/filename).write_text(json.dumps(doc)+'\n') PY timeout 30s python3 -B -S score.py --baseline scratch/score/cli-bank.json --rows scratch/score/cli-candidate.json > scratch/score/cli-rounds.jsonl 2> scratch/score/cli-provenance.jsonl ``` Source readback, syntax, hashes, and CLI result validation: ```sh timeout 20s python3 -B - <<'PY' import ast,hashlib,json from pathlib import Path root=Path('scratch/score') rounds=[json.loads(line) for line in (root/'cli-rounds.jsonl').read_text().splitlines()] meta,status=[json.loads(line) for line in (root/'cli-provenance.jsonl').read_text().splitlines()] assert len(rounds)==1 and rounds[0]['n']==128 and rounds[0]['mean_diff']==-1 and rounds[0]['se']==0 assert status['status']=={'reason':'worse','n':128,'available':300} assert meta['baseline']['policy']=='COMBO' and meta['candidate']['policy']=='different-candidate' print('CLI: stdlib-only (-S), one round, n=128/300, reason=worse; provenance separate: PASS') files={} for name in ['score.py','tests/test_score.py','docs/score.md']: raw=Path(name).read_bytes();content=raw.decode('utf-8') assert 'Taylor (gtmcknight)' in '\n'.join(content.splitlines()[:4]) assert '4812a6382311624e1506aca131ca35c378f930d9' in '\n'.join(content.splitlines()[:4]) assert 'research/solved' in '\n'.join(content.splitlines()[:4]) if name.endswith('.py'): ast.parse(content,filename=name) else: assert content.count('```')%2==0 files[name]={'bytes':len(raw),'sha256':hashlib.sha256(raw).hexdigest()} (root/'files.json').write_text(json.dumps(files,indent=2)+'\n') print(json.dumps(files,indent=2)) PY ``` ## Resume/recheck Re-run the unittest command above. Local checkpoints are `scratch/score/tests.log`, `checks.json`, `benchmark.json`, `files.json`, and `cli-*.json*`. After scratch is removed, rerun the fixture creation command before the CLI smoke. The source and tests require no install. The trace/benchmark command recreates its own directory. Real live runs remain future integration checks using the examples in `docs/score.md`; no unperformed integration result is claimed here.