# Rows every 10 bps: plain-policy compression

## Method

**Selected: 10 bps rows, integer-bps int16 knots. Holdout loss 0.016353135, SE 0.009505956; data 71,484 bytes. Performance target passes; 12 KB target fails.** This is a compression experiment for the Gaussian + plain policy, credited to Taylor (gtmcknight), not a Solidity submission.

Input is the pinned [pol_mid.json](https://raw.githubusercontent.com/gtmcknight/amm-sim/4812a6382311624e1506aca131ca35c378f930d9/research/solved/pol_mid.json) at commit `4812a6382311624e1506aca131ca35c378f930d9`. Its SHA-256 is `efbb5ee102db4d312f64565d9bf7232b6becbe02b624164f744a402bbbf2983b`. The following dimensions/ranges are my direct JSON measurements (offsets in bps):

| Table | Shape | Minimum | Maximum |
|---|---:|---:|---:|
| A0 | 321 × 13 | -24.262 | 800.000 |
| B0 | 321 × 13 | -24.262 | 800.000 |
| A1 | 321 × 49 | -31.323 | 800.000 |
| B1 | 321 × 49 | -31.323 | 800.000 |
| A2 | 321 × 49 | -31.333 | 800.000 |
| B2 | 321 × 49 | -31.333 | 800.000 |

There are 71,262 policy offsets. All six row axes are own displacement `w = (row − 160) × 5 bps`, −800 through +800 inclusive. A0/B0 columns are rival displacement −30 through +30 bps, step 5. A1/B1/A2/B2 columns are `5 × W2L` bps, with nonuniform spacing: the exact knot list is in the decoder. Its successive gaps in bps are 150, 150, 100, 100, 50, 50, 50, 25, 25, 15, 15, 10, then 5 through the central region, then symmetrically wider gaps. The finest axis is the uniformly spaced 5 bps row axis (the column axis also reaches 5 bps in the center). Only rows are thinned; no columns, phases or sides are discarded. A is ask offset, B bid offset; suffix is phase 0, 1 or 2.

For each tested step `s ∈ {5,10,20}`, retain source row indices `0, s/5, 2s/5, …, 320`, including both endpoints. Round each retained value to the nearest whole bps using Python `round` (ties to even), then store it as signed int16. No scale multiplier, entropy coding, symmetry folding, clipping, extrapolation or additional representation is used. Reconstruct missing rows by linear interpolation between rounded knots; do not round interpolated values. At runtime interpolate in the rival column axis exactly as in the original policy. No sd axis exists in these plain tables.

## Encoding

Payload has no header, padding or metadata. Each value is two bytes, little-endian two's-complement signed int16, in units of 1 bps. Concatenate A0, B0, A1, B1, A2, B2 in that order. Within each table serialize retained rows ascending, and columns ascending. Step and original axis metadata are constants outside the data. For the chosen 10 bps spacing, there are 161 rows:

| Table | Start byte (zero based) | Length bytes | Shape stored |
|---|---:|---:|---|
| A0 | 0 | 4,186 | 161 × 13 |
| B0 | 4,186 | 4,186 | 161 × 13 |
| A1 | 8,372 | 15,778 | 161 × 49 |
| B1 | 24,150 | 15,778 | 161 × 49 |
| A2 | 39,928 | 15,778 | 161 × 49 |
| B2 | 55,706 | 15,778 | 161 × 49 |

End byte is 71,484 (exclusive). Address of element `(i,j)` in a table with `nc` columns is `base + 2*(i*nc+j)`. Signed value is `lo + 256*hi`, subtract 65536 if that value is ≥32768. Original axes and phase select the table. Normalize own displacement to `x = clamp((w+800)/10,0,160)`, set `i=min(floor(x),159)` and `f=x−i`. Locate rival displacement between original column knots with clipping at the first/last; use `j=min(floor(column_position),nc−2)`, `g=column_position−j`. Perform three lerps on the four corners, separately for A and B. These boundary rules also cover the upper endpoint with fraction 1.

## Tradeoff

All following results are local measurements of unchanged Taylor replica/filter code, not externally published scores. Paired loss is `full edge − decoded edge` per seed, positive meaning compression hurts. SE is sample standard deviation of these paired differences divided by √n. Each run uses the replica's default 10,000 steps, randomized configuration, and `soft=1`; neither true-price nor true-arb diagnostics are enabled.

Choose set is **only seeds 60000..61023**, n=1024. These were the only tested settings; results include both interpolation and integer rounding costs:

| Row spacing | Rows kept | Bytes | Choose-set loss | SE |
|---|---:|---:|---:|---:|
| 5 bps | 321 | 142,524 | 0.010785683 | 0.010451818 |
| 10 bps | 161 | 71,484 | 0.013334441 | 0.012074033 |
| 20 bps | 81 | 35,964 | 0.323266845 | 0.014140580 |

Selection rule: choose the smallest payload among settings with choose-set mean loss <0.2. This selects 10 bps; no tested setting meets the byte target. The choice was frozen before scoring compressed tables on the report set. With all columns retained, size is exactly `2 × (1600/s + 1) × (2×13 + 4×49)` bytes. Thus even 20 bps requires 35,964 bytes; meeting 12 KB would require a representation beyond this item's prescribed row thinning.

## Chosen

**10 bps; 71,484 bytes; report-set loss 0.016353135; SE 0.009505956; n=2048.** Report seeds are exactly **10000..12047**, evaluated once with the frozen choice. Full mean edge 532.385686971; decoded mean edge 532.369333837. No tuning used these results. Worst five report seeds are the largest positive paired losses:

| Seed | Full edge | Decoded edge | Full − decoded |
|---|---:|---:|---:|
| 11701 | 659.900967937 | 653.154597997 | 6.746369940 |
| 11699 | 580.140701600 | 578.058994079 | 2.081707520 |
| 11311 | 574.152572421 | 572.722857870 | 1.429714550 |
| 12009 | 542.272189667 | 540.991260156 | 1.280929512 |
| 10359 | 666.107323259 | 664.878970313 | 1.228352947 |

### Reference gate and scoring execution

Before generating any compressed tables I ran unmodified `python3 -I gates.py` from the pinned checkout. All output matched the README to the printed digit:

```text
seed   perfect-info      gaussian+plain    gaussian+qm       lattice+qm
10000  448.261189  443.455598  443.522608  443.663520
10001  576.779199  571.463409  571.623551  571.977293
10002  531.886497  525.168028  525.865859  526.411631
```

Reference sources: [README.md](https://raw.githubusercontent.com/gtmcknight/amm-sim/4812a6382311624e1506aca131ca35c378f930d9/research/solved/README.md), [gates.py](https://raw.githubusercontent.com/gtmcknight/amm-sim/4812a6382311624e1506aca131ca35c378f930d9/research/solved/gates.py), [real.py](https://raw.githubusercontent.com/gtmcknight/amm-sim/4812a6382311624e1506aca131ca35c378f930d9/research/solved/real.py), [evalpol.py](https://raw.githubusercontent.com/gtmcknight/amm-sim/4812a6382311624e1506aca131ca35c378f930d9/research/solved/evalpol.py), [replica2.py](https://raw.githubusercontent.com/gtmcknight/amm-sim/4812a6382311624e1506aca131ca35c378f930d9/research/solved/replica2.py). I followed the README's `real.py <tables> plain:soft=1` syntax. The CLI hardcodes its start seed to 10000 and prints each policy versus HA, not compression loss. To implement the required choose seeds and paired SE, the harness calls the unchanged `real.ev((path, {'soft':1.0}, seeds))`, retaining its per-seed edge outputs. Each `(full,decoded)` pair uses the same seed and default replica configuration. Work ran under plain `python3 -I`, without a venv or package setup, in bounded batches. The three-seed gate is the only pre-selection report-seed diagnostic, as required by gate P.

On the report set I additionally ran the documented CLI unchanged for full and chosen decoded JSON. This is a repeat verification of the frozen setting, not another parameter selection. Its actual stdout was:

```text
plain:soft=1             edge  532.386 retail  564.827 arb -32.441 rv  77686 av  24942  vs HA +2.828 se 0.041  misclass 16.57%  p rms 14.7 bps (95s)
plain:soft=1             edge  532.369 retail  564.725 arb -32.356 rv  77656 av  24914  vs HA +2.812 se 0.041  misclass 16.60%  p rms 14.7 bps (97s)
```

The CLI HA SE values above are not the compression-loss SE. The paired per-seed calculation is the figure in Chosen.

## Lookup cost

A single offset needs four int16 reads (8 logical byte reads), then three lerps. Both ask and bid together require eight int16 reads (16 logical byte reads), six lerps, **6 multiplications and 12 additions/subtractions**, once coordinates and fractions are known (`lerp(a,b,t)=a+(b−a)t`). Additional work: table/phase selection, clamping, one row normalization, floor/min/subtraction; a rival-coordinate division and phase-0 shift, or at most six binary-search comparisons among 49 rival knots plus one gap division. Addressing for each element uses a row stride, an addition and a factor of two; extracting little-endian signed int16 needs byte assembly and sign extension. Physical EVM word reads can differ from logical byte reads. Fee conversion adds the original comparisons and exponentials and is not included in this table-interpolation count; the Gaussian filter is also excluded. This is an operation count, not measured EVM gas.

Direct coarse-knot lookup versus decoded-table `DPPolicy._fees` was checked at 10,000 deterministic random `(w,rival,phase)` points per setting, including outside-grid clipping. Maximum fee discrepancy in bps: 5 bps 0, 10 bps 0, 20 bps 1.1102230246251565e−12. All retained knots and all 71,262 decoded values were checked against the scorer's JSON. For chosen 10 bps, maximum pointwise offset error versus full tables is 40.525 bps (global RMS 1.200683079 bps); low average edge loss is not a uniform error guarantee.

## Decoder

One standalone function, standard library only, 32 lines. Input `data` is the raw payload bytes. It returns all six full-sized policy arrays and metadata in a dict directly serializable to the JSON that `real.py` loads. Optional `template` is the original parsed pol_mid.json: passing it preserves every original key, including the unused solver diagnostics `VD0` and `VD1`; these are not policy offsets and are not encoded. Without a template, the result is a complete policy table set, but omits those two unused diagnostics. Source `g` is retained as provenance, not a claim that decoded tables have the original solved value.

```python
def decode_rows(data, step_bps=10, template=None):
    """Decode raw little-endian int16 row knots into a real.py table dict."""
    import struct
    if step_bps not in (5, 10, 20):
        raise ValueError('step_bps must be 5, 10 or 20')
    stride = step_bps // 5
    nr = 320 // stride + 1
    if len(data) != 2 * nr * (2 * 13 + 4 * 49):
        raise ValueError('wrong payload length')
    out = dict(template) if template is not None else {
        'g': 0.05368206098114493, 'lam': 0.8, 'sigma': 9.45,
        'size': 20.0, 'H_bps': 5.0, 'NW': 321, 'W0': 160,
        'N2': 49, 'C2': 24, 'FB': 6,
        'W2L': [-160., -130., -100., -80., -60., -50., -40., -30.,
                -25., -20., -17., -14., -12., -11., -10., -9., -8.,
                -7., -6., -5., -4., -3., -2., -1., 0., 1., 2., 3.,
                4., 5., 6., 7., 8., 9., 10., 11., 12., 14., 17., 20.,
                25., 30., 40., 50., 60., 80., 100., 130., 160.]}
    off = 0
    for name in ('A0', 'B0', 'A1', 'B1', 'A2', 'B2'):
        nc = 13 if name.endswith('0') else 49
        values = struct.unpack_from('<' + 'h' * (nr * nc), data, off)
        off += 2 * nr * nc
        rows = []
        for i in range(321):
            a = min(i // stride, nr - 2)
            f = (i - a * stride) / stride
            rows.append([values[a * nc + j] +
                         (values[(a + 1) * nc + j] - values[a * nc + j]) * f
                         for j in range(nc)])
        out[name] = rows
    return out
```

For example, after saving the function to `decoder.py` in a working directory:

```sh
python3 -I - <<'PYTHON'
import importlib.util, json
from pathlib import Path
spec = importlib.util.spec_from_file_location('decoder', 'decoder.py')
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
source = json.loads(Path('pol_mid.json').read_text())
tables = m.decode_rows(Path('rows10.bin').read_bytes(), 10, source)
Path('decoded10.json').write_text(json.dumps(tables, separators=(',', ':')))
PYTHON
```

## Data

Chosen payload is **71,484 bytes**, SHA-256 **`263872e09040ca9e63a4ac0f9c0887f54c79c20cc4bbac9d2cd7eaa24a83fcba`**. It exceeds 40 KB, so the permitted SHA-256/size/regeneration form is used instead of inline base64. This exact command regenerates it from the pinned `pol_mid.json`; no network or packages are needed when that input is present:

```sh
python3 -I - pol_mid.json rows10.bin <<'PYTHON'
import hashlib, json, struct, sys
from pathlib import Path
src = Path(sys.argv[1]).read_bytes()
assert hashlib.sha256(src).hexdigest() == 'efbb5ee102db4d312f64565d9bf7232b6becbe02b624164f744a402bbbf2983b'
tables = json.loads(src)
data = b''.join(struct.pack('<h', round(value))
                for name in ('A0','B0','A1','B1','A2','B2')
                for row in tables[name][::2]
                for value in row)
assert len(data) == 71484
assert hashlib.sha256(data).hexdigest() == '263872e09040ca9e63a4ac0f9c0887f54c79c20cc4bbac9d2cd7eaa24a83fcba'
Path(sys.argv[2]).write_bytes(data)
PYTHON
```

For the two alternate settings the identical encoder uses `::1` (5 bps) or `::4` (20 bps). Their payloads have SHA-256 `e8901a773987beb66b77d01d2c28560aae2d3e82d775bb268d2ebf35ba2139f2` and `d896fac93e839dd31df47a37180090e6ba5f9b6bae27a14d1baebafcc374dbc0`, respectively. They were not evaluated on the report set.

### Reproduce paired scoring

Save the following as `paired.py` outside the pinned source checkout. It uses only the standard library and the unchanged pinned reference files; the input decoded JSON comes from the decoder above. Per-seed rows contain full and decoded edge, so the worst seeds and SE can be independently recomputed. No baseline files for the choose seeds are needed.

```python
import json, math, sys
from pathlib import Path
from multiprocessing import Pool

# argv: solved_directory decoded_json first_seed seed_count output_json
SOLVED = Path(sys.argv[1]).resolve()
DECODED = Path(sys.argv[2]).resolve()
sys.path.insert(0, str(SOLVED))
import real


def score(seed):
    full = real.ev((str(SOLVED / 'pol_mid.json'), {'soft': 1.0}, [seed]))[0][1]
    decoded = real.ev((str(DECODED), {'soft': 1.0}, [seed]))[0][1]
    return [seed, full, decoded]


if __name__ == '__main__':
    first, n = int(sys.argv[3]), int(sys.argv[4])
    if n < 2:
        raise ValueError('at least two seeds are needed for SE')
    output = Path(sys.argv[5])
    rows = []
    with Pool(6) as pool:
        for row in pool.imap_unordered(score, range(first, first+n)):
            rows.append(row)
            if len(rows) % 32 == 0:
                output.write_text(json.dumps({'partial_rows': sorted(rows)}))
    rows.sort()
    losses = [a-b for _, a, b in rows]
    mean = sum(losses) / n
    se = math.sqrt(sum((x-mean)**2 for x in losses) / (n-1) / n)
    worst = sorted(rows, key=lambda r: (-(r[1]-r[2]), r[0]))[:5]
    result = {'n': n, 'mean_loss': mean, 'se': se,
              'worst5': [[s, a, b, a-b] for s, a, b in worst], 'rows': rows}
    output.write_text(json.dumps(result))
    print(json.dumps({k:v for k,v in result.items() if k != 'rows'}, indent=2))
```

Run in the directory containing the payload and decoded JSON, with the checkout's `research/solved` directory substituted as `SOLVED`:

```sh
python3 -I paired.py SOLVED decoded10.json 60000 1024 choose10.json
python3 -I paired.py SOLVED decoded10.json 10000 2048 report10.json
```

For reproducing the full tradeoff table, first regenerate/decode each of 5, 10, 20 bps and run only the choose command for each. Apply the selection rule before running the chosen report command. Each command in this experiment finished within eight minutes; the scoring harness used six workers, and source `real.py` CLI used its default fourteen.

## Problems

- The single prescribed representation cannot reach ≤12 KB at any of the three requested spacings. Chosen data alone is 71,484 bytes, larger even than the entire 24,576-byte deployed-code ceiling; it cannot fit inline in one strategy contract. No further compression is silently assumed.
- 20 bps saves space but its choose-set loss is 0.323267, above 0.2. Coarser spacing or column compression would be a separate experiment, outside these three settings.
- Gate P and the replica compression comparison Q are covered. This item builds no Solidity strategy. Gate F (contract callbacks), real-engine paired results versus t03, gas statistics and deployment size measurements are therefore not supplied. No Rust engine seeds were used. These results establish replica compression behavior only; they do not establish a strategy beating t03 or meeting gas limits.
- Integer-bps rounding is part of this representation; 5 bps rows are not lossless. Missing rows are interpolated after rounding, as scored. Keeping source diagnostic arrays in an optional template does not add them to the deployable policy payload.