# svd-lowrank: truncated-SVD compression of pol_mid.json plain tables Source: github.com/gtmcknight/amm-sim @ 4812a6382311624e1506aca131ca35c378f930d9, research/solved (Taylor / gtmcknight). pol_mid.json sha256 efbb5ee102db4d312f64565d9bf7232b6becbe02b624164f744a402bbbf2983b. **Bottom line:** the best setting tried that fits in 12 KB (ranks 4/6/6, 11,571 bytes) loses **0.464 se 0.014** on the report seeds. **It misses the < 0.2 target.** Low rank can't fit the jump in the tables well (see Problems). ## Gate (P) On seeds 10000..12047, the full pol_mid.json with `plain:soft=1` scored **+2.828 vs HA**, from per-seed edges against `calib_out_2048.json` HA rows. The README states +2.828. So the plain/Gaussian version reproduces. gates.py itself was not run. ## Tables (step 1) | name | shape | rows (axis, step) | columns (axis) | min | max | |---|---|---|---|---|---| | A0, B0 | 321x13 | w = (row-160)*5 bps, -800..+800 | rival R: -30..+30 bps in 5 bps steps (FB=6) | -24.262 | 800.0 | | A1, B1 | 321x49 | same | rival at W2L x 5 bps (W2L non-uniform in 5-bps units: 0,±1..±12 step 1, ±14,17,20,25,30,40,50,60,80,100,130,160, i.e. -800..+800 bps) | -31.323 | 800.0 | | A2, B2 | 321x49 | same | same W2L | -31.333 | 800.0 | Values are ask (A) and bid (B) offsets in bps. **B is the exact point mirror of A**: B[i][j] = A[320-i][C-1-j], max difference 0.0. So only A0, A1 and A2 are stored. The tables have a discontinuity: along a row, the offset jumps from about -18 to about +33 where the policy switches to the "displaced" regime. Rows above the center equal w, which means a zero fee. ## Method - Each of A0, A1 and A2 is fitted with a truncated SVD of rank k_t. - The fit only targets what the fee depends on. A cell with fee 0 (A <= w) may decode to any value <= w. This is done with an alternating fill over 60 iterations: refit the SVD, then replace the free cells by min(reconstruction, w). - Factors: U*sqrt(S) and V*sqrt(S). U's columns are equalised to the same range, then U and V are each quantised to int16 with one shared per-table scale. - I also tried a row weighting toward |w| < 60 bps. It made the central error worse, so it is not used. - Scoring uses real.py's own `RealPolicy`/`report()` with `plain:soft=1`. A wrapper (`pair.py`) imports real.py and calls `report()` on a chosen seed range, run with `python3 -I`. **This differs from the brief's command:** real.py hard-codes seeds 10000..10000+NSEED, so the choose set 60000.. can't be passed on its command line. The policy code and the replica are unmodified. - Loss = mean over seeds of (full edge minus decoded edge), on the same seeds. ## Encoding (exact byte layout) All little-endian. Blob = `b"SVD1"` (4 bytes), then three table records in the order A0 (C=13), A1 (C=49), A2 (C=49). Each record is: - `uint8 k` - `uint32 SCALE` - `int16 U[321][k]`, row-major (row i at offset 2*k*i) - `int16 V[C][k]`, row-major Value in bps: `A[i][j] = (Σ_r U[i][r]*V[j][r]) * SCALE / 2^40`. The dot product fits in int64; in Solidity use int256, then `* SCALE >> 40` (signed: divide, or use sar). B tables: `B[i][j] = A[320-i][C-1-j]`. Size of the chosen blob: 4 + (5+2*4*334) + 2*(5+2*6*370) = **11,571 bytes**. ## Tradeoff (choose set, seeds 60000..61023, n=1024) | setting (k for A0,A1,A2; fee-aware fit) | bytes | loss | se | |---|---|---|---| | 3,3,3 | 6,463 | 1.331 | 0.026 | | 5,5,5 | 10,759 | 0.522 | 0.022 | | **4,6,6** | **11,571** | **0.468** | **0.020** | Not scored on seeds, cell error only (fee-space rms over the central rows 100..220): - Plain SVD without the fee-aware fill is slightly worse than with it. At k=5, A1 rms 3.64 vs 3.56. - Row weighting is worse. - k=6 on all three tables needs 12,9xx bytes, which is over 12 KB. On a 64-seed probe (60000..60063), plain rank 5 lost 0.739 se 0.097. ## Chosen - **Setting:** ranks 4/6/6, fee-aware fit, 11,571 bytes. - **Report set:** seeds 10000..12047, n=2048, loss **0.4641**, se **0.0143**. - **Score vs HA:** full +2.828 vs HA, decoded about +2.36. - **Per 512-seed segment:** 0.473, 0.484, 0.444, 0.456. - **Worst 5 seeds (full minus decoded):** 11826: 3.349, 10132: 2.942, 10772: 2.900, 11610: 2.752, 11770: 2.340. ## Lookup cost Bilinear interpolation of a rank-k table equals a dot product of interpolated factors: `u = (1-f)U[i] + fU[i+1]`, `v = (1-g)V[j] + gV[j+1]`, value = `Σ u_r v_r * SCALE >> 40`. For one offset (ask, k=6): - 4k = 24 int16 reads (2 U rows, 2 V rows). These are contiguous: 2k bytes per row, so a row is one 32-byte word read when k <= 16. - About 4k lerp multiplies, plus k multiplies and k-1 adds for the dot, plus 1 scale multiply and shift. That is roughly 30 multiplies and 30 adds. The bid uses the mirrored indices `(320-i-1.., C-1-j..)` with the same cost, so ask+bid costs 48 reads and about 60 multiplies. Phase 0 (k=4) costs 16 reads per offset. Any column search over W2L is extra and the same as for the full tables. ## Decoder (standalone, stdlib + numpy) ```python import struct import numpy as np def decode(blob): """Decode an SVD1 blob into the six plain tables of pol_mid.json (lists of lists, bps). Layout, little-endian: b'SVD1' then, for A0 (C=13), A1 (C=49), A2 (C=49) in that order: uint8 k, uint32 SCALE, int16 U[321][k] row-major, int16 V[C][k] row-major. A[i][j] = (sum_r U[i][r] * V[j][r]) * SCALE / 2**40 (bps; i = row, j = column) B tables are the point mirror of A (exact in pol_mid.json): B[i][j] = A[320-i][C-1-j]. """ assert blob[:4] == b'SVD1' p = 4 out = {} for name, C in (('0', 13), ('1', 49), ('2', 49)): k, sc = struct.unpack_from(' 0: row weights 1 + 20 exp(-(w/rowsd)^2).""" import sys, glob, json, struct sys.path.insert(0, glob.glob('venv/lib/python*/site-packages')[0]) import numpy as np from dec import decode POL = 'amm-sim/research/solved/pol_mid.json' W = (np.arange(321) - 160) * 5.0 def fit(T, k, mode, rowsd): wt = np.ones(321) if rowsd <= 0 else 1.0 + 20.0 * np.exp(-(W / rowsd) ** 2) sw = np.sqrt(wt)[:, None] free = T <= W[:, None] + 1e-9 # fee 0 cells: any value <= w gives the same fee X = T.copy() for it in range(60 if mode == 1 else 1): U, S, Vt = np.linalg.svd(X * sw, full_matrices=False) Uk = U[:, :k] * np.sqrt(S[:k]) / sw; Vk = Vt[:k].T * np.sqrt(S[:k]) R = Uk @ Vk.T if mode == 1: X = np.where(free, np.minimum(R, W[:, None]), T) return Uk, Vk def encode(k, mode, rowsd): d = json.load(open(POL)) out = bytearray(b'SVD1') for name, kk in zip(('A0', 'A1', 'A2'), k if len(k) == 3 else k * 3): T = np.array(d[name], dtype=float) Uk, Vk = fit(T, kk, mode, rowsd) su = np.abs(Uk).max(axis=0) Uk = Uk / su * su.max(); Vk = Vk * su / su.max() # equalise column ranges of U su1 = np.abs(Uk).max() / 32767; sv1 = np.abs(Vk).max() / 32767 qU = np.round(Uk / su1).astype(np.int64) # given quantised U, refit V by least squares on the same target, then quantise qV = np.round(Vk / sv1).astype(np.int64) sc = int(round(su1 * sv1 * 2 ** 40)) assert 0 < sc < 2 ** 32, sc out += struct.pack('