Retiring our own best advice.

The uncomfortable table

After classes 02 and 04, nano-plaid's profile on full SciFact looked like this — class 03's scoreboard reduced to its two extremes: best quality in one column, best speed in another, never both:

The 1-bit kernel was fast because it never decompressed; the 4-bit scheme was accurate because it kept more information — and paid for it by decompressing 512 bytes of floats per token so a float engine could multiply them. We even wrote the justification into the README: "BLAS is already the fast path a hand kernel can't beat." True — and, it turns out, beside the point. The rest of this class removes that row's red number without touching its NDCG column.

Anatomy of a residual token

A residual-4 token is not a mystery blob. It is exactly two things — a pointer and corrections (dim 128 shown; our worked example uses dim 8 to fit on screen):

And that ×7.5 blow-up is charged per token, per query, at search time — the price of admission to GEMM. Chapter 06 of class 02 told you why that hurts: GEMM's whole power is FLOPs per byte, and decompression multiplies the bytes:

Split the dot product

Class 04 split the dot over set and unset bits. Here the split follows the token's own anatomy — the dot product distributes over centroid + residual:

Now stare at each term until it stops looking like work:

■ The centroid term is already computed. Stage 1 of the search scored every query token against every centroid — one small GEMM, [nq, K] — to pick which clusters to probe. The rescore gets that matrix for free; per (query row, token) this term is one table lookup.

■ The residual term never needs floats. Quantize the query to int8 (class 04's move, unchanged) and the 16 weights to int8 (a new, 16-entry quantization — near-lossless, they're just 16 numbers). Then it's an integer dot between the query row and looked-up bytes. Watch it on the worked example:

One table to rule every dim

Here is the fact the whole kernel stands on, and it is a property of how the codec is trained. Training the quantizer is nothing more than choosing 16 numbers: pool every residual value in the corpus — every dimension of every token of every document, one giant pile of plain scalars — sort the pile, cut it into 16 equal-count bins, and keep one representative value per bin. Those 16 numbers are the whole decoder; a 4-bit code just says which bin. ONE table, shared by every dimension of every token of every document. Hover any code below:

Two things about those sixteen numbers are doing quiet work. Read their spacing: neighbors near zero sit 10–11 apart (−5, 5, ±16, ±26), the middle pairs 16–21 apart, and the outermost jump is 37 (−127 to −90). That shape is equal-count binning meeting a peaked histogram: residuals are corrections, so small values are common and large ones rare — narrow bins land where the data is dense, and most codes get the finest resolution. And why is one table enough for 128 different dimensions? Because the centroid already absorbed what made dimensions differ. After token − centroid[cid], every dimension is left holding the same kind of thing: a small, near-zero-mean correction at a similar scale. Statistically interchangeable leftovers can share one quantile grid — that sharing is exactly the "little accuracy" the note above trades (per-dim tables would hug each dimension's histogram slightly tighter), and it buys the 16-byte decoder the rest of the class is built on.

2P − T was a special case all along

Write the residual term grouped by table entry instead of by dimension: Σ_d q_d·w[code_d] = Σ_c w_c · S_c, where S_c sums the query values whose code is c. Now drag the bit width down and watch class 04 fall out:

tbl / pshufb: sixteen lookups, one instruction

Class 04 needed a dot-product instruction; this class adds one more to your toolbox. NEON's tbl (x86: pshufb) treats one register as a 16-entry byte table and another as 16 indices — and performs all 16 lookups in a single instruction. Step through it:

The kernel: fusion, again

Everything else you already know from class 04 — same fusion, same loop order, one new wrinkle at the end:

// expand ONCE, amortize over all query rows
for chunk in 0..4:        // 16 B codes each
  v  = ld1  codes[chunk]
  hi = ushr v, #4          // even dims
  lo = and  v, 0x0F        // odd dims
  w  = tbl  table, hi/lo   // ← ch 06
// per query row: class 04's inner loop
for qi:
  acc = 8 × sdot(q_perm, w)
  s = s_qw[qi]·acc + cdot[cid][qi]  // f32!
  best[qi] = max(best[qi], s)

w  = pshufb(table, nibbles)   // lookup
// no signed×signed byte-dot on AVX2 —
// transfer q's sign onto w instead:
ws = psignb(w, q)             // w·sign(q)
p  = pmaddubsw(|q|, ws)       // u8×i8 pairs
acc = pmaddwd(p, 1) + acc     // widen i32
// |q|,w ≤ 127 → pair sum ≤ 32258:
// the i16 lanes can never saturate —
// that's WHY quantize_lut clips to ±127

The wrinkle: the max moved to float. Class 04's kernel kept every score an integer until the very end, because one shared scale can't change which token wins a max. Here the centroid term differs per token, so each (query row, token) score must fold to f32 before comparing. Still bit-identical across scalar, NEON, and AVX2 rungs — the f32 operation order is part of the spec, tested on every CI platform. That fold is the whole reason this kernel doesn't already match binary — which is worth pinning down, because it also answers a question the numbers provoke.

Measured, three ways

Kernel microbench (Apple M4, idle, 32-token query × 2000 docs × 80 tokens), then end-to-end, then across CPUs:

Read the quality column carefully: the fused path costs −0.0005 NDCG — the int8 error lands only on the residual, and the centroid term (most of the magnitude) stays float. The uncomfortable table from chapter 01 is resolved: best-quality scheme, binary-path speed.

The chapter-09 vectorized fold (the shipped rung) replicates on every platform — 1.5× on Neoverse to 2.2× on the M1. One inversion worth its own note: on AVX2 the fold helps residual-1 most (2.0×), so its affine-SAD form lands at 5.5 µs/doc — faster than every other residual rung and faster than the binary kernel's own SAD loop. Same source, three CPUs, three orderings.

The third runner earned its place the day it was added: it caught the sdot assembly relying on Apple's baseline features (a one-line .arch_extension fix), and it settled class 04's SMMLA cliffhanger — the instruction that only tied SDOT on the M4 wins 1.40× on Neoverse, so the dispatcher now prefers it wherever i8mm exists. Measure on more than one microarchitecture; keep your negative results.

Why 4, 2, and 1 bits all cost the same

The fused family measures 2.19 / 2.17 / 2.35 µs per doc (M4) while streaming 64 / 32 / 16 bytes of codes per token. Three payload sizes, one speed — and the smallest payload is the slowest. If bytes set the pace, residual-1 would be 4× faster than residual-4. It isn't, and the reason is the most instructive thing this kernel has to say.

The whole answer in one sentence: nbits only changes work that happens once per doc token, and that work is amortized over all ~32 query rows the token is scored against — so it barely moves the total. The rest of this chapter is why that's true, and one experiment that proves it.

Fusion changed which resource is scarce. The decode → GEMM baseline is memory-bound (class 02, chapter 06): every token round-trips through a float buffer, so time is proportional to bytes moved — and sure enough, the numpy baselines order exactly by nbits. The fused kernels don't:

The fused kernel expands a token's codes into registers and never writes them back; the expanded bytes are then reused by every query row. That kills the memory traffic that made bytes matter. What's left is an instruction count — and it splits into two very unequal parts, charged at two different rates:

• Expand (unpack codes → look up 128 weight bytes): the only part nbits changes — 4-bit unpacks differently than 1-bit. Charged once per doc token.
• Score core (8 SDOTs → the float fold → the max): identical for every nbits, because expand already normalized all three to the same 128 weight bytes. Charged once per query row — so ~32× as often as expand.

nbits moves only the small charge, and only the one that's paid rarely. Drag the slider to watch the nbits-dependent slice shrink as the token is shared across more rows:

At real query sizes the nbits-dependent slice is ~5% of the instructions, so 64-byte and 16-byte codes finish within noise of each other. The model also predicts the two apparent exceptions. Binary's ~1.9 µs is not "fewer bytes" either — it has the same amortization but a cheaper core: no centroid term, so its max stays integer and it skips the per-row float fold (chapter 07). And residual-1 lands slowest on ARM yet fastest-residual on AVX2 because its expand and fold differ in kind, not in bytes — on x86 its affine form rides the cheap masked-SAD path. Once bytes stop explaining the time, microarchitecture is all that's left.

Bytes still matter where memory is genuinely the scarce resource: the resident index (85 / 47 / 28 MB), and they would matter again if the working set outgrew cache and streaming bandwidth re-entered. The claim is not "bytes never matter" — it's that this loop, after fusion, no longer pays for them.

Does it cover 1, 2, and 4 bits?

The identity covers all three — chapter 05 is the proof. The kernels stand today like this:

Three honest observations. First, binary didn't need retrofitting: its specialized form is better than running it through the general kernel — with only two weights the "lookup" degenerates to the mask class 04 already built, there's no centroid term, and the max stays in integers. The general identity tells you why the special case was possible; it doesn't replace it. Second, residual-2 transferred exactly as the identity predicted (its 4-entry table fits the same register sixteen times over; only the unpack changes) and measured 82 → 6.9 ms — but chapter 09's flat line means speed is no longer an argument for it: the argument is now quality per byte, and it is strong — with chapter 13's renormalization its column reads 0.7635, parity with exhaustive float at 47 MB against 610, arguably this index's sweet spot. Third, the pattern is the same one that produced class 04: find the identity, quantize the operands, fuse the loop, measure on every platform. And the pattern survived contact with a production engine — the next three chapters are that story, and they are why this class is titled fused int8 residual scoring rather than "the LUT kernel."

The receipt: beating a real GEMM engine

Chapter 01 promised to remove the red number without touching NDCG. Here is the receipt, measured against the two GEMMs you'd actually reach for: next-plaid's own maxsim_score (per-doc BLAS sgemm + SIMD max — the honest baseline) and mixedbread-ai/maxsim-cpu, a production-tuned GEMM (batched sgemm + hand-vectorized fold). Full SciFact, single-thread, and the decode-to-float is handed to the GEMM rows for free:

NDCG tracks the f32 ceiling to a rounding error (r4 0.7569 vs 0.7599 exact), because our int8 ranking of the codes reproduces the reconstruction's order. Binary is the standout — the biggest win at 1.98×, because with no centroid term its max stays integer (chapter 07) and it skips the per-row float fold; it also holds 98.5% NDCG at the same 16 bytes where residual-1 keeps only 80%, so binary — not r1 — is the right 16-byte route. The other surprise is that mixedbread's optimized GEMM is the slowest column — and the reason is the whole point of the class:

Optimizing the fold — all a "better GEMM" can do — only creeps toward the 5.1 floor; mixedbread's genuine best case (uniform API, all 10 cores) lands at 6.0, right on it, and its variable-length API overhead is what pushes the single-thread number to 15. You cannot beat BLAS by out-BLAS-ing BLAS. Our kernel is under the floor for the one reason a GEMM never can be: it never runs the f32 GEMM — it scores an int8 view of the codes and never materializes the 512-byte reconstruction the GEMM is optimizing around. Reproduce with bench/compare_gemm.py.

The port: generalize, then re-specialize

The kernel then left the classroom. next-plaid's asymmetric residual path now scores packed codes directly inside its two-stage engine: the same tbl + sdot on NEON, pshufb + psignb + pmaddubsw on AVX2, bit-identity-tested against the scalar spec on all three CI targets. Quality on the full grid — 3 models × 3 datasets × r4/r2/r1 — moved by |ΔNDCG@10| ≤ 0.0021 against decompress → GEMM. The uncomfortable table generalized: it was never a SciFact fact, it's a scheme fact. The port also taught four lessons the toy repo could not.

The porting lesson. The production codec packs codes for any nbits and dim, with its own LSB-first bit-reversal — so the port's first decoder was the honest general one: a 256-entry byte → weights table. Correct, general, and unSIMDable — tbl/pshufb address 16 entries, not 256 — so the "fast" port was a scalar walk in disguise, and at mixedbread's 786-token doc shape it measured 0.46×: slower than the GEMM it replaced. The fix was not a new idea; it was chapter 06 recovered with a proof: factor the byte table into per-key-position 16-entry nibble tables (codes never straddle a nibble), verify the factorization over all 256 byte values when the table is built, and fall back to scalar if it ever fails. Eight little tables instead of one — each key position reads its own permutation of the same 16 weights, because that's what the bit-reversal does. Generalize for the engine; re-specialize for the register; make the equivalence a checked invariant, not a hope.

What the port forced us to admit. Inside a real engine the obvious question finally got asked: if the win is skipping decompression, do we need a lookup table at all — couldn't this just be two matrix products, centroid and residual, each against the query? Answer: it is two matrix products. That is what the kernel has been since chapter 03:

The table holds weights, not scores — it never sees the query. It is the decoder the packed storage format forces, and the hardware makes decoding ~free: inside next-plaid, rescore time is flat across r4/r2/r1 (16.5 / 16.3 / 16.7 ms, x86 CI, 1024-doc shortlists) — chapter 09's flat line, reproduced in production. Two corners of the design space fall out for free. Store residuals as raw int8 (128 B/token) and no table exists at all — the table's entire job is buying r4's 64 B at zero decode cost. Expand codes to float and hand the residual to a real GEMM, and you have reinvented decompression — the very term the split exists to delete. So the honest name for this class's kernel is fused int8 residual scoring — the name this class now carries; "LUT kernel" keeps making smart people ask where the lookup went.

Renormalize without the vector

nano-plaid's first release never normalized a reconstruction — its float rescore scored centroid + weights[codes] raw, and score_residual_lut matched that spec bit for bit. Consistent on both sides, so the toy was internally correct. Production's float path has a different contract: next-plaid renormalizes every reconstructed token before its GEMM (row /= ‖row‖, in the codec), because quantization bends unit vectors — the stored token is t̂ = centroid + quantized residual, and ‖t̂‖ is only near 1. A port that skips this is bit-faithful to nano and wrong for the engine.

The fused kernel has a problem nano's float path doesn't: there is no vector to divide — deleting the reconstruction was the whole point. The escape is the same move this class has now made three times (the scales in chapter 03, the centroid gather, and now the norm): push the float work out of the vector world and onto one scalar,

Normalizing the vector and scaling the finished score are the same number — and the right-hand factor is a gift, because ‖t̂‖ never sees the query. It depends only on the stored centroid id and codes, so it is computed once per index — walking each token's packed bytes, reconstructing in float, one 1/√Σv² per token — and cached: 4 bytes per token, built lazily on the first asymmetric search (a one-time 29–59 ms on our test indexes). It is computed from the float bucket weights, not the int8 ones, so it equals exactly the divisor the float path would have used; the int8 lane plays no part in it. One placement rule matters: MaxSim is a max across tokens, and each token owns its own 1/‖t̂‖ — a per-token positive scale changes who wins — so the multiply happens inside the token loop, before tokens compete, never after.

The receipt is production's, and it is not subtle: skipping the renormalize measures up to −0.17 NDCG@10 at nbits=1 on real corpora — a thousand times the ≤ 0.0021 the whole fused path is held to. The mechanism is the widget's: the coarser the codec, the more reconstruction lengths vary token to token, and without the correction MaxSim starts rewarding tokens whose reconstruction happens to come out long — length masquerading as similarity. At residual-4 the norms hug 1 and the factor is nearly invisible; at residual-1 it is the difference between "identical NDCG" and a broken scheme.

And then the toy paid the same bill, which is why the numbers all over this course changed. Porting inv_norms back into nano-plaid — one cached scalar per token, one multiply in every kernel's fold — moved full-SciFact NDCG@10 by +0.004 (residual-4), +0.030 (residual-2), and +0.116 (residual-1: 0.6312 → 0.7470), ordered exactly by how coarsely each codec bends the unit sphere. Two teaching claims died honorably: residual-1 does not lose 11 points to binary at equal bytes (it now edges it), and residual-2 turns out to sit at parity with exhaustive float. The bit-identity suite and the cross-language bridge pin the factor on every rung, every platform.

One production scar to keep: the cache is filled lazily behind a OnceLock, and the first version filled it with a rayon parallel loop — callable from inside rayon workers. A cold index under concurrent load could park every thread on the lock with no one left to run the fill. Lazy + threads is a design decision, not a convenience; review caught it before production did.

The ablation: attribution under oath

The port carried three optimizations: the chapter-09 vectorized fold, the ARM transpose-reduce, and one this class never needed — transposing stage 1's [nq, K] matrix to centroid-major, so a doc token's centroid scores sit in one contiguous strip instead of being gathered K floats apart (look back at the split line above: the word gather is doing a lot of quiet work). A switch (NP_ASYM_ABLATE) turns off exactly one component per run — same binary, same cached indexes, same queries, and the bit-exactness suite runs under every mode, so an ablation cannot quietly change the computation it is timing. Each row adds one component to the row above (r4, exact-kernel ms/query, 1024-doc shortlists):

Three findings, none of which we predicted. The layout did the work, everywhere — on Neoverse it alone is worth 1.6–1.8×; the fold vectorization this class spent two chapters on is the smaller half on every CPU. The fold's sign flips with the microarchitecture: it pays on Neoverse and x86 but is a 17–20% regression on the M4 by itself — writing every row's accumulator to scratch and re-reading it costs more there than folding four rows at once saves — and the transpose-reduce's real job turns out to be repairing exactly that (fold + tr ≈ scalar fold + good layout on the M4: a wash). And chapter 09's own 2.1× did not transfer. nano's kernel is compiled for dim = 128 — the expanded weights live in eight registers, the loops straight-line. The port's runtime-dim kernel re-streams those weights from a stack buffer on every query row, so the fold's instruction savings sat on top of an untouched load stream. An optimization's value is a property of the kernel shape it was measured in; it moves to a new shape as a hypothesis, not as a number.

Two harness lessons, earned the hard way. The profiler first applied the layout outside the ablation switch, so the pre-work cell handed the kernel a transposed matrix and tripped the kernel's own shape assert — the benchmark must route through the same layout policy it claims to measure, or it silently loses its baseline row. And an inert ablation (here: force-AVX2 on machines that have no AVX-512) re-measures the production kernel under a different name, so its deviation from 1.00× is the harness's noise floor, for free: 1–3% on CI VMs, ~6% on the M4 — and one hosted mac runner read 1.43×, which retired that runner's numbers, not the kernel's.

And where the port moved the story. Whole-pipeline profiling — production stage 1 + rescore, 1024-doc shortlists — says the next chapter is not in this loop:

Synthetic 2,000-doc corpus, 180 tokens/doc, dim 128, shared CI VMs — ratios, not absolutes.

Kernel wins dilute end-to-end because stage 1 — query × centroids, IVF probing — does not shrink when stage 2 does. At a mere 2,000 documents the binary rescore is already cheaper than the stage that feeds it, and stage 1 grows with the corpus while the shortlist stays ~1,024 docs. Beat decompression, and the bottleneck walks upstream. That is the next kernel's problem — and it will not be a lookup table. It is class 06.

Eight questions

Instant feedback, explanations included. Miss one? The chapter number is next to it.