Your CPU can do sixteen multiplications in one instruction.

Scalar vs SIMD: a race

A normal ("scalar") instruction touches one number at a time. A SIMD instruction touches a whole row of them. Same task below — double 32 numbers — two ways. Press run.

Registers & lanes

A SIMD register is just a fixed-width box of bits — 128, 256, or 512 of them, depending on the CPU. You choose how to slice it: into 8-bit, 16-bit, or 32-bit pieces. Each piece is a lane, and every lane gets the same operation.

Lane-wise operations

The bread and butter: take two registers, apply one operation independently in every lane, get a register of results. No lane ever talks to its neighbor. Pick an operation and execute it.

The sleeper hit is compare. It doesn't produce true/false — it produces a mask: all-ones (FFFF) where the test passed, all-zeros where it failed. AND-ing data with a mask keeps some lanes and zeroes others — which is how SIMD code does "if" without branching. Hold that thought; the case study in chapter 09 is built on exactly this.

The horizontal tax

SIMD is blazing across lanes ("vertical") and awkward within a register ("horizontal"). Summing all lanes into one number takes log₂(lanes) rounds of shuffling — watch:

Consequence: good SIMD code keeps results spread across lanes for as long as possible — accumulate vertically a thousand times, fold horizontally once at the very end. When you see a weird data layout in a fast kernel, this is almost always why.

Two worlds: x86 and ARM

Every mainstream CPU has SIMD, but the two big instruction-set families evolved it differently. x86 (Intel/AMD) kept widening the register. ARM (Apple, AWS, phones) kept the register at 128 bits and instead added smarter instructions and more parallel execution pipes.

So is a 512-bit x86 chip 4× faster than a 128-bit Apple chip? No — and this is the most common beginner miscalibration. An Apple M4 has four 128-bit SIMD pipes running every cycle, so its throughput per cycle rivals one 512-bit unit, with none of the downclocking older AVX-512 chips suffered. Width is one axis; pipes × width × what each instruction does is the real budget.

Reading the mnemonics

Instruction names look like cat-on-keyboard but are strict little sentences. The same operation, both dialects:

= vpdpbusd (x86 AVX-512 VNNI)

= sdot v0.4s, v1.16b, v2.16b (ARM NEON)

Dot-product instructions

Multiply pairs of numbers and add them up — the dot product — is the inner loop of neural networks, embeddings, and similarity search. It matters so much that both families added single instructions that do multiply + add + accumulate across many int8 lanes at once. Here is ARM's sdot, live:

Three ways in

You almost never write raw machine code. There are three doors, in escalating order of effort and control — real code from a real kernel behind each one:

// Write branch-free code and the compiler
// emits SIMD for you. The trick: turn
// `if (bit) sum += q[i]` into a mask:
let mask = -((byte >> j) & 1);  // 0 or -1
p += q[i] & mask;               // no branch!
// Same math, but now it's straight-line
// arithmetic LLVM can vectorize.

// C/Rust functions that map 1:1 to
// instructions. You pick the ops; the
// compiler still allocates registers.
let s = _mm256_sad_epu8(
    _mm256_and_si256(mask, qbytes),
    zero);
// ^ one AVX2 instruction each

// When the intrinsic doesn't exist yet
// (Rust's sdot was nightly-only):
asm!("sdot {out:v}.4s,
       {a:v}.16b, {b:v}.16b", …);
// Total control, zero safety net —
// YOU must check the CPU supports it.

Honest expectations, measured on the kernel in chapter 09: the autovectorizer bought ~2× over naive scalar. Hand-written intrinsics bought ~40×. The compiler is good at easy patterns and hopeless at exotic ones — it will never invent a data layout or discover that psadbw can fake a dot product. Layout is the human's job.

Dispatch & the traps

Here's the problem: you compile one binary, but your users' CPUs have different instruction sets. Run an AVX-512 instruction on a chip without it and the program doesn't get slower or wronger — it dies instantly (SIGILL: illegal instruction). So real libraries check the CPU at runtime and pick a kernel. Try it:

Field notes: three ways this bites in real life

Case study: one kernel, five rungs

From nano-plaid, a search engine that stores documents as 1-bit vectors (just the signs — 25× smaller). Scoring needs a dot product between an int8 query and those bits. The identity that makes it possible, then the ladder of implementations — every chapter of this page appears in it:

Read the ladder bottom-up and every lesson clicks into place:

Rung 2 is the shocker — the clever algebra made it slower than the float loop (a branch per bit is poison). Rung 3, the branchless mask rewrite, let the autovectorizer help a little. Rung 4 is the real jump — sdot on ARM, the psadbw impersonation on x86, plus the deciding human insight: expand each document's bits once in registers and reuse them across all 32 query rows (the horizontal-tax lesson wearing work clothes). Rung 5 is the humility check: smmla does 2× the math per instruction and ties anyway, because the M4 issues it at half the rate. Paper MACs lose to measured microarchitecture.

End to end: that kernel took real search over 5,183 scientific documents from 18 ms to 5.7 ms per query — 3.3× faster than exhaustive float search, using 22× less memory.

Six questions

Instant feedback, explanations included. Miss one? The chapter number is next to it.