The toll booth is memory, not math
Prefill runs once, in parallel. Every token after that is its own round trip to memory to reload the model's weights and KV cache, which is why decode speed lives or dies on memory bandwidth.
An inference request has two phases that behave nothing alike. Prefill processes the whole prompt at once, every token attending to every other in parallel, and it happens exactly once. Decode is sequential — one token out, fed back in, one token out again — and there's no parallelizing around it; token 200 can't start until token 199 exists. The instinct is to treat both as a compute problem, because that's how training works. Decode doesn't work that way. Every step has to reload the model's weights and the running KV cache out of memory before it can produce one token. The arithmetic is cheap. The trip to memory, repeated once per output token, is what the clock is actually measuring — decode is memory-bandwidth-bound, not FLOPs-bound. This isn't abstract on my own setup: a Mac mini M4 with 16GB of unified memory, memory-constrained enough that it dictates the whole design. Weights get quantized from 16-bit down to 4-bit, and a router sends simple prompts to the small quantized model to keep tokens-per-second high. The upgrade I've got planned, an M4 Pro with 48GB, buys room to run larger models locally, not raw speed. The same constraint shows up at enterprise scale wearing different clothes — a fleet problem optimizing for the same two numbers, plus cost multiplied across every concurrent request. Prefill is a sum you pay once. Decode is a toll you pay per token, and the toll booth is memory, not math.