Optimizing Inference Latency: Where the Milliseconds Go

"Make the model faster" is not an actionable instruction, because LLM latency isn't one number. It's two very different phases stitched together, measured by metrics that pull against each other, on hardware that's bottlenecked somewhere you probably haven't looked. Before you reach for a quantization library, it pays to know where the time actually goes.

The two phases that explain everything

Generation splits into two stages with opposite performance characteristics:

• Prefill processes your whole prompt in parallel to produce the first token. Every prompt token is a matrix-multiply against the weights, so the GPU's math units stay busy — it's compute-bound and scales with prompt length. This is your time-to-first-token (TTFT).
• Decode generates the rest, one token at a time. Each step reads the entire model and the growing KV cache out of GPU memory to produce a single token, so the math units sit mostly idle waiting on memory. It's memory-bandwidth-bound. This sets your inter-token latency (TPOT).

That split is the whole game. Prefill has high arithmetic intensity (lots of FLOPs per byte read); single-stream decode has almost none — at batch size 1 you read billions of parameters to compute one token, so you're paying for memory bandwidth, not FLOPs. A long prompt hurts prefill; a long output hurts decode; and most "the model is slow" complaints are really about one phase. Fixing the wrong one wastes weeks.

Pick the metric your users actually feel

Optimize the number that maps to experience. Four matter, and they don't move together:

• TTFT — how long until the first token. Dominated by prefill and queueing.
• TPOT / inter-token latency — the gap between tokens once streaming starts. Dominated by decode.
• Total latency — what a non-streaming caller (a tool call whose full JSON gets parsed) waits for.
• Throughput — tokens/sec across the whole fleet, which is what your bill tracks.

For a streaming chat UI, TTFT and TPOT dominate perception — users start reading before generation finishes. For a batch pipeline, total latency and throughput win. And always look at the tail: p99, not the average, is where users churn. A useful framing from recent serving research is goodput — not raw throughput, but the share of requests that actually meet your latency SLO (Zhong et al., 2024). A server can post great average throughput while quietly missing the SLO on a third of requests.

The levers, in priority order

1. Pick the right model (the biggest lever)

Most latency is decided before you tune anything: a smaller or distilled model that meets your quality bar will beat any amount of serving cleverness on a model that's too big. Route easy requests to a small model and reserve the large one for hard cases. The cheapest millisecond is the one you never compute.

2. Shrink what decode has to read

Because decode is memory-bound, every byte you don't read is a faster token. Two structural wins:

• Quantization. Weights in 8- or 4-bit (AWQ, GPTQ, FP8) mean less to read each step, plus a 4–8× memory cut that fits the model on smaller or fewer GPUs. The trade is quality — measure it on your task, since degradation is task-dependent, not a single leaderboard number.
• A smaller KV cache. The cache grows with batch × sequence length and is read on every decode step. Grouped-query attention (Ainslie et al., 2023) shares key/value heads across query heads, cutting cache size several-fold — which is why nearly every modern model ships with it. KV-cache quantization pushes the same idea further.

3. Serving: batching, paging, and attention kernels

This is where modern inference engines earn their keep. PagedAttention (Kwon et al., 2023), the idea behind vLLM, manages the KV cache in non-contiguous blocks like virtual memory — killing the fragmentation that used to waste GPU memory and letting you hold far more concurrent sequences. Continuous batching then slots new requests into the running batch as others finish, instead of waiting for the slowest sequence. Underneath, an IO-aware kernel like FlashAttention (Dao et al., 2022) keeps attention from spilling to slow HBM, which mostly helps the compute-bound prefill.

The honest caveat: batching optimizes throughput, sometimes at the expense of a single request's latency. If you serve one latency-critical user, aggressive batching can hurt; if you serve many, it's a large win. Know which world you're in.

4. Stop prefill and decode from fighting

Here's a problem the two-phase model predicts: a compute-heavy prefill and latency-sensitive decodes want the same GPU. Run them together naively and a long prompt's prefill stalls everyone else's token stream — TTFT and TPOT trade off against each other. Two camps have emerged, and which one fits depends on your traffic:

• Chunked prefill (Sarathi-Serve, Agrawal et al., 2024) splits a long prompt into pieces and piggybacks each chunk onto ongoing decode batches, so neither phase fully starves the other on one machine.
• Phase disaggregation (DistServe, Splitwise) runs prefill and decode on separate GPU pools, sized and tuned independently, shipping the KV cache between them over a fast interconnect. It removes the interference entirely at the cost of more moving parts.

You don't have to build either — but knowing the failure mode tells you what to look for when TTFT spikes only under mixed load.

5. Speculative decoding

Decode is memory-bound, so the GPU has spare compute — speculative decoding spends it. A cheap drafter proposes several tokens; the big model verifies them in one forward pass, accepting the longest matching run. Leviathan et al. (2023) showed this is lossless — the output distribution is unchanged. The classic version needs a separate draft model; newer self-drafting methods like Medusa and EAGLE-2 attach lightweight prediction heads to the model itself, avoiding a second model to host. The catch is workload-dependent acceptance: low-entropy text like code or templated output accepts 70–90% of draft tokens, while open-ended generation can fall to 20–40%, where the overhead barely pays off.

6. Don't compute it twice

The fastest inference is a cache hit. Prefix caching — like vLLM's automatic prefix caching — keeps the KV blocks of a shared prompt prefix (a long system prompt, a RAG preamble, a chat history) so repeat requests skip recomputing it, cutting TTFT directly. Semantic caching serves a stored answer when a new query is close enough in embedding space. And the cheapest prefill win of all: shorten the prompt. Trimming a bloated system prompt or oversized few-shot block cuts TTFT on every single call.

The short version

Sources

• Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023) — arXiv:2309.06180
• Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (2022) — arXiv:2205.14135
• Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (2023) — arXiv:2305.13245
• Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve" (2024) — arXiv:2403.02310
• Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving" (2024) — arXiv:2401.09670
• Patel et al., "Splitwise: Efficient Generative LLM Inference Using Phase Splitting" (2023) — arXiv:2311.18677
• Leviathan et al., "Fast Inference from Transformers via Speculative Decoding" (2023) — arXiv:2211.17192
• Cai et al., "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads" (2024) — arXiv:2401.10774 · Li et al., "EAGLE-2" (2024) — arXiv:2406.16858