LLM inference serving
An LLM API turns a prompt into a stream of tokens on very expensive, very scarce GPUs. Requests are wildly uneven (10 tokens or 200,000 in, 1 or 8,000 out), so the design is about keeping GPUs full without letting latency explode. The standard answer: token-based rate limits and priority queues at the front, continuous batching so a finished sequence’s slot is refilled on the next step, a paged KV cache so memory, not compute, sets the batch size, prefix caching for repeated prompts, and admission control that sheds load before queues grow. Prefill costs O(prompt) compute once; each decode step costs one read of the weights, shared by the whole batch; KV memory is O(tokens) per sequence, hundreds of KB per token.
The problem
Section titled “The problem”A client sends a conversation and wants a reply, usually streamed token by token. Behind the API sit pools of GPUs, each pool holding one model’s weights. Generating one token means a full pass through the model. A reply of 500 tokens is 500 sequential passes.
| Goal | What it means | Pulls toward |
|---|---|---|
| High GPU utilisation | GPUs cost dollars per hour each and are hard to get | Large batches, always full |
| Low latency | Fast first token (TTFT), steady tokens per second | Small batches, short queues, spare capacity |
| Fairness and isolation | One customer’s 200k-token batch job doesn’t stall everyone’s chat | Token-based limits, priority tiers |
| Predictable cost | Bill and limit by tokens actually used | Metering in the serving path |
Utilisation and latency pull hardest against each other. Most of the design is about getting both at once, and choosing which to give up under overload.
Requirements and scope
Section titled “Requirements and scope”| In scope (typical) | Ask about | |
|---|---|---|
| Functional | POST /messages: model, messages, max_tokens, stream. Streaming over server-sent events (SSE). Usage in the response. Prompt caching. |
Tool use, images, batch API with 24 h SLA? Several models and sizes? |
| Scale | Tens of thousands of requests/s across models; millions of tokens/s generated. | Traffic mix of chat vs agents vs offline? Typical prompt and output lengths? |
| Latency | Interactive: TTFT p99 of a few seconds; 50-100 tokens/s per stream. | Different SLOs per tier? |
| Consistency | Usage and billing exact. Limits approximately right (a small overshoot is fine). | Hard spend caps? |
| Availability | API stays up; under overload, reject quickly with a clear error rather than time out. | Multi-region failover? Data residency? |
| Out of scope | Training, model internals, safety classifiers (mention they sit in the path). |
Back of the envelope
Section titled “Back of the envelope”The number that sizes everything is KV cache bytes per token: each layer stores a key and a value vector per KV head for every token in context. Drag a 70B-class model with grouped-query attention to a 128k context and watch the batch size fall.
What to say out loud: “Decode is memory-bandwidth bound: every step reads all the weights, about 140 GB for a 70B model in fp16. On two GPUs at roughly 3.35 TB/s each that’s about 20 ms a step, so about 50 tokens/s per sequence whatever the batch size. That makes batching nearly free per sequence, and the batch size is capped by KV cache memory, not compute. So the design is about packing as many sequences into KV memory as possible.”
API and data model
Section titled “API and data model”POST /v1/messages x-api-key, anthropic-version { model, max_tokens, system?, messages: [...], stream: true, tools? } cache_control markers on long, stable prefix blocks (tools, system, documents)
stream (SSE): message_start -> content_block_start -> content_block_delta × N -> content_block_stop -> message_delta { stop_reason, usage } -> message_stoperrors: 429 rate_limit_error (you are over your limits; retry-after) 529 overloaded_error (we are over capacity; back off and retry)
Records: usage_event { request_id, org, model, input_tokens, cache_read, cache_write, output_tokens, ts } limit_bucket { org, model, kind: rpm | input_tpm | output_tpm, tokens, updated_at } sequence { id, tier, block_table: [physical block ids], length, state } -- inside a replica prefix_entry { hash(prefix blocks), replica, block ids, last_used } -- router's viewTwo decisions hide in that sketch:
- Limits are in tokens, per direction. Anthropic limits requests per minute, input tokens per minute and output tokens per minute, with a token bucket that refills continuously. Input is known up front; output is only known at the end. So you reserve, then settle (pseudocode below).
- Errors say whose fault it is. A 429 means the caller exceeded its own limit. An overloaded error means the service is out of capacity. Clients treat them differently, and dashboards must not mix them.
High-level design
Section titled “High-level design”| Component | Role | Why it’s there |
|---|---|---|
| API gateway (×N) | Auth, token counting, rate limits, SSE to the client, retries that are safe. | Cheap CPU work kept away from GPUs; rejects early. |
| Limit store | Token buckets per org and model, sharded in memory. | Many gateways share one limit; see the rate limiter deep dive. |
| Router | Picks a replica: model, load, free KV memory, prefix-cache affinity. Holds per-tier queues. | Placement decides cache hit rate and tail latency. |
| GPU replicas | One model copy per replica, sharded over several GPUs (tensor parallel). Runs a batching engine (vLLM, TensorRT-LLM, SGLang or in-house). | Where tokens are made; everything else exists to keep these busy. |
| KV / prefix cache | Per-replica KV blocks in GPU memory, spilling to CPU memory. | Reuses prefill work across requests that share a prefix. |
| Capacity control | Autoscaling, admission thresholds, load shedding, cross-region spill. | GPUs take minutes to add; overload must be handled by policy first. |
| Usage events | Stream of per-request token counts. | Billing, limit settlement, analytics. Off the hot path. |
Deep dive: prefill, decode and batching
Section titled “Deep dive: prefill, decode and batching”Generating text has two phases with opposite costs:
| Prefill | Decode | |
|---|---|---|
| What it does | Processes the whole prompt in one forward pass, fills the KV cache | Produces one token per forward pass, appending to the KV cache |
| Bound by | Compute (many tokens in parallel: big matrix multiplies) | Memory bandwidth (read all weights to produce one token) |
| Time | Grows with prompt length: ≈ 0.1-1 s for thousands of tokens | ≈ 10-30 ms per step, nearly flat in batch size until compute-bound |
| User metric | TTFT (time to first token) = queue wait + prefill | TPOT / inter-token latency, tokens/s per stream |
Because a decode step costs about the same for 1 sequence as for 32, you batch. The question is when a batch changes. Run both policies with mostly short replies and a few long ones.
| Policy | How it works | Utilisation | TTFT |
|---|---|---|---|
| Static batching | Form a batch, run until its longest reply finishes, then form the next | Poor: finished slots idle as padding | Bad: new requests wait for the whole batch |
| Continuous (in-flight) batching | Re-form the batch every step: finished sequences leave, waiting ones join | High | Low: wait only for a free slot |
| + Chunked prefill | Split long prefills into chunks mixed into decode steps | High | Steady inter-token latency: a 100k prompt doesn’t freeze everyone’s stream |
| Prefill/decode disaggregation | Separate GPU pools for prefill and decode; ship the KV cache between them | Each pool tuned to its bottleneck | Good for long prompts; costs KV transfer bandwidth |
Continuous batching comes from the Orca paper (OSDI 2022), which calls it iteration-level scheduling. Anyscale measured about 8× throughput over static batching from it alone, and up to 23× combined with vLLM’s memory management. The Scheduling concept page covers queue disciplines in general.
Deep dive: the KV cache and paging
Section titled “Deep dive: the KV cache and paging”The KV cache grows by one entry per token per sequence, and nobody knows in advance how long a reply will be. Early servers reserved a contiguous region for max_tokens per request. Most of it stayed empty, and the vLLM paper (PagedAttention, SOSP 2023) showed large fractions of KV memory lost to this reservation and to fragmentation.
PagedAttention borrows virtual memory from operating systems:
- KV memory is split into fixed-size blocks (for example 16 tokens each), like pages.
- Each sequence has a block table mapping its logical blocks to physical ones, like a page table. Blocks need not be contiguous.
- A sequence gets a new block only when it crosses a block boundary, so waste is at most one partly filled block per sequence.
- Sequences that share a prefix (the same system prompt, or parallel samples) point at the same physical blocks, with reference counts and copy-on-write.
- When blocks run out, the scheduler preempts a sequence: drop its blocks and recompute later, or swap them to CPU memory.
append_token(seq): if seq.length % BLOCK == 0: -- crossed into a new block b <- free_blocks.pop() or preempt_lowest_priority_and_retry() seq.block_table.append(b) ; refcount[b] <- 1 last <- seq.block_table[-1] if refcount[last] > 1: -- shared with another sequence: copy on write fresh <- copy_block(last) ; refcount[last] -= 1 ; refcount[fresh] <- 1 seq.block_table[-1] <- fresh ; last <- fresh write K, V for each layer into last at offset seq.length % BLOCK seq.length += 1
release(seq): for b in seq.block_table: refcount[b] -= 1; if 0: free_blocks.push(b)The vLLM paper reports 2-4× throughput over FasterTransformer and Orca at the same latency, mostly because more sequences fit in memory. See Memory efficiency for the general fragmentation story.
Prompt caching is the same block sharing, across requests and across time. Hash the token blocks of a prefix; if a replica already holds those blocks, skip their prefill. Agents and chat resend the same system prompt, tools and history every turn, so the saving is large. On the Claude API, cached prefixes have a 5-minute default lifetime (1 hour optional), cache reads cost a fraction of the input price (0.1× on most models), and cached input tokens don’t count toward the input-tokens-per-minute limit. The cache is a prefix: tools, then system, then messages, so a change early in the prompt invalidates everything after it. For the router, this means prefix affinity: send requests with the same prefix hash to the same replica, balanced against load (consistent hashing with bounded load works well). See Caching.
Deep dive: admission, priorities and overload
Section titled “Deep dive: admission, priorities and overload”GPUs cannot be added in seconds, so overload is handled by policy. The simulator feeds a pool of generation slots with random, long-tailed requests. Move the load toward 100% and watch the tail.
What it shows, and what production systems do about it:
- Latency is flat, then a cliff. Queueing delay grows roughly like 1 / (1 − utilisation). Serving pools target perhaps 60-80% of capacity for interactive traffic, and fill the rest with work that can wait.
- Priority tiers. Interactive chat ahead of agents ahead of offline batch jobs (a batch API with a 24-hour window is the extreme). Low tiers absorb the queueing and are shed first.
- Shed at the door. Reject a request in milliseconds when it would wait longer than its deadline, instead of letting it time out after holding a connection and a queue slot for 60 s. Rejected clients back off with jitter; see the rate limiting page.
- Token-aware admission. A replica admits a sequence only if it can reserve its prompt’s KV blocks plus some headroom; the router tracks free KV memory per replica, not just request counts.
on_request(org, model, req): in_tokens <- count_tokens(req) - cached_prefix_tokens(req) -- cached reads don't count if not buckets[org, model].input.take(in_tokens): return 429 with retry-after if not buckets[org, model].requests.take(1): return 429 if not buckets[org, model].output.has_any(): return 429 -- output charged as generated if pool_overloaded(model, tier(org)): return 529 overloaded stream tokens; every k tokens: buckets[org, model].output.take(k) -- may go briefly negative on finish: emit usage_event(actual input, cache read/write, actual output)Multi-region. Each region runs pools per model. A global router (or DNS plus per-region gateways) sends traffic to the nearest region with headroom. Capacity is planned per model per region weeks ahead; minute-to-minute, overflow spills to regions with spare GPUs, subject to data-residency rules. Prompt caches are per replica, so spilling costs cache hits; it’s still better than shedding.
Failure modes
Section titled “Failure modes”| What breaks | What you see | Design response |
|---|---|---|
| Traffic spike | Queues grow; TTFT p99 explodes | Admission control, shed lowest tier, spill to other regions, then autoscale |
| One customer’s huge batch job | Others’ latency rises | Token-based per-org limits; batch tier; per-tier queues |
| KV cache exhausted | Sequences can’t grow | Paged blocks, preempt by priority (recompute or swap), admit with headroom |
| Very long prompt | Everyone’s stream stutters during its prefill | Chunked prefill; separate prefill pool |
| GPU / replica failure | Streams cut mid-reply; KV lost | Health checks, drain; retry on another replica only if no tokens were sent; else error event |
| Cold start | New replicas take minutes (weights are 100s of GB) | Keep warm spare capacity; preload weights from local NVMe; scale on queue trends |
| Low prefix-cache hit rate | High prefill cost, slow TTFT | Prefix-affinity routing; cache-aware load balancing |
| Client retry storm | Overload feeds itself | retry-after, exponential backoff with jitter in SDKs, cheap rejection path |
| Metering lag | Customer exceeds limit or spend cap briefly | Settle continuously during streaming; accept small overshoot; hard caps checked up front |
What it costs
Section titled “What it costs”| Operation | Time | Space |
|---|---|---|
| Prefill a prompt of P tokens | O(P) forward compute (attention adds O(P²), dominant at long context); ≈ 0.1-1 s per few thousand tokens | O(P) KV: P × KV bytes per token |
| One decode step for a batch of b | ≈ weights ÷ memory bandwidth (10-30 ms), nearly flat in b until compute-bound | +b tokens of KV |
| Whole reply of N tokens | N sequential decode steps: N × step time | KV grows to (P + N) × bytes per token |
| KV per token | — | 2 × layers × KV heads × head dim × bytes (≈ 320 KB for a 70B GQA model in fp16) |
| Prefix-cache hit | Skips prefill for the cached part: O(new tokens) | Shared blocks, ref-counted |
| Rate-limit check | O(1) per bucket, one store round trip | ≈ tens of bytes per org per model |
| Throughput per replica | ≈ batch size × (1 / step time) tokens/s, e.g. 32 × 50 = 1,600 tokens/s | batch size capped by KV memory |
In the interview
Section titled “In the interview”- Clarify (5 min). Models and sizes, prompt and output lengths, streaming, tiers and SLOs (TTFT, tokens/s), batch API, regions.
- Numbers (5 min). Decode is bandwidth bound; KV bytes per token; batch size limited by KV memory. Say it with a concrete model.
- Request path (10 min). Gateway with token limits, router with queues and prefix affinity, replicas with a batching engine, SSE back, usage events.
- Engine (15 min). Prefill vs decode, static vs continuous batching, paged KV cache, preemption, prompt caching, chunked prefill or disaggregation.
- Overload and failures (10 min). Utilisation vs latency curve, priority tiers, shedding, 429 vs overloaded, mid-stream failure, multi-region spill.
Follow-ups to expect: “How do you rate-limit output you haven’t generated yet?” Charge as tokens stream, settle at the end; don’t reserve max_tokens. “Why not just add GPUs?” They take minutes to load and are capacity-planned weeks ahead; policy has to absorb spikes. “How would you make decode faster per stream?” Speculative decoding (a small draft model proposes tokens, the big one verifies several per step), quantisation (fp8 halves bytes read), and more tensor parallelism. “How do you test the scheduler?” Replay recorded traffic traces in a simulator and compare goodput: requests per second that meet their SLO.
Check yourself
Section titled “Check yourself”Go deeper
Section titled “Go deeper”- Kwon et al.: Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): block tables, sharing, preemption, and the 2-4× throughput result behind vLLM.
- Yu et al.: Orca, A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022): iteration-level scheduling and selective batching.
- Anyscale: How continuous batching enables 23x throughput in LLM inference: a clear explanation of static vs continuous batching, with benchmarks.
- Claude API docs: Prompt caching and Rate limits: cache lifetimes and pricing, RPM/ITPM/OTPM, token buckets, cache-aware limits.
Concept pages to drill: Scheduling · Memory efficiency · Rate limiting · Streaming · Caching · Load balancing · Resource allocation