This chapter builds the vocabulary the scenarios assume. It is organized by problem, not by feature list, because that is how you should be able to reproduce it: an interviewer asks "how would you handle X" and you should be reaching for the mechanism, not the flag name.

A caution that applies to the whole chapter: flags and version numbers move. The mechanisms do not. Where something is version-specific it is marked. Where it is physics, it will still be true in five years.

4.0 What SGLang is

Two things wearing one name.

A frontend language — a Python DSL for expressing structured LLM programs: multi-call flows, branching, parallel forks, constrained generation, tool use. It exists because complex prompting patterns have structure the runtime could exploit if only it could see it.

A runtime (SRT, SGLang Runtime) — a high-performance serving engine: scheduler, KV cache manager, model executor, attention kernels. This is what people mean 95% of the time, and what every scenario in this book is about.

The original insight tying them together: if the runtime can see that many requests share prefixes — because a program forks, or because every request carries the same system prompt — it can avoid recomputing them. That insight became RadixAttention, and the runtime grew from there into a general-purpose engine that competes on raw serving throughput regardless of whether you use the frontend at all.

4.1 The request path

   HTTP request (OpenAI-compatible)
        │
        ▼
   ┌──────────────────┐
   │  Tokenizer proc  │   text → token ids, multimodal preprocessing
   └────────┬─────────┘
            │  (ZMQ IPC)
            ▼
   ┌──────────────────────────────────────────────┐
   │  Scheduler process                           │
   │   • waiting queue + admission policy         │
   │   • batch formation (prefill vs decode)      │
   │   • radix cache: match / insert / evict      │
   │   • grammar / speculative bookkeeping        │
   └────────┬─────────────────────────────────────┘
            │
            ▼
   ┌──────────────────────────────────────────────┐
   │  TP worker ranks  (one process per GPU)      │
   │   • model forward (attention backend + MoE)  │
   │   • paged KV pool                            │
   │   • CUDA graph replay                        │
   └────────┬─────────────────────────────────────┘
            │
            ▼
   ┌──────────────────┐
   │ Detokenizer proc │   token ids → text, streamed out
   └──────────────────┘

Why separate processes? Python's GIL. Tokenization, scheduling, and detokenization are CPU work that would otherwise serialize against the GPU-driving loop. Splitting them into processes connected by ZMQ lets CPU work overlap with GPU execution.

Why this matters in an interview: at small model sizes and large batch counts, the CPU scheduler becomes the bottleneck, not the GPU. A decode step for an 8B model can take 3 ms while naive Python scheduling overhead is 2 ms — you'd be leaving 40% on the table. SGLang's zero-overhead / overlapped scheduler runs the CPU-side work for step N+1 concurrently with GPU execution of step N, hiding it entirely. When someone asks "why is my small model slower than the roofline says it should be," CPU overhead and kernel launch overhead are the first two suspects, and CUDA graphs plus the overlapped scheduler are the answers.

4.2 Continuous batching, and the two phases

The problem. Requests arrive at different times and finish at different lengths. Static batching — collect N requests, run them to completion together — wastes the GPU twice: idle slots while waiting to fill the batch, and idle slots after short sequences finish while the batch waits for the longest one.

Continuous batching (a.k.a. in-flight batching). The batch is re-formed every decode step. Finished sequences leave, waiting requests join. Utilization stays high regardless of length variance. This is table stakes in every modern engine, and the reason it works is §3.5: decode is bandwidth bound, so adding sequences to a batch costs almost nothing until you hit the compute crossover or run out of KV.

The two phases, and why they conflict:

	Prefill	Decode
Work	All input tokens at once	One token per sequence
Bound by	Compute (FLOPs)	Memory bandwidth
Wants	Few, large chunks	Many concurrent sequences
Latency it drives	TTFT	ITL
Batch effect	Saturates fast	Nearly free scaling

Running both on the same GPU means they interfere. A 32k-token prefill occupies the device for ~200 ms, during which every decoding sequence stalls — the user watching a stream sees a hitch. This is prefill-induced ITL jitter, and it is the origin of both chunked prefill and PD disaggregation.

Chunked prefill. Split a long prefill into fixed-size chunks (say 2,048 or 8,192 tokens) and interleave them with decode steps. The long request now takes slightly longer overall, but it stops holding the decode batch hostage. The knob is chunk size: smaller chunks mean smoother ITL and worse prefill efficiency (each chunk re-reads weights); larger chunks mean the reverse. Naming this tradeoff and giving a number is a reliable way to demonstrate depth.

4.3 RadixAttention — automatic prefix caching

The problem. Requests share prefixes constantly: a shared system prompt, a conversation's history on every turn, an agent re-sending its whole trajectory each step, a batch of few-shot prompts with identical examples. Recomputing that prefill is pure waste.

The mechanism. SGLang maintains a radix tree (compressed trie) keyed on token sequences, where each node owns the KV cache pages for its token span. When a request arrives:

Walk the tree, matching its token prefix as far as possible.
The matched span's KV is reused directly — zero prefill for those tokens.
Prefill runs only for the unmatched suffix.
On completion, the request's tokens are inserted into the tree so the next request can match them.

Because the tree is keyed on tokens, sharing is automatic and transparent — no caller-declared cache keys, no explicit "cache this prompt" API. Two unrelated users who happen to send the same system prompt share its KV.

The economics. With a 4k-token shared system prompt and 200-token user turns, a naive system prefills 4,200 tokens per request; with prefix caching it prefills 200. That is a 95% reduction in prefill FLOPs, which for a prefill-heavy workload can mean a fleet that is a third of the size.

The costs — and these are what the deep dive is about:

Memory contention. Cached prefixes and running requests draw from the same KV pool. Every cached token is a token not available for concurrency. Under pressure the eviction policy arbitrates, typically LRU with reference counting so in-flight requests' pages are pinned.
Eviction storms. Evict a hot shared prefix and the next N requests all re-prefill it simultaneously.
Routing coupling. The cache is per replica. A request only benefits if it lands on the node that holds its prefix. This makes your load balancer cache-aware and puts cache affinity in direct tension with load balance — a tension that gets its own full scenario (S06).
Correctness edges. Anything that changes the KV — a different LoRA adapter, a different sampling path that alters the tokens, a model version bump — must be part of the cache identity or you will serve corrupted state.

The number to ask for: cache hit rate, meaning the fraction of prefill tokens served from cache. Under 20%, prefix caching is bookkeeping. Over 70%, it is the dominant cost lever in your system.

4.4 Paged KV memory

KV cache is allocated in fixed-size pages (blocks) rather than contiguous per-sequence buffers. Two consequences:

No external fragmentation, so you can pack far more sequences into the same memory.
Sharing is possible at page granularity — the prefix tree points multiple sequences at the same physical pages with reference counts.

The page size is a real tradeoff: small pages give finer sharing and less internal waste, larger pages reduce indexing overhead and suit some attention kernels better. Typical values are 1, 16, 32, or 64 tokens.

Preemption. When KV runs out, the scheduler must evict a running request. Two options: recompute (drop its KV, re-prefill later) or swap (move KV to CPU, bring it back). Recompute is simpler and often faster than a PCIe round trip; swap preserves work at the cost of bandwidth. Either way, preemption under sustained overload is a throughput death spiral — preempted work returns as new prefill work, which increases pressure. The real fix is admission control upstream, and saying that is the correct answer when an interviewer describes collapsing throughput.

4.5 Parallelism

Tensor parallelism (TP). Split each layer's weights across GPUs; all-reduce activations every layer. Reduces per-GPU weight bytes, which directly lowers the decode bandwidth floor — TP is the main lever on single-stream latency. Costs an all-reduce per layer, so it wants NVLink and should stay inside a node. TP=8 is the standard unit.

Data parallelism (DP). Full model replicas, requests distributed across them. Linear throughput scaling, no communication, but each replica needs the whole model and holds its own cache.

Pipeline parallelism (PP). Split layers across devices. Cheap communication (activations at stage boundaries only) so it tolerates slower links, but it introduces bubbles and adds latency. Used mainly when a model cannot fit any other way.

Expert parallelism (EP). For MoE: distribute experts across GPUs. Each token is routed to its top-k experts, which requires an all-to-all dispatch of tokens to expert-owning ranks and a second all-to-all to combine results. This is how you serve a 671B model without replicating 671 GB per GPU.

DP attention. A DeepSeek/MLA-specific trick worth knowing by name. Under plain TP, MLA's single latent KV head gets duplicated across all TP ranks — the KV cache is replicated N times, wasting most of your memory. Running attention data-parallel while the MoE layers stay expert-parallel eliminates that duplication. It is the difference between a workable and an unworkable DeepSeek deployment, and mentioning it unprompted in a DeepSeek scenario is a strong signal.

How they compose in a large MoE deployment: attention layers run DP (or modest TP), MoE layers run EP across many nodes, and the two are stitched together per layer. A published SGLang configuration for DeepSeek-V3 on 96 H100s ran prefill and decode as separate pools with large-scale EP across both.

4.6 MoE at scale: the load-balance problem

Routing is learned, therefore uneven. Some experts are far more popular than others. Under EP this means the GPUs holding hot experts become stragglers, and every rank waits for the slowest at the all-to-all barrier. Utilization drops toward the ratio of mean to max expert load.

Mechanisms that address it:

EPLB (expert-parallel load balancing). Redistribute or replicate experts across ranks based on observed load statistics. Hot experts get multiple replicas. Rebalancing can be periodic (offline, using logged distributions) or dynamic.
DeepEP-style optimized all-to-all kernels with separate low-latency (decode) and high-throughput (prefill) paths.
Two-batch overlap (TBO). Split the batch in two and overlap one micro-batch's communication with the other's computation, hiding all-to-all latency behind compute.

The interview framing: MoE trades memory for compute, and the bill arrives as communication and load imbalance. If someone proposes a large MoE deployment without mentioning expert imbalance, that is exactly where the follow-up is going.

4.7 Prefill/decode disaggregation (PD)

The idea. Run prefill on one pool of GPUs and decode on another. Prefill computes the KV cache; it is transferred to a decode worker which runs the generation loop.

Why. The two phases want different things (§4.2). Disaggregation lets you: - Size the pools independently to match your actual input:output ratio. - Eliminate prefill-induced ITL jitter entirely — decode workers only decode. - Use different hardware per phase: high-FLOP GPUs for prefill, high-bandwidth/high-memory GPUs for decode. (This is where H20's 4 TB/s and low FLOPs suddenly make sense as a decode-tier part.) - Tune batch size and parallelism per phase.

What it costs. The KV cache must cross the network. For Llama-70B at 4k tokens that's ~1.3 GB in FP16 — about 26 ms on a 400 Gb/s link, and it consumes NIC bandwidth you may need elsewhere. For MLA models the transfer is ~55x smaller, which is a large part of why disaggregation is so attractive for DeepSeek-class models specifically.

Where it stands (SGLang 0.5.12, May 2026). PD is a first-class runtime feature rather than an external orchestration layer. A GPU staging buffer accumulates KV on the prefill side and streams it block-by-block to the decode node, overlapping transfer with the remainder of prefill. A decode-side radix cache organizes received KV so that prefix matching still works on the decode tier — two requests sharing 2,000 tokens of prefix transfer that span once. Transport backends include NIXL (NVIDIA), Mooncake, and MORI-IO (AMD).

When not to. If your prefills are short, your fleet is small, or your SLO has slack, co-located serving with chunked prefill is simpler, has no transfer cost, and one team can operate it. Disaggregation is a scale technique; proposing it for a 4-GPU deployment is over-engineering and interviewers notice.

EPD disaggregation extends the idea to vision-language models with a third tier: encoder servers running the ViT, then prefill, then decode. Vision encoders scale badly under tensor parallelism (communication dominates), so EPD scales them horizontally instead — distributing images across independent encoder replicas — and ships embeddings over ZMQ or RDMA. Published results on Qwen3-VL-235B showed 6–8x lower TTFT at low QPS and roughly 2x throughput versus co-located, at the cost of extra GPUs and a third tier to operate.

4.8 The KV cache hierarchy

GPU HBM is small and expensive. A hierarchical cache (HiCache) tiers it:

GPU HBM        →  CPU DRAM     →  local NVMe   →  remote/object store
~3–8 TB/s         ~64 GB/s PCIe    ~3–14 GB/s      ~1–10 GB/s

The economics only work when fetching is cheaper than recomputing. Compare: re-prefilling 4k tokens on 70B costs ~90 ms of GPU time; fetching 1.3 GB from CPU over PCIe at 64 GB/s costs ~20 ms. Fetch wins. From NVMe at 5 GB/s it's 260 ms — fetch loses unless the GPU is saturated with other work, in which case the opportunity cost of recompute changes the answer.

This is the reasoning an interviewer wants: not "we'll add a cache tier" but "the tier pays for itself when transfer time is below recompute time, here are both numbers, and the crossover moves with GPU load."

4.9 Speculative decoding

The problem. Single-stream decode is bandwidth bound: one token per full model read. At batch=1 you are using perhaps 1% of the GPU's FLOPs.

The mechanism. A cheap draft proposes k tokens; the target model verifies all k in one forward pass (a batch-of-k, which is nearly the same cost as batch-of-1 because you were bandwidth bound anyway). Accepted tokens are kept; the first rejection truncates. Output is mathematically identical to standard sampling — this is not an approximation, and saying so correctly is a credibility marker.

Variants: a separate small draft model; EAGLE / EAGLE-3, which predicts in the target's feature space with a tree of candidates and achieves high acceptance rates; MTP (multi-token prediction), where the model ships with its own extra prediction heads (DeepSeek-V3 does); and n-gram/lookup drafting for high-repetition workloads like code editing.

When it pays. Speculation trades FLOPs for latency. It is a large win at low batch size (idle FLOPs are free) and can be a net loss at high batch size, where you were already compute bound and the verification overhead plus rejected tokens cost real throughput. Sophisticated deployments therefore adapt speculation depth to current batch size, or run speculation only on latency-tier traffic.

Key metric: acceptance rate (average accepted tokens per draft round). Speedup ≈ acceptance length ÷ verification cost ratio. Below ~1.5 accepted tokens, most configurations stop being worth it.

4.10 Structured / constrained decoding

Force output to match a JSON schema, regex, or context-free grammar by masking logits to only permissible next tokens at each step.

The hard part is throughput. A naive implementation computes an allowed-token mask over a 128k vocabulary on the critical path of every decode step, which can dominate step time. Fast backends (xgrammar, and SGLang's compressed-FSM approach) precompute and cache the automaton, compress mask computation, and overlap it with GPU work.

Jump-forward decoding is the elegant trick: when the grammar makes the next several tokens deterministic — after {"name": the next characters are forced — emit them directly without running the model. Structured output workloads often get a large speedup from this alone.

Edges worth knowing: grammar compilation is expensive, so cache compiled grammars by schema hash; tokenizer boundaries mean the "obvious" character-level automaton needs care to map onto token space; constrained decoding changes the output distribution relative to unconstrained sampling, which matters for evaluation.

4.11 Multi-LoRA

Serve many fine-tuned adapters against one base model. Adapters are small (tens to hundreds of MB), so hundreds can live in GPU or host memory while the base weights are shared. Batches can mix requests using different adapters, with batched grouped-GEMM kernels applying per-request adapter weights.

The system consequences: adapter identity becomes part of the KV cache key (prefix sharing across different adapters is invalid); routing prefers nodes with the adapter already resident; adapter swap latency becomes a cold-start problem in miniature; and a long tail of rarely-used adapters can thrash.

4.12 Quantization

Format	Weight bytes	Typical use
BF16/FP16	2	Baseline, training
FP8 (E4M3)	1	The production default on Hopper/Blackwell — near-lossless, hardware-accelerated
INT8 / W8A8	1	Older hardware
INT4 / AWQ / GPTQ	0.5	Weight-only; helps memory-bound decode, doesn't add FLOPs
FP4 / NVFP4	0.5	Blackwell-native, both weights and activations

Two distinctions that come up constantly:

Weight-only vs. weight-and-activation. Weight-only INT4 halves the bytes you read per decode step — a direct win on the bandwidth-bound path — but activations are still BF16, so it does not increase your prefill FLOP ceiling. FP8/FP4 with quantized activations accelerate both.

KV cache quantization is separate from weight quantization and is often the higher-leverage one. FP8 KV halves your KV footprint, which doubles concurrency, which nearly doubles decode throughput. The accuracy cost is usually small but must be measured, especially at long context.

4.13 The router

A fleet of SGLang replicas needs a front door that is not round-robin, because round-robin destroys prefix cache locality. SGLang's router (sgl-router) does cache-aware load balancing: it approximates which replica holds the longest matching prefix for an incoming request and prefers that replica, while falling back to load-based routing when the preferred node is overloaded.

The tension it manages is the central one in fleet design: affinity concentrates load, balance destroys cache hits. Real policies blend them — route to the best cache match unless that node's queue depth exceeds a threshold. It also handles PD orchestration (pairing prefill and decode workers), health checking, and draining for rollout.

4.14 RL integration: weight updates

In RLHF/RLVR loops, SGLang is the rollout engine: it generates trajectories, a trainer updates weights, and the new weights must reach every inference worker before the next iteration. This is a serving problem with an unusual requirement — the weights change every few minutes.

The APIs to know by name:

update_weights_from_tensor — trainer and rollout engine are co-located on the same GPUs; weights are handed over via CUDA IPC handles, no copy over the wire.
update_weights_from_distributed — trainer and rollout are separate; weights broadcast over NCCL through a shared process group.
update_weights_from_disk — checkpoint-mediated. Simple, slow, and the right answer only for infrequent updates.

Why this is hard at scale. NCCL broadcast from one source rank is redundant (the same bytes traverse the fabric many times), leaves most trainer ranks idle, and is rigid (a new engine instance can't join an existing group). Recent work replaces it with RDMA-based peer-to-peer transfer: every trainer rank sends its own shards directly to the ranks that need them, using one-sided RDMA with kernel bypass, with CPU-resident replicas on the source side to avoid burning VRAM. Published figures show the difference clearly:

Model	Params	Nodes	NCCL broadcast	RDMA P2P	Speedup
Qwen3-235B	235B	8	10.7 s	3.2 s	3.4x
DeepSeek-V3.2	744B	16	58.3 s	8.5 s	6.9x
Kimi-K2	1T	32	53.3 s	7.2 s	7.4x

Every second of weight sync is a second the entire cluster — trainers and rollout workers — is idle. At 1T scale, going from 53 s to 7 s on a 3-minute iteration is roughly a 25% throughput improvement on the whole cluster. That framing — sync time as a tax on total cluster throughput — is the one to use in an interview.

One more consequence, easy to miss and frequently probed: when weights change, every cached KV computed under the old weights is invalid. The prefix cache must be flushed on each update, which means RL rollout workers lose their cache every iteration and pay full prefill on the next one. Designs that keep prompts stable across iterations (so the flush cost is bounded) or that overlap re-prefill with sync are where the interesting engineering lives.

4.15 A checklist of mechanisms

If you can give the four sentences — what it does, why it's needed, what it costs, how it fails — for each of these, you are equipped for any scenario in this book.

continuous batching · chunked prefill · RadixAttention / prefix cache
paged KV + preemption · CUDA graphs · overlapped scheduler
TP · DP attention · EP · PP · EPLB · all-to-all overlap
PD disaggregation · EPD · KV transfer backends
hierarchical KV cache · KV quantization
speculative decoding (EAGLE / MTP) · acceptance rate
constrained decoding · jump-forward · grammar cache
multi-LoRA · quantization formats
cache-aware routing · admission control
weight update paths (tensor / distributed / disk / P2P RDMA)

Next: Part I — Core Inference Serving