11.1 · L5 — Name the four phases of an RL iteration and say which dominates.

Generate (rollout), score, train, sync weights. Rollout is typically 60–80% of wall clock, which makes RL performance primarily an inference optimization problem. Everything is idle during weight sync.

11.2 · L5 — Why is rollout slower than training over the same tokens?

Training processes all tokens in one parallel pass — dense, high-MFU work. Generation is autoregressive: one token per sequence per step, thousands of sequential steps for long chain-of-thought, bandwidth bound at every step. Same tokens, very different structure.

11.3 · L5 — Why must the KV cache be flushed on every weight update?

Cached KV was computed under the old weights. Reusing it mixes two models' representations inside one forward pass — silently wrong, no error, plausible-looking loss curve. Cache identity must include the policy version. It's a correctness requirement, not an optimization.

11.4 · L6 — Derive the rollout efficiency formula.

With N sequences all starting concurrently, wall-clock is max(L_i) steps while useful work is Σ L_i = N·E[L]. So efficiency = E[L] / max(L). At E[L]=4,000 and a 16,384 cap, that's 24% — three-quarters of the rollout phase is near-idle.

11.5 · L6 — Why doesn't longest-first scheduling fix the rollout tail?

Because in RL the sequences typically all start at step 0 — KV capacity exceeds the sequence count, so there's no queue to reorder. The last sequence finishes at max(L) regardless of start order. Longest-first only helps when you're capacity-constrained, so check the regime first. (You drift into that regime as the model learns to generate longer.)

11.6 · L6 — Which straggler fixes bias the training data and which don't?

Bias: truncating at a length cap (removes the longest reasoning traces — exactly wrong for a reasoning model), and over-generate-and-take-first-K (the first to complete are the short ones). Approximately unbiased: partial rollouts / continuation (save state, resume next iteration — all sequences eventually finish), and backfill with the next iteration's prompts.

11.7 · L6 — What's the fully-correct answer to the rollout tail?

Backfill: as the batch drains, fill freed capacity with the next iteration's prompts. Efficiency → ~100% because there's no shrinking batch and no barrier. This is asynchronous RL — it requires policy-version tagging on trajectories, weight updates without draining the engine, and a trainer that consumes a stream. It dissolves the problem rather than mitigating it.

11.8 · L5 — Name the three weight-update paths.

update_weights_from_tensor (co-located; CUDA IPC handles, no copy over the wire), update_weights_from_distributed (separate processes; NCCL broadcast through a shared group), and update_weights_from_disk (checkpoint-mediated; simple, slow, right only for infrequent updates).

11.9 · L6 — Why is NCCL broadcast a poor fit for weight sync?

Three reasons plus one. Redundancy — every inference rank receives the full model when it needs only its shard. Idle ranks — only the source transmits, leaving most of the fabric unused. Rigidity — fixed communicator membership, so a new rollout replica can't join without a restart. And underneath: it's a resharding problem, since the trainer and the inference engine partition the model differently, which is many-to-many, not one-to-many.

11.10 · L6 — What does RDMA P2P weight transfer buy, with numbers?

Every trainer rank sends its shards directly to the ranks that need them, one-sided, zero-copy, kernel-bypass, using the whole fabric. Published: Qwen3-235B on 8 nodes 10.7 s → 3.2 s (3.4x); DeepSeek-V3.2 744B on 16 nodes 58.3 s → 8.5 s (6.9x); Kimi-K2 1T on 32 nodes 53.3 s → 7.2 s (7.4x). The speedup grows with scale, because broadcast's inefficiency is proportional to the fabric it fails to use.

11.11 · L6 — Frame weight sync cost the way you'd present it to leadership.

Not as latency — as a tax on total cluster throughput. 53 s of a 180 s iteration is 29% of a 256-GPU cluster idle, permanently. That's roughly \$1.6M/year and, more importantly, 29% fewer experiments per week. Fixing it is equivalent to buying 70 GPUs, for a few weeks of engineering.

11.12 · L6 — Co-located vs disaggregated train/rollout: give the one non-obvious argument.

The phase ratio drifts. As a reasoning model learns, response lengths grow, and rollout wall-clock grows faster than training's. A split that's optimal at 62:38 early becomes 77:23 later. A disaggregated deployment with a fixed split sees its training pool go progressively idle; co-location adapts automatically because whichever phase needs more time simply takes it on the whole cluster.

11.13 · L6 — RL isn't learning at cluster scale but works in a single process. Prime suspect?

The logprob mismatch. π_old from the rollout engine and π_new from the trainer are computed on different numerical paths — different precision (FP8 vs BF16), different kernels, different TP reduction trees, batch-dependent kernel selection. The importance ratio that should be exactly 1 at zero staleness isn't, and clipping discards real learning signal. The single-process detail is the tell: that's the configuration where the numerics match.

11.14 · L6 — Give the ten-minute test for that.

Take one iteration's trajectories with trainer weights equal to rollout weights, recompute log π_new for every sampled token, and histogram r = exp(log π_new − log π_old). Expect a spike at exactly 1.0. Report the fraction outside the clip range at zero staleness (should be ~0; 5%+ is the bug) and mean absolute log-ratio (~1e-7 is rounding; ≥1e-3 means a structural path difference). Make it a standing metric, not a debugging step.

11.15 · L6 — Two fixes for the mismatch, and why the obvious one is insufficient.

Carry the rollout engine's own logprobs in the trajectory as π_old — never recompute them in the trainer, since recomputation introduces the mismatch. That's necessary but not sufficient: the trainer's π_new is still on a path the rollout engine would never produce. Reduce the gap with matched precision, matched parallelism for the logprob computation, and deterministic/batch-invariant kernels on that path.

11.16 · L5 — Pretraining FLOPs: give the formula and use it.

6 × active_params × tokens (2 forward, 4 backward). For a 400B MoE with 40B active on 15T tokens: 6 × 40e9 × 15e12 = 3.6e24. At 8,192 H100 × 990 TF × 0.40 MFU = 3.24e18 FLOP/s → ≈ 12.9 days at perfect availability, ~14.5 in practice.

11.17 · L6 — At 8,192 GPUs, how often do you lose a node, and what follows?

cluster MTBF ≈ per-GPU MTBF / N ≈ 50,000 / 8,192 ≈ 6.1 hours — about 57 failures over a 14.5-day run. That makes failure handling a first-order design input: async checkpointing every 10–15 minutes, hot spares pre-allocated, checkpoint shards on local NVMe for parallel reload, and elastic restart replacing only the failed rank. Optimize restart time harder than checkpoint frequency — a 20-minute restart across 57 failures costs more than the redone work.

11.18 · L6 — What's the data-pipeline throughput requirement for that run?

3.24e18 FLOP/s ÷ (6 × 40e9) ≈ 13.5 M tokens/second. As uint16 token IDs that's ~27 GB/s sustained for two weeks. Achievable if the 30 TB corpus is staged on cluster-local NVMe and read in parallel (a few MB/s per rank); a bad bet if you stream from shared object storage. And the pipeline must be finished before the run — you can't filter and tokenize petabytes inline.

Next: Drill Set 12 — Operations, Cost & Incidents