core practice · system design prompt
Design exercise: Continuous Batching
Iteration-level scheduling, dynamic admission, prefill/decode interleaving, chunked prefill, and fairness. Implement designOutline() returning non-empty values for: admissionPolicy, prefillDecode, fairness. Each value must name a concrete mechanism or decision.
The mechanism
Continuous batching changes the batch after each decoding step, filling freed slots without waiting for the slowest request. Throughput improves at the cost of scheduler complexity.
Expected evidence
A design outline with admissionPolicy, prefillDecode, fairness plus an explicit failure mode or trade-off.
Try this drill in the practice workspace →
Hints when you need them
- Name the failure mode before selecting the mechanism.
Check the sources
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI '24)
- USENIX OSDI '22 — Orca: A Distributed Serving System for Transformer-Based Generative Models
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI '22)
- SGLang v0.4: Zero-Overhead Batch Scheduler (LMSYS)
- vLLM: Easy, Fast, and Cheap LLM Serving