core practice · system design prompt
Design exercise: vLLM & Inference Engines
Request scheduling, model execution, memory management, distributed serving, APIs, and engine architecture. Implement designOutline() returning non-empty values for: requestScheduler, memoryManager, executionBackend. Each value must name a concrete mechanism or decision.
The mechanism
An inference engine is a runtime: it schedules requests, owns model and KV memory, dispatches kernels, exposes metrics, and enforces admission control.
Expected evidence
A design outline with requestScheduler, memoryManager, executionBackend plus an explicit failure mode or trade-off.
Try this drill in the practice workspace →
Hints when you need them
- Name the failure mode before selecting the mechanism.
Check the sources
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI '22)
- SGLang: Efficient Execution of Structured Language Model Programs
- Stanford CS336 — Lecture 10: Inference (Language Modeling from Scratch, Spring 2025)
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Fast and Expressive LLM Inference with RadixAttention and SGLang (LMSYS)
- vLLM Documentation
- Learn Inference — interactive guide to inference engineering
- Inference Engineering (Philip Kiely, Baseten)