core practice · system design prompt

Design exercise: Inference Cost & Latency Optimization

Time to first token, inter-token latency, throughput, tail latency, utilization, quality, and cost per request. Implement designOutline() returning non-empty values for: workload, latencyBreakdown, costModel. Each value must name a concrete mechanism or decision.

The mechanism

Serving optimization is a constrained frontier: quality, TTFT, token latency, throughput, availability, and cost must be measured on the same representative workload.

Study Inference Cost & Latency Optimization

Expected evidence

A design outline with workload, latencyBreakdown, costModel plus an explicit failure mode or trade-off.

Try this drill in the practice workspace →

Hints when you need them

Name the failure mode before selecting the mechanism.

Check the sources

Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Fast Inference from Transformers via Speculative Decoding
MLPerf Inference Benchmark
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI '24)
Hugging Face — LLM Inference Optimization

Browse all practice briefs