Learning track · 12 concepts
Inference & Serving
Inference engines, batching, KV caches, attention kernels, decoding, routing, hardware utilization, and serving economics.
What mastery looks like
This track contains 12 connected concepts rather than an unordered reading list. Mastery means you can move from vocabulary to mechanisms, predict how the system behaves under pressure, and support a design decision with code, measurements, or a failure-recovery exercise. For Inference & Serving, use the track description as the boundary: learn enough detail to reason clearly about inference engines, batching, kv caches, attention kernels, decoding, routing, hardware utilization, and serving economics.
A useful explanation names the state involved, the operation that changes it, the resource or safety constraint, and the observable signal that tells you whether the mechanism works. Avoid stopping at product names. Compare at least two approaches, state what each optimizes, and identify what breaks first as scale, concurrency, latency, or uncertainty increases.
Suggested study sequence
Start with the core concepts at the top of the list and write a one-paragraph mechanism note for each. Continue through the core concepts by alternating explanation with an executable drill. Treat Browser ML Runtime, WebGPU Compute as integration work: they should combine earlier mechanisms rather than introduce disconnected facts.
At the end of each session, record one decision you can now make, one failure mode you can now predict, and one unanswered question. Revisit that question through the linked primary sources, then prove the answer in the Playground or a real repository. The track is complete when you can transfer the reasoning to an unfamiliar system, not when every page has been opened.
Roadmaps
Concepts in this track
core
Model Routing
Sending each request to the cheapest model that can handle it.
advanced
Browser ML Runtime
Web Workers, WASM, OPFS, TypedArrays.
advanced
WebGPU Compute
WGSL, compute kernels, matmul, CPU parity.
core
vLLM & Inference Engines
Request scheduling, model execution, memory management, distributed serving, APIs, and engine architecture.
core
Continuous Batching
Iteration-level scheduling, dynamic admission, prefill/decode interleaving, chunked prefill, and fairness.
core
KV Caching & PagedAttention
Attention-state reuse, KV memory sizing, paging, fragmentation, prefix caching, eviction, and multi-tenant pressure.
core
FlashAttention & Attention Kernels
IO-aware tiling, fused kernels, SRAM/HBM movement, numerical stability, and hardware-aware attention.
core
Speculative Decoding
Draft models, token verification, acceptance rates, tree speculation, latency, and quality preservation.
core
GPU Utilization
Compute occupancy, memory bandwidth, kernel launch overhead, tensor parallelism, profiling, and saturation.
core
Inference Cost & Latency Optimization
Time to first token, inter-token latency, throughput, tail latency, utilization, quality, and cost per request.
core
Local & On-device Inference
llama.cpp, WebGPU, mobile accelerators, model formats, privacy, offline operation, and constrained memory.
core
Inference Hardware
GPUs, TPUs, NPUs, CPUs, memory bandwidth, interconnects, topology, precision support, and deployment fit.