core practice · system design prompt
Design exercise: Local & On-device Inference
llama.cpp, WebGPU, mobile accelerators, model formats, privacy, offline operation, and constrained memory. Implement designOutline() returning non-empty values for: modelFormat, memoryBudget, deviceFallback. Each value must name a concrete mechanism or decision.
The mechanism
On-device inference trades cloud elasticity for privacy, offline latency, and strict memory/energy budgets. Model format, quantization, kernels, and thermal limits become product constraints. Scope: this card owns running outside the browser — llama.cpp, mobile accelerators, model formats, quantisation, and thermal and memory budgets. In-browser execution is `ml-browser-runtime`, and its GPU path is `ml-webgpu`.
Expected evidence
A design outline with modelFormat, memoryBudget, deviceFallback plus an explicit failure mode or trade-off.
Try this drill in the practice workspace →
Hints when you need them
- Name the failure mode before selecting the mechanism.
Check the sources
- WebLLM: A High-Performance In-Browser LLM Inference Engine
- USENIX ATC '25 — CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
- Deploy Machine Learning and AI Models On-device with Core ML (WWDC24)
- LLM in a flash: Efficient Large Language Model Inference with Limited Memory
- llama.cpp