core practice · system design prompt
Design exercise: Quality, Cost & Latency Measurement
Task success, calibrated quality, token and tool cost, latency distributions, reliability, and Pareto frontiers. Implement designOutline() returning non-empty values for: qualityMetric, costMetric, latencyMetric. Each value must name a concrete mechanism or decision.
The mechanism
No AI metric stands alone. Compare candidate systems on the same workload and plot quality, cost, latency, and failure rate together. Scope: this card owns comparing candidate SYSTEMS on one workload — the Pareto frontier across quality, token and tool cost, latency distribution, and failure rate. Scoring an individual output is `llm-evals`; gating a release on the result is `ai-regression-testing`.
Expected evidence
A design outline with qualityMetric, costMetric, latencyMetric plus an explicit failure mode or trade-off.
Try this drill in the practice workspace →
Hints when you need them
- Name the failure mode before selecting the mechanism.