core practice · system design prompt
Design exercise: AI Regression Testing
Frozen eval sets, golden cases, rubric versions, stochastic thresholds, canaries, and release gates. Implement designOutline() returning non-empty values for: evalSet, graderVersion, releaseThreshold. Each value must name a concrete mechanism or decision.
The mechanism
AI regression tests compare distributions and task outcomes, not exact strings. Freeze representative cases, version graders, repeat stochastic trials, and gate meaningful deltas. Scope: this card owns the CI application — frozen sets, versioned rubrics, repeat trials against stochastic output, and the threshold that blocks a release. How a grader is built lives in `llm-evals`; what to measure beyond quality lives in `quality-cost-latency-measurement`.
Expected evidence
A design outline with evalSet, graderVersion, releaseThreshold plus an explicit failure mode or trade-off.
Try this drill in the practice workspace →
Hints when you need them
- Name the failure mode before selecting the mechanism.