# Design exercise: AI Regression Testing

> Frozen eval sets, golden cases, rubric versions, stochastic thresholds, canaries, and release gates. Implement designOutline() returning non-empty values for: evalSet, graderVersion, releaseThreshold. Each value must name a concrete mechanism or decision.

Canonical: https://learn.significanthobbies.com/curriculum/drills/practice-ai-regression-testing

Frozen eval sets, golden cases, rubric versions, stochastic thresholds, canaries, and release gates. Implement designOutline() returning non-empty values for: evalSet, graderVersion, releaseThreshold. Each value must name a concrete mechanism or decision.

## The mechanism

AI regression tests compare distributions and task outcomes, not exact strings. Freeze representative cases, version graders, repeat stochastic trials, and gate meaningful deltas. Scope: this card owns the CI application — frozen sets, versioned rubrics, repeat trials against stochastic output, and the threshold that blocks a release. How a grader is built lives in `llm-evals`; what to measure beyond quality lives in `quality-cost-latency-measurement`.

## Expected evidence

A design outline with evalSet, graderVersion, releaseThreshold plus an explicit failure mode or trade-off.

## Hints

- Name the failure mode before selecting the mechanism.

## Sources and practice

- [LLM Evals: Everything You Need to Know (Hamel Husain)](https://hamel.dev/blog/posts/evals-faq/)
- [Demystifying Evals for AI Agents (Anthropic)](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
- [Evaluating the Effectiveness of LLM-Evaluators (Eugene Yan)](https://eugeneyan.com/writing/llm-evaluators/)
- [A Survey on LLM-as-a-Judge](https://arxiv.org/abs/2411.15594)
- [Using LLM-as-a-Judge For Evaluation: A Complete Guide (Hamel Husain)](https://hamel.dev/blog/posts/llm-judge/)
- [OpenAI Evals Documentation](https://platform.openai.com/docs/guides/evals)
- [Study AI Regression Testing](https://learn.significanthobbies.com/curriculum/concepts/ai-regression-testing)
- [Try the interactive drill](https://learn.significanthobbies.com/drills/practice-ai-regression-testing)
- [All practice briefs](https://learn.significanthobbies.com/curriculum/drills/)