core practice · implementation task
Build an LLM eval
Create a dataset of inputs with expected behavior and a grader (deterministic or LLM-as-judge). Run it as one command and score a prompt.
The mechanism
Evals are unit tests for non-deterministic systems. Fix a dataset of inputs with expected behavior, define a grader (exact match, rubric, or LLM-as-judge), and run it on every prompt/model change. Scope: this card owns constructing the grader — dataset design, rubric writing, and LLM-as-judge including its calibration and bias checks. Running those graders as a release gate is `ai-regression-testing`; grading an agent's actions rather than its text is `tool-use-evaluations`.
Expected evidence
A pass-rate score that changes when you change the prompt.
Try this drill in the practice workspace →
Hints when you need them
- Cover real failure modes, not just happy paths.
- Validate an LLM judge against human labels.
Watch for these mistakes
- Shipping prompt changes on vibes with no eval
- LLM-as-judge without checking the judge against humans
- Eval set that does not cover real failure modes