# Evaluation & AI Reliability

> LLM and agent evaluations, regression gates, failure detection, tracing, verification, human review, and quality economics.

Canonical: https://learn.significanthobbies.com/curriculum/tracks/ai-reliability

LLM and agent evaluations, regression gates, failure detection, tracing, verification, human review, and quality economics.

This track contains 14 connected concepts. Mastery means explaining each
mechanism, predicting its failure modes, and supporting decisions with code,
measurements, or a reviewable design artifact.

## Roadmaps

- [12-Week Evaluation & AI Reliability](https://learn.significanthobbies.com/curriculum/roadmaps/ai-reliability-12w) — Build an evidence-backed evaluation and observability system for models, tools, and agents.
- [Harness Engineering](https://learn.significanthobbies.com/curriculum/roadmaps/harness-engineering) — Design, build, verify, and evaluate the environment around a coding agent so it can make reliable progress across bounded and long-running tasks.

## Concepts

- [Prompt & Version Logging](https://learn.significanthobbies.com/curriculum/concepts/prompt-versioning) (core) — Treating prompts as versioned artifacts with logged inputs/outputs.
- [LLM Evals](https://learn.significanthobbies.com/curriculum/concepts/llm-evals) (core) — Measuring LLM output quality with datasets, graders, and LLM-as-judge.
- [Model Evaluation](https://learn.significanthobbies.com/curriculum/concepts/ml-evaluation) (core) — Held-out loss, baselines, hallucination, leakage.
- [Coding-agent Benchmarks](https://learn.significanthobbies.com/curriculum/concepts/coding-agent-benchmarks) (core) — Issue resolution tasks, repository setup, patch grading, test-based scoring, contamination, and benchmark validity.
- [Tool-use Evaluations](https://learn.significanthobbies.com/curriculum/concepts/tool-use-evaluations) (core) — Tool selection, argument correctness, sequencing, recovery, side-effect safety, and end-state verification.
- [AI Regression Testing](https://learn.significanthobbies.com/curriculum/concepts/ai-regression-testing) (core) — Frozen eval sets, golden cases, rubric versions, stochastic thresholds, canaries, and release gates.
- [Hallucination & Failure Detection](https://learn.significanthobbies.com/curriculum/concepts/hallucination-failure-detection) (core) — Unsupported claims, citations, abstention, tool errors, constraint violations, uncertainty, and escalation.
- [Agent Observability](https://learn.significanthobbies.com/curriculum/concepts/agent-observability) (core) — Runs, steps, prompts, model calls, tool calls, tokens, costs, errors, state changes, and outcome metrics.
- [Tracing & Replay](https://learn.significanthobbies.com/curriculum/concepts/tracing-replay) (core) — Deterministic inputs, event logs, snapshots, prompt/model versions, tool fixtures, and counterfactual re-execution.
- [Evidence-backed Verification](https://learn.significanthobbies.com/curriculum/concepts/evidence-backed-verification) (core) — Claims, source provenance, executable checks, screenshots, diffs, test outputs, and acceptance criteria.
- [Human Review Systems](https://learn.significanthobbies.com/curriculum/concepts/human-review-systems) (core) — Review queues, risk routing, disagreement, calibration, escalation, auditability, and learning from corrections.
- [Quality, Cost & Latency Measurement](https://learn.significanthobbies.com/curriculum/concepts/quality-cost-latency-measurement) (core) — Task success, calibrated quality, token and tool cost, latency distributions, reliability, and Pareto frontiers.
- [Independent Agent Verification](https://learn.significanthobbies.com/curriculum/concepts/independent-agent-verification) (advanced) — Maker-checker role separation, independent evidence gathering, adversarial review, and acceptance gates that do not trust the producer's self-report.
- [Harness Evaluation](https://learn.significanthobbies.com/curriculum/concepts/harness-evaluation) (advanced) — Task suites, outcome rubrics, trajectory metrics, cost and latency budgets, ablations, and regression gates for the harness rather than the model alone.