Learning track · 14 concepts
Evaluation & AI Reliability
LLM and agent evaluations, regression gates, failure detection, tracing, verification, human review, and quality economics.
What mastery looks like
This track contains 14 connected concepts rather than an unordered reading list. Mastery means you can move from vocabulary to mechanisms, predict how the system behaves under pressure, and support a design decision with code, measurements, or a failure-recovery exercise. For Evaluation & AI Reliability, use the track description as the boundary: learn enough detail to reason clearly about llm and agent evaluations, regression gates, failure detection, tracing, verification, human review, and quality economics.
A useful explanation names the state involved, the operation that changes it, the resource or safety constraint, and the observable signal that tells you whether the mechanism works. Avoid stopping at product names. Compare at least two approaches, state what each optimizes, and identify what breaks first as scale, concurrency, latency, or uncertainty increases.
Suggested study sequence
Start with the core concepts at the top of the list and write a one-paragraph mechanism note for each. Continue through the core concepts by alternating explanation with an executable drill. Treat Independent Agent Verification, Harness Evaluation as integration work: they should combine earlier mechanisms rather than introduce disconnected facts.
At the end of each session, record one decision you can now make, one failure mode you can now predict, and one unanswered question. Revisit that question through the linked primary sources, then prove the answer in the Playground or a real repository. The track is complete when you can transfer the reasoning to an unfamiliar system, not when every page has been opened.
Roadmaps
Concepts in this track
core
Prompt & Version Logging
Treating prompts as versioned artifacts with logged inputs/outputs.
core
LLM Evals
Measuring LLM output quality with datasets, graders, and LLM-as-judge.
core
Model Evaluation
Held-out loss, baselines, hallucination, leakage.
core
Coding-agent Benchmarks
Issue resolution tasks, repository setup, patch grading, test-based scoring, contamination, and benchmark validity.
core
Tool-use Evaluations
Tool selection, argument correctness, sequencing, recovery, side-effect safety, and end-state verification.
core
AI Regression Testing
Frozen eval sets, golden cases, rubric versions, stochastic thresholds, canaries, and release gates.
core
Hallucination & Failure Detection
Unsupported claims, citations, abstention, tool errors, constraint violations, uncertainty, and escalation.
core
Agent Observability
Runs, steps, prompts, model calls, tool calls, tokens, costs, errors, state changes, and outcome metrics.
core
Tracing & Replay
Deterministic inputs, event logs, snapshots, prompt/model versions, tool fixtures, and counterfactual re-execution.
core
Evidence-backed Verification
Claims, source provenance, executable checks, screenshots, diffs, test outputs, and acceptance criteria.
core
Human Review Systems
Review queues, risk routing, disagreement, calibration, escalation, auditability, and learning from corrections.
core
Quality, Cost & Latency Measurement
Task success, calibrated quality, token and tool cost, latency distributions, reliability, and Pareto frontiers.
advanced
Independent Agent Verification
Maker-checker role separation, independent evidence gathering, adversarial review, and acceptance gates that do not trust the producer's self-report.
advanced
Harness Evaluation
Task suites, outcome rubrics, trajectory metrics, cost and latency budgets, ablations, and regression gates for the harness rather than the model alone.