Agent Systems · advanced
Harness Evaluation
Task suites, outcome rubrics, trajectory metrics, cost and latency budgets, ablations, and regression gates for the harness rather than the model alone.
Mental model
A harness is a policy that transforms a model and environment into task outcomes. Evaluate the whole policy on frozen tasks, record success, regressions, cost, time, interventions, and unsafe actions, then ablate one harness component at a time to learn which constraints actually carry performance.
How to study Harness Evaluation
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Anthropic — Harness design for long-running application development, OpenAI — Harness engineering to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Harness Evaluation with AI Regression Testing, Tool-use Evaluations, Agent Observability. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Evaluate and ablate an agent harness and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Coding-agent benchmarks, release gates, tool-policy experiments, prompt and instruction changes, cost controls, and harness simplification.
Common mistakes
- Measuring only final pass rate and ignoring cost, retries, intervention, or unsafe actions
- Changing model, tools, prompts, and evaluator together so no causal conclusion is possible
- Using the same model-generated rubric and answer as both system output and ground truth
Learn from primary sources
Practice and explain it back
Evaluate and ablate an agent harness
Implement compareHarness(baseline, candidate, limits). Each run has tasksPassed, tasksTotal, costUsd, minutes, interventions, and unsafeActions. Return successRateDelta, costDelta, timeDelta, and release. release is true only when candidate success rate does not regress, unsafeActions is zero, costUsd and minutes are within limits, and interventions do not exceed baseline.
Expected evidence: A multi-objective harness comparison that prevents pass-rate-only releases.
Open the interactive drill →Review prompts
- A new harness raises task pass rate from 80% to 86% but doubles cost and triples human interventions. How should the team evaluate the change and discover which harness component caused it?
Build evidence
Automate and Evaluate a Maker-Checker Loop
Automate a bounded loop in which a maker attempts one task, an independent checker evaluates explicit acceptance criteria, failures return actionable evidence, and the orchestrator stops on success, escalation, cancellation, or budget exhaustion. Compare it with a baseline harness on a frozen task suite.
- Maker and checker roles have separate inputs and the checker ignores unsupported self-reported success
- The loop has explicit goal, task and retry budgets, durable state, and semantic termination criteria
- A frozen evaluation suite measures success, unsafe actions, cost, latency, retries, and human interventions
- At least one component ablation isolates whether instructions, tools, verification, or retry policy caused the observed change
- Release or rejection follows a declared multi-objective gate rather than pass rate alone