advanced practice · coding
Evaluate and ablate an agent harness
Implement compareHarness(baseline, candidate, limits). Each run has tasksPassed, tasksTotal, costUsd, minutes, interventions, and unsafeActions. Return successRateDelta, costDelta, timeDelta, and release. release is true only when candidate success rate does not regress, unsafeActions is zero, costUsd and minutes are within limits, and interventions do not exceed baseline.
The mechanism
A harness is a policy that transforms a model and environment into task outcomes. Evaluate the whole policy on frozen tasks, record success, regressions, cost, time, interventions, and unsafe actions, then ablate one harness component at a time to learn which constraints actually carry performance.
Expected evidence
A multi-objective harness comparison that prevents pass-rate-only releases.
Try this drill in the practice workspace →
Hints when you need them
- Normalize pass count by task count before comparing suites.
- Unsafe actions are a hard gate; cost, time, and intervention remain first-class even when success improves.
Watch for these mistakes
- Measuring only final pass rate and ignoring cost, retries, intervention, or unsafe actions
- Changing model, tools, prompts, and evaluator together so no causal conclusion is possible
- Using the same model-generated rubric and answer as both system output and ground truth