# Design exercise: Human Review Systems

> Review queues, risk routing, disagreement, calibration, escalation, auditability, and learning from corrections. Implement designOutline() returning non-empty values for: riskRouting, reviewRubric, feedbackLoop. Each value must name a concrete mechanism or decision.

Canonical: https://learn.significanthobbies.com/curriculum/drills/practice-human-review-systems

Review queues, risk routing, disagreement, calibration, escalation, auditability, and learning from corrections. Implement designOutline() returning non-empty values for: riskRouting, reviewRubric, feedbackLoop. Each value must name a concrete mechanism or decision.

## The mechanism

Human review is a risk-control system. Route uncertain or high-impact cases, give reviewers evidence and clear rubrics, measure agreement, and feed corrections into evals.

## Expected evidence

A design outline with riskRouting, reviewRubric, feedbackLoop plus an explicit failure mode or trade-off.

## Hints

- Name the failure mode before selecting the mechanism.

## Sources and practice

- [People + AI Guidebook — Feedback + Control (Google PAIR)](https://pair.withgoogle.com/chapter/feedback-controls/)
- [People + AI Guidebook — Errors + Graceful Failure (Google PAIR)](https://pair.withgoogle.com/chapter/errors-failing/)
- [NIST AI Risk Management Framework 1.0 (NIST AI 100-1, PDF)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)
- [Study Human Review Systems](https://learn.significanthobbies.com/curriculum/concepts/human-review-systems)
- [Try the interactive drill](https://learn.significanthobbies.com/drills/practice-human-review-systems)
- [All practice briefs](https://learn.significanthobbies.com/curriculum/drills/)