core practice · system design prompt
Design exercise: Distributed Failure Recovery
Partial failure, timeouts, retries, deduplication, fencing, repair, anti-entropy, and disaster recovery. Implement designOutline() returning non-empty values for: failureDetection, duplicateControl, repairPath. Each value must name a concrete mechanism or decision.
The mechanism
In a distributed system, failure is ambiguous. Recovery needs bounded retries, unique operation identity, stale-writer fencing, repair mechanisms, and tested restore objectives. Scope: this card owns correctness under partial failure — operation identity, fencing a stale writer, repair and anti-entropy, and tested restore objectives. What target you are recovering TO is `reliability-fault-tolerance`; the caller-side backoff policy is `retries-and-circuit-breakers`.
Expected evidence
A design outline with failureDetection, duplicateControl, repairPath plus an explicit failure mode or trade-off.
Try this drill in the practice workspace →
Hints when you need them
- Name the failure mode before selecting the mechanism.
Check the sources
- The Network is Reliable (Bailis & Kingsbury)
- Gray Failure: The Achilles' Heel of Cloud-Scale Systems (HotOS '17)
- Partitions for Everyone! — Kyle Kingsbury (Strange Loop 2013)
- Simple Testing Can Prevent Most Critical Failures (OSDI '14)
- Jepsen: etcd and Consul (Kyle Kingsbury)
- Site Reliability Engineering — Addressing Cascading Failures
- Jepsen Analyses