Distributed Systems · core
Distributed Failure Recovery
Partial failure, timeouts, retries, deduplication, fencing, repair, anti-entropy, and disaster recovery.
Mental model
In a distributed system, failure is ambiguous. Recovery needs bounded retries, unique operation identity, stale-writer fencing, repair mechanisms, and tested restore objectives. Scope: this card owns correctness under partial failure — operation identity, fencing a stale writer, repair and anti-entropy, and tested restore objectives. What target you are recovering TO is `reliability-fault-tolerance`; the caller-side backoff policy is `retries-and-circuit-breakers`.
How to study Distributed Failure Recovery
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use The Network is Reliable (Bailis & Kingsbury), Gray Failure: The Achilles' Heel of Cloud-Scale Systems (HotOS '17), Partitions for Everyone! — Kyle Kingsbury (Strange Loop 2013) to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Distributed Failure Recovery with Distributed Workflows & Temporal. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Design exercise: Distributed Failure Recovery and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Learn from primary sources
- The Network is Reliable (Bailis & Kingsbury) (doc)
- Gray Failure: The Achilles' Heel of Cloud-Scale Systems (HotOS '17) (doc)
- Partitions for Everyone! — Kyle Kingsbury (Strange Loop 2013) (video)
- Simple Testing Can Prevent Most Critical Failures (OSDI '14) (paper)
- Jepsen: etcd and Consul (Kyle Kingsbury) (article)
- Site Reliability Engineering — Addressing Cascading Failures (doc)
- Jepsen Analyses (doc)
Practice and explain it back
Design exercise: Distributed Failure Recovery
Partial failure, timeouts, retries, deduplication, fencing, repair, anti-entropy, and disaster recovery. Implement designOutline() returning non-empty values for: failureDetection, duplicateControl, repairPath. Each value must name a concrete mechanism or decision.
Expected evidence: A design outline with failureDetection, duplicateControl, repairPath plus an explicit failure mode or trade-off.
Open the interactive drill →Review prompts
- What is a fencing token, and which failure does it stop that a lease timeout alone does not?
Build evidence
Synthesize: Distributed Systems
Reason about coordination, data placement, logs, durable workflows, consistency, and recovery under partial failure. Produce one working system, benchmark, or evidence-backed design that integrates the path.
- Implements or precisely models the core mechanisms from all three milestones
- Includes at least one injected failure or adversarial case and demonstrates recovery
- Reports quality, latency, resource, reliability, or usability measurements relevant to the domain
- Ships a concise architecture note explaining decisions, trade-offs, and remaining risks