Backend · core

Retries & Circuit Breakers

Backoff, jitter, budgets, and breakers — retrying without turning a blip into an outage.

backendreliability

Mental model

A naive retry is a load multiplier aimed at a service that is already struggling, and synchronised clients retrying on the same schedule produce a thundering herd. Exponential backoff spreads them in time and jitter breaks the synchronisation; a retry budget caps the amplification. A circuit breaker is the admission that retrying is now pointless: after enough failures it fails fast, gives the dependency room to recover, then lets a trial request through. Retries are only safe at all if the operation is idempotent.

How to study Retries & Circuit Breakers

Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Timeouts, retries and backoff with jitter (AWS) to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.

Next, compare Retries & Circuit Breakers with Rate Limiting, Retries & DLQ. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Exponential backoff with full jitter and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.

Proof of understanding

Explain the mechanism from first principles and identify the state it reads or changes.
Give one situation where the concept is the right choice and one where it is not.
Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.

Where it matters

The difference between a dependency blip and a cascading outage, and the first thing a postmortem examines.

Common mistakes

Retrying non-idempotent writes, so a timeout becomes a double charge
Backoff without jitter, which keeps every client synchronised and reproduces the herd
Retrying at several layers at once — three layers of three retries is twenty-seven requests
Retrying a 4xx: the request is wrong, and repeating it will not make it right

Learn from primary sources

Timeouts, retries and backoff with jitter (AWS) (doc)

Practice and explain it back

Exponential backoff with full jitter

Implement backoffDelays(attempts, baseMs, capMs, rand) returning the delay for each attempt using capped exponential backoff with FULL jitter: delay = rand() * min(capMs, baseMs * 2^attempt), floored to an integer. `rand` is a supplied function returning [0,1) so the result is deterministic under test. Attempt numbering starts at 0.

Expected evidence: backoffDelays(4, 100, 1000, () => 1) -> [100, 200, 400, 800]

Open the interactive drill →

Review prompts

A dependency slows down, clients retry three times at each of three layers, and the outage gets worse. Trace the amplification and name the two mechanisms that stop it.