Infrastructure & Platforms · core
Retries & DLQ
Backoff + jitter, poison messages, dead-letter queues.
Mental model
Retries fix transient failures; backoff with jitter stops a retry storm; a dead-letter queue catches messages that will never succeed so they stop blocking the queue and become debuggable. Scope: this card owns one mechanism, queue-side: poison messages, the dead-letter queue, and getting a stuck consumer moving again. Caller-side backoff, jitter and breakers are `retries-and-circuit-breakers`; the reasoning about ambiguous failure is `distributed-failure-recovery`.
How to study Retries & DLQ
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Timeouts, retries and backoff with jitter (AWS) to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Retries & DLQ with Message Queues, Background Jobs. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Implement backoff with jitter and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
SQS/Kafka consumers, every background job system.
Common mistakes
- Retrying without backoff and creating a thundering herd
- Retrying without jitter so all clients sync up
- No DLQ, so one poison message stalls the pipeline
- Retrying non-idempotent work and causing double effects
Learn from primary sources
Practice and explain it back
Implement backoff with jitter
Implement exponential backoff with full jitter for a retry loop, and route a message to a DLQ after N failed attempts.
Expected evidence: Retry delays that grow exponentially and are randomized; poison messages land in a DLQ.
Open the interactive drill →Review prompts
- What problem does a dead-letter queue solve?
- Why add jitter to retry backoff?
Build evidence
Job queue with retries and DLQ
A background job queue with backoff, jitter, and a dead-letter queue.
- Retries with exponential backoff and jitter
- Poison messages routed to a DLQ
- Idempotent consumer handling
Synthesize: Infrastructure & Platforms
Design and operate a reproducible, observable, fault-tolerant platform for untrusted workloads. Produce one working system, benchmark, or evidence-backed design that integrates the path.
- Implements or precisely models the core mechanisms from all three milestones
- Includes at least one injected failure or adversarial case and demonstrates recovery
- Reports quality, latency, resource, reliability, or usability measurements relevant to the domain
- Ships a concise architecture note explaining decisions, trade-offs, and remaining risks