Dead-Letter Queue & Poison Pill Lab (Interactive)
Feed a retry storm malformed payloads and quarantine the poison pill with redrive and jitter. Sweep poison-message rate, maxReceiveCount, and base backoff to watch exponential retries with full jitter drain the queue—or pile it into the DLQ for redrive.
Poison Pills, Retry Backoff & DLQ Redrive
Run a worker over malformed payloads and watch retries with exponential backoff + jitter decide: recover or quarantine.
ACK'd
0
healthy pipeline
Worker Attempts
0
each = a full dequeue cycle
Retry Delay
0 ms
Σ backoff paid downstream
DLQ Depth
0
Zero-tolerance alert quiet
Primary Queue (0)
empty
DLQ Quarantine (0)
no quarantined messages
› Primary queue empty — enqueue a batch of orders.
Without quarantine a poison pill crash-loops every worker head-of-line behind it. maxReceiveCount bounds the damage, full jitter — T = U(0, min(cap, base·2ⁿ)) — spreads retries so 10,000 failing workers do not hit recovering Stripe in lockstep, and diagnostic headers turn triage into a redrive script, exactly like Stripe's dashboard replay.
How It Works Under the Hood
A poison pill—any message that fails deterministically, like a payload with an unsupported currency—would retry forever if left in the main queue, burning workers and delaying healthy messages. Redrive policies cap attempts: once receive count exceeds maxReceiveCount the broker quarantines the message onto a dead-letter queue with X-Original-Queue and X-Exception headers for forensics. Exponential backoff with full jitter, T = U(0, min(cap, base·2^n)), spreads retries so recovery does not thunder-herd, and alarms on DLQ growth turn silent data loss into a paged incident.
Core Architectural Principles
- maxReceiveCount bounds attempts; the (n+1)-th failed receive redrives the message to the DLQ.
- Full-jitter backoff draws delay from U(0, min(cap, base·2^n)), de-synchronizing retry storms across workers.
- Redrive replays DLQ payloads after the code fix; purge deletes them—either way no healthy message starves.
Treat the DLQ as non-negotiable infrastructure: "every queue gets a redrive policy and an alarm; a DLQ nobody watches is a data-loss bucket." Explain jitter’s role after outages, keep maxReceiveCount small for deterministic failures, and describe fixing the consumer then redriving, not hand-editing messages in production.
Aggressive quarantine protects queue throughput but hides bugs unless DLQ alarms and redrive tooling exist from day one.