Idempotency in Distributed Calls Lab (Interactive)
Lose responses to timeouts and see retries turn into double charges without idempotency keys. Simulate a payment gateway with configurable response loss and client retries, counting gateway executions per purchase with and without stored idempotency keys.
Idempotency Key Retry Lab
A payment charge whose response is lost is not a failed charge — it may have already run. Retrying without an idempotency key turns dropped responses into double charges.
- Charges at the gateway
- 25
- Customers double-charged
- 0
- Cache replays
- 8
- Exactly one charge
- 25/25
| Customer | First response | Gateway executions | Verdict |
|---|---|---|---|
| Ava | arrived | 1 | charged once |
| Ben | arrived | 1 | charged once |
| Cara | arrived | 1 | charged once |
| Dan | lost in network | 1 | retry replayed from cache |
| Eli | lost in network | 1 | retry replayed from cache |
| Faye | lost in network | 1 | retry replayed from cache |
| Gus | arrived | 1 | charged once |
| Hana | arrived | 1 | charged once |
| Ivan | arrived | 1 | charged once |
| Jade | lost in network | 1 | retry replayed from cache |
The server stores the key (a client-generated UUID, kept ~24h) with the final response: the first execution charges, later attempts with the same key replay the stored answer. Keys make at-least-once delivery safe — combined with a unique constraint on the charge itself, the effect is exactly-once processing, not exactly-once delivery.
Sample: 25 purchases, ≈33+ client attempts under 35% loss.
How It Works Under the Hood
In distributed calls a timeout never means failure — the server may have committed while the response died in transit, so a client retry silently becomes a second execution. The fix is a client-generated idempotency key: the server records the first execution’s outcome against the key and replays that stored response to every duplicate attempt, converting at-least-once delivery into exactly-once effect. This lab sends twenty-five real purchases through that uncertainty and prints a per-customer ledger of gateway executions, cache replays, and over-charges, making the retry-safety contract measurable instead of theoretical.
Core Architectural Principles
- Response-loss versus execution are separated: dropped acks, not dropped work, trigger retries.
- Keyed requests replay the cached response; unkeyed ones re-execute at the gateway each attempt.
- Double-charge counter quantifies the bug that idempotency keys exist to make impossible.
When asked how to make POST endpoints safe, name the whole contract: client generates a UUID, server persists key plus response for a retention window, duplicates replay it, conflicts on same-key-different-body are rejected, and a unique constraint backs the effect itself. Note Stripe-style headers so the interview sees production grounding.
Idempotency storage adds bookkeeping and retention choices, but retries without it are guaranteed corruption.