Distributed Lock Fencing Token Lab (Interactive)
Pause a lock holder for 15 seconds of GC hell and watch a stale writer reach the storage anyway. Step through a lock TTL expiring during a client stall, the second client acquiring, and whether monotonic fencing tokens catch the zombie write in time.
GC Pause Lock Hazard & Fencing Tokens
Replay the classic TTL-lock failure one step at a time, then decide: does your storage check the token, or does payroll get double-written?
- → Lock service idle. Storage row "payroll_run_2024" has accepted token #0.
How It Works Under the Hood
Distributed locks break in the gap between “my lease expired” and “my process thinks it still holds the lock”: clock skew, a full GC pause, or a network stall lets the TTL lapse while the holder is still mid-write. Redlock tries to survive this across independent nodes; Martin Kleppmann’s critique shows the real fix is server-side — every lock grant carries a monotonically increasing fencing token, and the storage service rejects any write whose token is older than the highest it has seen. This lab animates that timeline and lets you toggle enforcement to see how many corrupted writes slip through without the token check.
Core Architectural Principles
- TTL expiry during a stalled client lets a second acquirer legitimately take the lock.
- Fencing tokens increase per grant; storage compares highest_seen_token before applying writes.
- Enforcement off lets the resumed zombie overwrite the new holder’s data undetected.
In a lock design, volunteer the failure mode before they ask: “leases expire; a paused holder can write after expiry, so every mutation carries a fencing token the resource validates.” Then distinguish ZooKeeper-style quorum locks with sessions from Redlock’s timing-sensitivity to independent clocks.
Locks with leases keep liveness during crashes but only fencing tokens make the unsafe stale write impossible.