Kafka Offset & Event Replay Lab (Interactive)
Commit manually or auto, crash mid-batch, then rewind offsets and see duplicates or loss. Process partition offsets one batch at a time, crash the consumer mid-processing, and watch commit strategy decide whether replay yields duplicates or silently skipped records.
Kafka Offsets & Event Replay
Crash consumers at the wrong instant to expose auto-commit loss, then rewind offsets to replay the immutable log.
Committed / LEO
0 / 12
lag: 12 records
Sink Rows
0
0 distinct orders written
Silently Lost
0
auto-commit crash skips
Duplicates
0
re-written on redelivery
Partition 0 log (__consumer_offsets key: ('analytics-prod', 'orders', 0))
› Consumer group analytics-prod started at committed offset 0.
Kafka never deletes on read — consumption is a cursor over immutable bytes, which is precisely what makes replay a superpower: New Relic rewinds parse-bugged telemetry, teams seed a new Elasticsearch from offset 0. The price is at-least-once semantics, so every sink must be idempotent. Toggle auto-commit + append and crash; then switch to manual + upsert and crash again to feel the difference.
How It Works Under the Hood
Kafka consumers own their position: the committed offset in __consumer_offsets is just a bookmark, and the log persists long past consumption. Auto-commit on a timer marks offsets finished before your handler truly succeeded, so crashes lose in-flight records; manual commit after processing gives at-least-once, which replays duplicates into the sink unless writes are idempotent. The replay superpower is rewinding to earliest—rebuilding projections or backfilling a new consumer—without asking the producer for anything.
Core Architectural Principles
- Auto-commit can advance the bookmark before the sink write lands; a crash then permanently skips those offsets.
- Manual post-process commit survives crashes as duplicates: replayed offsets re-apply events the sink already has.
- Rewind-to-earliest rebuilds state from the retained log—append-only sinks grow copies, upsert sinks converge.
Say "Kafka gives at-least-once for free; exactly-once is built downstream by idempotent sinks or transactional produce-and-commit." Explain commit-before-process versus after, why rewinding offsets is a rebuild button, and that a duplicate order applied twice into an append-only ledger is the bug to prevent.
Late commits maximize safety but require idempotent processing; early commits stay duplicate-free but trade away records on crash.