Change Data Capture vs Dual-Write Lab (Interactive)
Commit order batches with a flaky Elasticsearch connection and count permanent desyncs until CDC replaces dual-writes. Compare application dual-writing, log-based Debezium CDC, and the Transactional Outbox under injected network failures and consumer restarts.
CDC Debezium vs Dual-Write Sync Lab
Keep Elasticsearch in sync with PostgreSQL — application dual-writes, log-based CDC, or Transactional Outbox.
PostgreSQL rows
0
Elasticsearch docs
0
Desync (missing docs)
0
Failed 2nd writes
0
Dual-writes cannot commit across two systems atomically: every network timeout between write 1 and write 2 leaves the search index permanently behind. Log-based CDC makes the WAL the single source of truth — zero app changes, replayable event stream — while the Transactional Outbox adds rich domain events committed in the same ACID transaction.
How It Works Under the Hood
Keeping PostgreSQL, Elasticsearch, and Redis in sync with application-level dual writes fails on every timeout between the first and second write, leaving the search index permanently behind with no atomicity spanning two systems. Log-based CDC treats the committed WAL as the single source of truth: Debezium tails it as a replication follower, publishes ordered before/after events to Kafka, and sinks re-apply them idempotently. The Transactional Outbox commits domain events inside the same ACID transaction for guaranteed at-least-once emission.
Core Architectural Principles
- Partial dual-write failures create desyncs that self-heal only through CDC replay from the log.
- Debezium emits before/after change events keyed by log sequence, requiring zero application changes.
- At-least-once delivery makes idempotent consumers with dedup keys mandatory downstream.
When asked how to sync a search index, reject dual-writing by name and explain the atomicity gap, then propose Debezium on the WAL into Kafka with idempotent sinks keyed by document id. Add the Transactional Outbox when consumers need rich domain events rather than raw row deltas, and mention Schema Registry governance for event evolution.
Replayable, race-free synchronization versus operating Kafka Connect plus consumers that must be idempotent.