Split-Brain and Fencing Mechanisms Lab (Interactive)
Cut a three-node cluster in half and pick which defense stops two primaries writing at once. Partition the cluster, slide how many nodes land isolated, and test no-defense chaos against quorum step-down, epoch fencing, and STONITH power-killing.
Split-Brain Defense Grid
A network partition makes both sides believe the other is dead. Without a tie-breaker, two primaries accept writes and the data diverges forever. Choose a fencing strategy before you pull the cable.
Side A (local)
PRIMARY — accepting writes- Node A
- Node B
2/3 nodes — majority
Side B (isolated)
stepped down / blocked- Node C
1/3 nodes — minority
- Dual-master?
- no
- Conflicting writes / min
- 0
- Zombie master blocked
- no
- Write availability
- both
Cluster healthy. Pull the link or grow the minority side to test your defense.
Fencing must be agent-proof: assume the old master is alive but partitioned (it thinks everyone else died). That is why production stacks combine quorum step-down with storage-level epoch rejection — and why heartbeat-only failure detection is never sufficient.
How It Works Under the Hood
Split-brain happens when a partition makes both sides conclude the other is dead, so each elects itself primary and both accept writes — data diverges and merge is impossible after the fact. Defenses differ in who decides: quorum rules let the minority side step down autonomously, epoch or generation fencing makes shared storage reject writes carrying stale tokens even from a zombie that skipped step-down, and STONITH (shoot-the-other-node-in-the-head) has the majority physically cut power to the unreachable peer, the only option that works for even-sized clusters. This lab renders both sides’ master state and a conflicting-writes counter as you vary them.
Core Architectural Principles
- Minority-side self-fencing via quorum versus majority-driven STONITH escalation.
- Epoch tokens at storage level fence zombie writers whose own logic failed them.
- Even-cluster edge case: two-node split freezes under quorum, needs STONITH or witness.
For a primary-backup design, volunteer the split-brain question before they ask: “the minority steps down by quorum, and every write carries an incrementing epoch the storage validates — a paused old master is harmless.” Mention STONITH for two-node clusters where quorum freezes everything.
Every defense trades some availability during partitions to make simultaneous mastership structurally impossible.