Failure Detection Phi Accrual Lab (Interactive)
Score each heartbeat gap against observed jitter so the network picks its own suspicion level. Feed a monitored node’s inter-arrival history, inject a GC pause, and compare fixed-timeout breaches against phi scores that adapt to real variance.
Phi Accrual Suspicion Meter
A fixed timeout says “dead” after T ms — but is T 10× normal jitter or barely over it? Phi Accrual scores each gap against the observed arrival distribution, so the network sets its own failure threshold.
- 112
- 113
- 72
- 94
- 114
- 97
- 119
- 99
- 108
- 127
- 108
- 111
- Observed mean / σ
- 103 / 14 ms
- Current phi
- 0.5
- Fixed-timeout breaches
- 0
- Phi detections (≥8)
- 0
φ = −log₁₀(P(arrival later than 111 ms | μ=103, σ=14)) = 0.5. healthy — no suspicion. Drop the fixed timeout below ~117 ms and you evict healthy nodes on pure jitter — Cassandra ships phi-accrual (threshold 8–12) for exactly this reason.
How It Works Under the Hood
A fixed failure timeout must be set above worst-case jitter or it evicts healthy nodes, which makes it silently environment-dependent — a 400 ms timeout on a clean datacenter network is a coin flip on a congested one. Phi accrual (Hayashibara et al., used by Cassandra and Akka) instead computes φ = −log₁₀(P(later arrival | observed history)), treating the inter-arrival distribution itself as the clock: normal jitter yields φ below 2, and a genuinely stalled sender’s φ climbs continuously past the 8–12 suspicion threshold. This lab maintains rolling mean and variance, plots samples as bars, and lets you fire a pause to watch the two detectors disagree.
Core Architectural Principles
- Gaussian-tail phi computed live from rolling inter-arrival mean and standard deviation.
- Fixed-timeout breaches versus phi detections plotted on the same sample window.
- Injected 6× GC pause demonstrates thresholds that must track observed variance.
Justify failure detection adaptively: “fixed timeouts embed a guess about network conditions; phi accrual converts elapsed time since the last heartbeat into a probability of death, so suspicion scales with observed jitter.” Note the residual truth: no detector distinguishes slow from dead — only quorum fencing contains the lie.
Adaptive suspicion cuts false positives on noisy links but still cannot prove death, only risk.