Kubernetes Health Probes Lab (Interactive)
Boot 12 slow JVM pods, stall their database, and watch who gets SIGKILLed. Step a kubelet tick-by-tick through startup, liveness, and readiness decisions, then wire liveness to the shared DB and trigger the cluster-wide restart storm.
Kubelet Probe FSM: Startup / Liveness / Readiness
12 slow-booting pods. Tune the boot budget, wire liveness to the wrong endpoint, then pull the DB rug.
› Idle: press Deploy fleet to start the rolling boot of 12 order-api pods.
How It Works Under the Hood
The three probes control different things: the startup probe suppresses everything else until a slow JVM finishes booting within failureThreshold x periodSeconds; the readiness probe gates Service endpoint membership without ever killing the container; the liveness probe SIGKILLs only genuinely wedged processes. Couple liveness to an external dependency and a 40-second database stall becomes a mass execution event: all pods die on the same tick, then 12 fresh JVMs stampede the connection pool together, converting a brownout into a CrashLoopBackOff death spiral. This lab runs that exact FSM.
Core Architectural Principles
- Readiness failure removes the pod IP from EndpointSlice; the container keeps running to drain and recover.
- Liveness failure after 3 consecutive checks sends SIGKILL — so it must test in-process health only.
- Startup probe with a large failureThreshold (30 x 5s) protects slow boots without loosening liveness.
Answer probe questions with the control-plane split: readiness routes traffic, liveness restarts processes, startup guards boot. Then tell the cascade story — liveness checking SELECT 1, a DB stall killing the whole fleet simultaneously, and the rebooting thundering herd — and note that exec probes at high pod density burn host CPU and PIDs.
Aggressive probes recover deadlocked processes fast but convert shared-dependency stalls into fleet-wide restart storms; shallow probes need real deadlock detection to be trustworthy.