Phase 12 Interactive Simulators(13)
Browse all 13 labs →Log Pipeline: DaemonSet → Kafka → OpenSearch vs Loki
Drive structured JSON log volume, trigger an outage error storm, then price head-based sampling and the two storage engines.
OpenSearch builds a Lucene inverted index on every field (1.2–1.5x raw size on $0.10/GB SSD) for sub-second arbitrary queries. Loki indexes only labels like service/env, greps LZ4 chunks in S3 ($0.023/GB) — ~80–90% cheaper, slower ad-hoc full-text search. Errors emitted: 1,000/sec.
- Log Pipeline Cost LabFREE
- RED/USE & Cardinality LabFREE
- Trace Sampling WorkbenchFREE
- Alert Fatigue Shift LabFREE
- K8s Probe Cascade Lab
- Nines & Buffer Calculator
- Error Budget Policy Game
- Chaos Blast Radius Lab
- P1 Incident Commander
- Pod Shutdown Race Lab
- Canary Guardrail Gate
- Blue-Green Cutover Board
- A/B Bucketing & Stats Lab
Observability, Reliability & Ops
Operating mission-critical distributed systems requires deep observability, quantitative reliability engineering, and disciplined continuous delivery. Phase 12 covers the complete production operations lifecycle:
All Topics in Phase 12
0 of 13 completedReplace unstructured print statements with high-performance JSON logging, daemon-based log shippers (FluentBit, Vector), Kafka ingestion buffers, and tiered storage engines (OpenSearch vs. Grafana Loki).
Structure production telemetry: RED Method (Rate, Errors, Duration) for application services, USE Method (Utilization, Saturation, Errors) for infrastructure, Prometheus TSDB mechanics, and preventing high-cardinality crashes.
Trace distributed requests across microservice fleets: W3C TraceContext headers, OpenTelemetry (OTel) SDK and Collector pipelines, Span lifecycle DAGs, and head vs. tail-based sampling.
Design actionable production alerting: Symptom-based alerting vs cause-based noise, multi-window multi-burn-rate SLO alerts, Alertmanager deduplication and inhibition, and executable runbooks.
Master container lifecycle orchestration: Readiness probes for traffic gating, Liveness probes for deadlock recovery, Startup probes for slow boot processes, and preventing cascading cluster restart storms.
Master SRE reliability math: Service Level Indicators (SLI), Service Level Objectives (SLO), Service Level Agreements (SLA), availability calculations from 99.9% to 99.999%, and safety buffer architectures.
Balance feature velocity with system stability: Error Budget math ($100\% - \text{SLO}$), burn rate policies, automated release gating, feature freeze enforcement, and SRE toil reduction.
Build confidence in distributed system resilience: Hypothesis-driven fault injection, Netflix Simian Army (Chaos Monkey, Chaos Kong), Chaos Mesh, Litmus, and blast radius safety controls.
Master production incident response and organizational learning: Incident Commander roles, triage and mitigation protocols, the 5 Whys methodology, and psychological safety in blameless postmortems.
Deploy code with zero dropped connections: POSIX signal lifecycles (SIGTERM vs SIGKILL), Kubernetes pod termination race conditions, preStop hooks, connection draining, and rolling deployment parameters.
Decouple code deployment from feature release: Feature flagging architectures (LaunchDarkly, Unleash), progressive canary traffic shifting (1% -> 10% -> 100%), Automated Canary Analysis (ACA), and dark launching.
Achieve atomic zero-downtime releases: Dual identical production environments (Blue and Green), router-level pointer flipping, instant sub-second rollbacks, and the Expand-Contract database pattern.