Log Aggregation Pipeline Lab (Interactive)

Drive log volume, trigger an error storm, and price OpenSearch vs Loki storage. Model structured JSON logs flowing through DaemonSet shippers and a Kafka buffer, then compare inverted-index vs label-only storage costs under bursty load.

Log Pipeline: DaemonSet → Kafka → OpenSearch vs Loki

Drive structured JSON log volume, trigger an outage error storm, then price head-based sampling and the two storage engines.

App log rate 50k logs/sec

2xx head-sampling drop 99%

Retention 30 days

Logs ingested /sec (4xx/5xx kept 100%)1,490

Raw volume per day41.2 GB (1.2 TB stored)

Monthly pipeline cost$39

Loki vs OpenSearch delta77% cheaper (128/mo)

Indexer backpressure verdictKafka absorbs the burst on append-only disk (3x replication) while consumers index at the sustained 2,235 logs/sec. Zero storm logs lost.

Why the engines diverge

OpenSearch builds a Lucene inverted index on every field (1.2–1.5x raw size on $0.10/GB SSD) for sub-second arbitrary queries. Loki indexes only labels like service/env, greps LZ4 chunks in S3 ($0.023/GB) — ~80–90% cheaper, slower ad-hoc full-text search. Errors emitted: 1,000/sec.

{"timestamp":"2026-09-27T10:14:02.194Z","level":"ERROR","service":"checkout-service","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","action":"order_failed","duration_ms":18.4}

How It Works Under the Hood

Application containers write typed JSON to stdout, FluentBit or Vector DaemonSets tail it, and Kafka absorbs outage bursts that would otherwise hit Lucene queues with 429s and drop the exact error logs you need. Head-based sampling discards 99% of 2xx lines while keeping every 4xx/5xx. At the storage tier, OpenSearch indexes every field at 1.2-1.5x raw size on SSD, while Loki indexes only labels and greps LZ4 chunks on S3 at 80-90% lower cost. This lab computes ingest rate, GB/day, and monthly dollars from your own knobs.

Core Architectural Principles

DaemonSet shippers decouple application latency from log-cluster availability; direct in-handler push causes cascading thread starvation.
Kafka persists 10x-50x outage log bursts to disk so indexers consume at a sustainable, capacity-matched rate.
OpenSearch full inverted index buys sub-second arbitrary search; Loki label-only indexing buys 80-90% cost reduction on object storage.

Interview Round Script

In interviews, always state the pipeline shape first: structured JSON to stdout, node-level DaemonSet forwarder, Kafka buffer, then tiered storage. Then name the cost-performance fork explicitly — Elasticsearch for hot ad-hoc search, Loki-plus-S3 for cheap high-volume retention — and mention sampling, dynamic level elevation, and edge PII redaction as the governance layer.

Key Trade-Offs

Full-text inverted-index query speed and its 1.35x RAM/SSD bill versus label-indexed brute-force grep at commodity object-storage prices.

Related Curriculum Chapter