Real-Time Analytics Pipeline Lab (Interactive)
Stream Kafka events into MergeTree, toggle on-insert rollups, and dodge the TOO_MANY_PARTS write storm. Compare dashboards that re-scan billions of raw events against AggregatingMergeTree materialized-view rollups, while sizing insert batches against background merge capacity.
Kafka → ClickHouse Streaming Rollups
Move aggregation from query-time to ingestion-time and size the insert batches behind MergeTree parts.
How It Works Under the Hood
Real-time OLAP shifts aggregation from query-time to ingestion-time. A ClickHouse Kafka Engine table consumes partitions, and an on-insert materialized view folds each micro-batch into 1-minute buckets of sumState and uniqState HyperLogLog sketches stored in an AggregatingMergeTree. Dashboards then merge thousands of rollup rows in milliseconds instead of re-scanning billions of raw events at query time. The catch is MergeTree mechanics: every insert creates an immutable part, so batching too finely starves the cluster with background merges instead of serving reads.
Core Architectural Principles
- Sparse primary index marks every 8,192 rows, keeping multi-gigabyte indexes in RAM.
- AggregatingMergeTree stores incremental aggregate states so queries merge instead of scan.
- Insert batching governs parts-per-minute; exceeding merge throughput triggers TOO_MANY_PARTS.
For live analytics designs, propose a Kappa architecture: Kafka as the single append-only truth with ClickHouse materialized views pre-aggregating hot windows, and cite replay by rewinding offsets instead of a parallel batch layer. Quantify the trade: rollups answer at 3 ms but lose raw granularity, so retain a TTL-bounded raw table for retrospective queries.
Ingestion-time rollups buy sub-10 ms queries at the price of fixed bucket granularity and merge tuning.