Kafka Partitions & Consumer Groups Lab (Interactive)
Kill brokers, retune acks and min.insync, and see ISR shrinkage reject writes or drop them. Drive a replicated topic partition by partition: assignment caps consumers at partition count, acks=all plus ISR voting decide durability, and broker kills reshape the group live.
Kafka Partitions, ISR & Consumer Groups
Hash keys into partitions, scale the consumer group against the cardinality rule, and kill brokers to test acks=all.
Active Consumers
3
min(P, C) rule
ISR Size
3/3
lagging followers ejected
Rejected Writes
0
NotEnoughReplicas errors
Group Lag
0
LEO 0 − committed
Topic "orders" — MurmurHash(key) % 3
empty log — produce a record
empty log — produce a record
empty log — produce a record
Zero-loss: ISR ≥ min.insync with acks=all
One partition, one consumer per group — the assignment stays 1:1 until you add more consumers than partitions. Raising partitions later re-hashes future keys and breaks per-key order, so plan the count up front; Netflix runs 64–256 per topic.
How It Works Under the Hood
A Kafka topic is split into ordered partitions, each replicated across brokers with a leader serving reads and writes and in-sync replicas (ISR) voting on durability. Consumer groups give scale-out semantics: within a group every partition belongs to exactly one consumer, so more consumers than partitions are idle. Producer acks choose the failure contract—0 for fire-and-forget, 1 for leader-only, all for ISR commitment which rejects writes once ISR falls below min.insync.replicas.
Core Architectural Principles
- Effective parallelism is min(partitions, consumers): extra group members sit idle, extra partitions cost segments and open files.
- acks=all with min.insync.replicas=2 rejects produce when ISR shrinks after a broker kill—durability over availability.
- Keyed records route via hash(key) % partitions, so one key’s ordering lives and dies with one partition.
Draw the partition/replica/ISR structure before answering anything else, then pin the two hard rules: consumer count beyond partition count is waste, and acks=all with min.insync.replicas ≥ 2 is the only no-loss combo. Mention adding partitions to scale out and that partition count can never decrease.
More partitions scale consumers but shard ordering wider and slow leader election; stricter acks protect data but reject writes during under-replication.