Consumer group
A set of consumers that jointly read a topic, with each partition assigned to exactly one member. It's how Kafka scales consumption: add consumers to a group to share partitions. Different groups each get the full stream (fan-out).
- Parallelizing stream consumption.
- Multiple independent consumers of one stream.
- When each message must go to one specific worker with custom routing (use a broker).
- Single-consumer simple cases.
Definition
A consumer group is a logical set of consumers sharing partition assignments for a topic. Within a group, each partition goes to one consumer; across groups, every group sees all messages independently. Rebalancing redistributes partitions on membership changes.
Why it exists
To scale consumption parallel to partitions and to support multiple independent downstreams of the same stream. One group shares the load; many groups each get a copy (pub/sub).
How it works
The group coordinator assigns partitions to members. On join/leave/failure, a rebalance reassigns. Offsets are committed per group, so each group tracks its own progress. Adding consumers up to the partition count increases parallelism.
Scaling limits
Can't have more active consumers than partitions (extras idle). Rebalances pause consumption. Stateful consumers lose state on rebalance (use sticky/cooperative rebalancing).
Cost shape
Negligible. The cost is rebalance overhead and the operational need to handle lag.
Alternatives
- Pub/sub broker — Fan-out without partition assignment, but no log replay.
- Single consumer — Simplest, but no parallelism.
Failure modes
- Rebalance storms from flapping consumers.
- Consumer lag from slow processing.
- State loss on rebalance.