Partition
A unit of parallelism and ordering in a distributed log or table. A topic/table is split into partitions; each is ordered and handled by one broker/shard. More partitions = more parallelism but more overhead.
- Any distributed log/table needing horizontal scale.
- When per-key ordering matters.
- When you need global ordering (single partition, low throughput).
- Small workloads where one partition suffices.
Definition
A partition is an ordered, append-only sequence (in Kafka) or a hash bucket (in DBs) that lives on one node. Producers write to a partition by key; consumers read partitions in parallel. Partitions are the scaling unit.
Why it exists
A single log/table can't handle all throughput or data. Partitioning splits work and data across nodes while preserving order within a partition, enabling horizontal scale.
How it works
Records with the same key go to the same partition, preserving per-key order. Partitions are replicated for fault tolerance. A consumer group assigns partitions to consumers; one partition → one consumer in a group. Rebalancing reassigns.
Scaling limits
Ordering is per-partition only. Partition count caps consumer parallelism. Too many partitions stress brokers (metadata, requests). Hot keys concentrate load on one partition.
Cost shape
More partitions = more broker overhead and metadata. Blow-ups: over-partitioning and hot partitions.
Alternatives
- Sharding key — Same idea for DBs; partitions are the streaming term.
- Single partition — Global order but no parallelism.
Failure modes
- Hot partition from a skewed key.
- Rebalance stalls.
- Too many partitions degrading brokers.