Skip to content
Devgo

Kafka

A distributed, partitioned, replicated append-only log used as a streaming platform and durable message bus. Producers write to topics, consumers read in order per partition. It decouples producers from consumers at huge throughput.

When to use
  • Event streaming, CDC, decoupling producers/consumers at scale.
  • Audit/event logs needing replay.
When not to
  • Simple task queues with per-message ack/retry (use SQS/RabbitMQ).
  • Low-throughput point-to-point RPC.

Definition

Kafka is a distributed log: topics are split into partitions, each an ordered, append-only, replicated sequence of records. Consumers in a group split partitions; offsets track position. Retention is time/size-based, so data persists for replay.

Why it exists

Point-to-point queues don't let many independent consumers replay the same stream, and traditional brokers don't handle huge throughput. Kafka's durable log enables event-driven architectures with decoupled, replayable consumers.

How it works

Producers append records to a (by key for ordering). Brokers replicate partitions (-ish in KRaft, or ZooKeeper historically). Consumers commit offsets; rebalancing reassigns partitions. Retention keeps data for days, enabling new consumers to replay history.

Scaling limits

Ordering is only within a , not across. Partition count sets max parallelism (can't have more consumers than partitions). Rebalances pause consumers. Very high partition counts stress brokers. No per-message priority or delayed delivery.

Cost shape

Broker compute + storage (retention). Blow-ups: long retention on high-throughput topics, over-, and too many partitions.

Alternatives

  • SQS — Managed task queue, simpler, but no replay or log semantics.
  • RabbitMQ — Rich routing and per-message ack, lower throughput.

Failure modes

  • Consumer lag going unnoticed.
  • Rebalance storms.
  • Unbounded retention filling disk.

, , , , ,