WAL (write-ahead log)
A durable log of changes written before applying them to the main store, enabling crash recovery and replication. Databases (Postgres, MySQL) and systems like Kafka use it. The foundation of durability.
- Any durable database or stateful system.
- Replication and point-in-time recovery.
- Ephemeral state that can be lost.
- Read-only or in-memory caches.
Definition
A write-ahead log appends intended changes to disk before applying them in memory. On crash, replaying the WAL reconstructs the in-memory state. It also feeds replication (logical/physical) and point-in-time recovery.
Why it exists
Writing changes directly to the data files is slow (random I/O) and unsafe (partial writes). The WAL turns writes into fast sequential appends and guarantees recoverability.
How it works
Append the change to the WAL (fsync for durability), then update the in-memory data structure. Checkpoints flush dirty pages to disk and truncate the WAL. Replicas stream the WAL to stay in sync. Archiving keeps WAL for point-in-time recovery.
Scaling limits
WAL throughput caps write throughput. fsync latency limits commit rate (group commit helps). Very long WAL replay slows recovery. WAL storage grows with write rate.
Cost shape
Sequential I/O (cheap) + storage. Blow-ups: synchronous replication adding commit latency and WAL archive growth.
Alternatives
- Direct writes — Simpler but no crash recovery or replication.
- Journaling filesystem — File-level durability, not DB-level semantics.
Failure modes
- fsync latency capping commits.
- WAL corruption breaking recovery.
- Archive gaps breaking PITR.