Logs
Immutable, timestamped records of discrete events with context, used for debugging and audit. Structured logs (JSON) are queryable. More expensive than metrics at scale, so often sampled or tiered.
- Debugging and audit.
- Per-request context metrics can't show.
- Aggregated health (use metrics).
- High-volume events (sample or use metrics).
Definition
Logs are append-only event records with fields (timestamp, level, message, context). Structured logging makes them queryable. They're collected, indexed, and searched for debugging and audit. Levels (debug/info/warn/error) filter volume.
Why it exists
Metrics tell you something's wrong; logs tell you why, with per-request context. They're the forensic record for debugging and compliance.
How it works
Apps emit structured log lines to stdout/files; collectors (Fluentd, Vector) ship to an index (Loki, Elasticsearch). Query by fields/time. Sampling and levels control volume and cost.
Scaling limits
Volume grows with traffic; indexing all logs is expensive. High cardinality (per-request IDs) stresses indexes. Retention is costly. Sampling loses rare events.
Cost shape
Ingestion + indexing + storage, often the priciest observability signal. Blow-ups: verbose logging, over-long retention, and indexing high-cardinality fields.
Alternatives
- Metrics — Cheap aggregation, but no per-request detail.
- Traces — Request flow with spans, but sampled.
Failure modes
- Cost blow-ups from verbose logging.
- Unstructured logs unqueryable.
- Lost logs during outages.