Sampling
Recording a fraction of events (traces, logs) to control cost while keeping statistical signal. Head sampling decides at the source; tail sampling decides after seeing the whole trace, keeping interesting ones. Essential for tracing at scale.
- Tracing and high-volume logs.
- Controlling observability cost.
- Metrics (aggregate, don't sample).
- Audit logs that must be complete.
Definition
Sampling keeps a subset of events. Probabilistic head sampling drops randomly at ingestion; tail sampling keeps traces based on attributes (errors, slow) after completion. Rate-based sampling keeps N per second.
Why it exists
Collecting every trace/log is too expensive at high traffic. Sampling keeps cost bounded while preserving enough signal to debug and compute aggregate stats (with care).
How it works
Head: drop a random fraction early (cheap, but can't keep all errors). Tail: buffer the trace, then keep if it's an error/slow/interesting (expensive, but keeps the valuable ones). Adjust rates per service to balance cost and coverage.
Scaling limits
Head sampling can miss rare errors; tail sampling needs buffering and is more expensive. Low rates lose rare-event signal. Stats from sampled data need weighting.
Cost shape
Reduces ingestion/storage linearly with the sampling rate. Blow-ups: too-high rates and tail-sampling buffer memory.
Alternatives
- Collect everything — Complete signal, prohibitive cost.
- Aggregation only — Cheap but no per-request detail.
Failure modes
- Missing the rare error due to head sampling.
- Tail-sampling buffer OOM.
- Biased sampling skewing stats.