Percentiles (p50/p95/p99)
Latency metrics that show the value below which that percent of requests fall. p99 = the slowest 1%. Averages hide tail latency; percentiles reveal it. Always report p95/p99 alongside the mean.
- Always for latency reporting and SLOs.
- Tail-latency debugging.
- Averaging percentiles across nodes (merge histograms instead).
- Low-traffic services where p99 is noise.
Definition
A percentile is the value below which P% of observations fall. p50 is the median, p95 the 95th percentile, p99 the 99th. Tail latencies (p99/p99.9) expose slow users that the average masks.
Why it exists
Latency distributions are skewed; a few slow requests drag the average but hide the bulk. Percentiles describe the distribution and the tail, which is what users at the tail experience.
How it works
Collect latencies into a histogram (bucketed) to compute percentiles cheaply at scale. Exact percentiles need all values (expensive); histograms approximate. Report p50, p95, p99; alert on p99 or SLO burn.
Scaling limits
Averaging percentiles across instances is mathematically wrong (use histograms). High percentiles (p99.9) are noisy at low traffic. Histograms trade memory for accuracy.
Cost shape
Histogram buckets cost memory per metric. Blow-ups: too many histogram buckets and high-cardinality labels.
Alternatives
- Average — Simple but hides the tail.
- Max — Shows the worst single request, very noisy.
Failure modes
- Averaging percentiles across instances.
- Alerting on p99 with low traffic (noisy).
- Histogram bucket misdesign.