Metrics

Numeric measurements of system behavior over time (CPU, latency, error rate), aggregated and stored in time-series. The backbone of dashboards and alerting. Cheap to collect at scale via aggregation, unlike logs.

When to use

System health, dashboards, alerting.
Aggregated trends over time.

When not to

Debugging individual requests (use logs/traces).
High-cardinality per-user data.

Definition

Metrics are time-series of numeric values (counters, gauges, histograms) tagged with labels. They're scraped or pushed, aggregated (sum/avg/percentiles), and queried for dashboards and alerts. Prometheus is the canonical stack.

Why it exists

You need to know, at a glance, system health and trends. Metrics are cheap, aggregatable, and queryable, making them ideal for dashboards and automated alerting at scale.

How it works

Instrument code to expose counters/gauges/histograms. A scraper (Prometheus) pulls them at intervals. Storage is a TSDB optimized for time + labels. Query with PromQL. Alert on thresholds or SLO burn rate.

Scaling limits

High-cardinality labels (per-user) explode the TSDB. Aggregation loses detail (use tracing for that). Resolution is limited by scrape interval. Long retention is expensive.

Cost shape

Cheap per metric; cost scales with cardinality and retention. Blow-ups: high-cardinality labels and over-long retention.

Alternatives

Logs — Per-request detail, but expensive at scale.
Traces — Request flow, but sampled and heavier.

Failure modes

High-cardinality labels blowing up storage.
Alert fatigue from too many noisy alerts.
Stale metrics after deploys.

Logs, Distributed tracing, SLO / SLI, Percentiles (p50/p95/p99)