Rate limiter
A mechanism that caps how many requests a client can make in a time window, protecting services from abuse and overload. It sits at the edge or in front of a handler and decides allow/deny/throttle per token, IP, or user. Limits are enforced with counters in memory or a shared store so multiple instances agree.
- Public APIs and login endpoints to stop brute force and credential stuffing.
- Protecting expensive downstream operations (search, exports, AI inference).
- Enforcing per-tenant quotas in multi-tenant SaaS.
- As your only abuse control against distributed attackers (many IPs) — pair with WAF/captcha.
- Internal trusted services where backpressure and circuit breakers already bound load.
- When you need strict global fairness but can't afford the shared-store hop.
Definition
A rate limiter restricts the rate of operations to a configured ceiling (requests per second, tokens per minute, concurrent calls). It returns success under the limit and rejects (429), delays, or drops requests over it. Algorithms include fixed window, sliding window, token bucket, and leaky bucket.
Why it exists
To protect finite capacity (CPU, DB connections, third-party quotas) from a single noisy or malicious client, and to enforce fair-share and API plan tiers. Without it one tenant can starve everyone else.
How it works
Each request increments a counter keyed by identity+window. Token bucket refills tokens at a steady rate and bursts up to bucket capacity; sliding window weighs recent requests for smoother limits. State lives in a fast store (Redis, in-process LRU) so checks are sub-millisecond. Distributed limiters use atomic INCR or Lua scripts to avoid races across nodes.
Scaling limits
In-process limiters are fastest but per-instance, so the effective limit scales with replica count. A shared Redis limiter is accurate globally but adds a network hop and can become a hot key under very high fan-in; sharding the counter by ID prefix mitigates this.
Cost shape
You pay for the store (Redis memory/requests) and the latency of every check. The blow-up is a hot key on a single popular ID, or unbounded memory if counters and windows aren't TTL'd.
Alternatives
- Circuit breaker — Protects callers from a bad downstream, not the service from callers.
- Backpressure — Slows producers naturally but doesn't enforce hard per-client caps.
- WAF / bot rules — Blocks known abuse patterns but not precise per-token rate ceilings.
Failure modes
- Clock skew between nodes breaks fixed-window counters.
- Hot counter key causing Redis hot-shard CPU spikes.
- Returning 429 without Retry-After, so clients retry-storm.