Rate Limiting & Throttling

The Problem

APIs must protect against abuse, prevent resource exhaustion, and ensure fair usage across clients. Without rate limiting, a single misbehaving client can overwhelm your servers, a bot can scrape your entire database, or a retry storm from a downstream service can cascade into a full outage. Rate limiting controls how many requests a client can make within a time window. The challenge is implementing it efficiently in a distributed system where requests hit different servers, with minimal latency overhead, configurable per-client limits, and graceful handling of limit breaches.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

Token Bucket with Redis Lua Script

Implementing a distributed token bucket requires atomic operations. A Redis Lua script ensures atomicity:

Algorithm:

1. Calculate tokens to add since last request: tokens_to_add = (now - last_refill) * rate
2. Update bucket: tokens = min(bucket_capacity, current_tokens + tokens_to_add)
3. Check if enough tokens: if tokens >= 1, consume and allow; else reject
4. Store updated state: {tokens, last_refill_time}

Redis Lua script (executed atomically):

• KEYS[1] = rate limit key
• ARGV[1] = bucket capacity, ARGV[2] = refill rate, ARGV[3] = current timestamp
• Returns: {allowed: 0/1, remaining: N, retry_after: seconds}

Advantages of Lua script: The entire check-and-update runs as a single atomic operation on Redis. No race conditions between concurrent requests. No distributed locks needed.

Performance: A Redis Lua script executing a token bucket check takes ~0.1ms. With Redis pipelining, you can check thousands of rate limits per second per Redis connection.

Multi-Dimensional Rate Limiting

Real APIs need limits on multiple dimensions simultaneously:

Per-client: 1000 requests/minute per API key Per-endpoint: 100 requests/minute to POST /api/upload per API key Per-IP: 50 requests/second from any single IP Global: 10,000 requests/second total across all clients for a specific endpoint

Implementation: Check all applicable limits for each request. The request is allowed only if ALL limits pass. Use Redis pipeline to check multiple keys in a single round-trip:

1. INCR api_key:{key}:global → check against 1000/min
2. INCR api_key:{key}:endpoint:{path} → check against 100/min
3. INCR ip:{ip}:second → check against 50/sec
4. INCR global:{path}:second → check against 10000/sec

Priority clients: Some clients need higher limits. Store per-client overrides that supersede tier defaults. Enterprise clients may have “soft” limits that alert but don’t reject.

Graduated response: Instead of hard rejection at the limit, implement a graduated response:

• 80% of limit: return warning header
• 100% of limit: start throttling (add 100ms delay per request)
• 120% of limit: return 429 Too Many Requests
• 200%+ of limit: temporarily block the client

Handling Rate Limit Failures

What happens when Redis (the rate limit store) is down?

Fail-open: Allow all requests when the rate limiter is unavailable. Keeps the API functional but removes all protection. Suitable when availability is more important than protection.

Fail-closed: Reject all requests when the rate limiter is unavailable. Maximum safety but causes a complete outage. Suitable for security-critical endpoints.

Local fallback: Each API gateway instance maintains a local in-memory rate limiter as a fallback. Less accurate (per-instance limits instead of global) but provides basic protection. Configure local limits at 1/N of the global limit (N = number of instances).

Circuit breaker: If Redis latency exceeds 5ms (normally 0.1ms), open the circuit breaker and fall back to local rate limiting. Close the circuit after Redis recovers. This prevents Redis slowness from adding latency to every API request.

Retry storms: When clients get 429 responses, they should use exponential backoff with jitter. But many clients retry immediately, creating a storm when the window resets. Mitigate by:

• Adding jitter to Retry-After headers
• Gradually releasing blocked clients instead of all-at-once
• Implementing client-side rate limiting in your SDKs

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: Rate Limiting & Throttling

You are an expert software architect and teacher. I want to understand rate limiting & throttling as a practical system design capability. (Token bucket, sliding window, and distributed rate limiting to protect your APIs).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

APIs must protect against abuse, prevent resource exhaustion, and ensure fair usage across clients. Without rate limiting, a single misbehaving client can overwhelm your servers, a bot can scrape your entire database, or a retry storm from a downstream service can cascade into a full outage. Rate limiting controls how many requests a client can make within a time window. The challenge is implementing it efficiently in a distributed system where requests hit different servers, with minimal latency overhead, configurable per-client limits, and graceful handling of limit breaches.

## Architectural Approaches

<ApproachList>
<Approach
  name="Token Bucket Algorithm"
  pros={[
    'Allows bursts while enforcing average rate',
    'Simple to implement and understand',
    'Smooth rate limiting — no hard window boundaries',
    'Memory efficient — one counter and timestamp per client',
  ]}
  cons={[
    'Burst size must be configured carefully — too large defeats the purpose',
    'Distributed implementation requires atomic operations',
    "Doesn't enforce strict per-second limits (bursts can exceed instantaneous rate)",
  ]}
>
Each client has a "bucket" that fills with tokens at a steady rate. Each request consumes one token. If the bucket is empty, the request is rejected. The bucket has a maximum capacity, allowing short bursts up to that capacity. The most widely used algorithm for API rate limiting.
<...

[Full prompt copied to clipboard]