Rate Limiting Implementation Patterns — Token Bucket, Redis, and API Gateway
In this tutorial, you'll learn about Rate Limiting Implementation Patterns. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Rate Limiting controls the number of requests a client can make to a server within a time window, preventing resource exhaustion, ensuring fair usage, and protecting backend services from abuse.
What You'll Learn
By the end of this tutorial, you will implement token bucket, Sliding Window, and distributed rate limiter algorithms, rate-limit API endpoints with middleware, and choose the right strategy for production deployments.
Why It Matters
A single misconfigured client or malicious attacker can saturate backend resources, causing latency spikes and outages for all users. Doda Browser's public API rate-limits clients per API key, preventing any single client from overwhelming the service.
Real-World Use
A payment gateway API limits merchants to 100 requests per second. A faulty merchant integration sends 1000 requests per second. The rate limiter returns HTTP 429 (Too Many Requests) with a Retry-After header, protecting the gateway from crashing and other merchants from latency.
Rate Limiting Architecture
Graph LR subgraph "Client Layer" C1[Client A] C2[Client B] C3[Attacker] end subgraph "Rate Limiter" RL[Rate Limiter Middleware] RC[(Redis Cluster)] end subgraph "Backend" API[API Server] DB[(Database)] end C1 --> RL C2 --> RL C3 --> RL RL --> RC RL -. "429" .-> C3 RL --> API API --> DB style RL fill:#f90,color:#fff
The rate limiter sits between clients and the API server. It checks each request against the stored State in Redis and either forwards allowed requests or rejects excess requests with 429.
Table: Rate Limiting Algorithms
| Algorithm | Memory Usage | Accuracy | Burst Handling | Use Case |
|---|---|---|---|---|
| Token Bucket | Low | Good | Allows bursts | API Rate Limiting |
| Leaky Bucket | Low | Good | Smooths traffic | Queue processing |
| Fixed Window | Lowest | Poor | Boundary bursts | Simple throttling |
| Sliding Window Log | High | Best | Precise | High-accuracy limits |
| Sliding Window Counter | Medium | Good | Acceptable | Distributed Systems |
Python: Token Bucket Implementation
# token_bucket.py
# Thread-safe token bucket rate limiter
import time
import threading
from typing import Optional
class TokenBucket:
"""Token bucket rate limiter with Thread Safety."""
def __init__(self, capacity: int, refill_rate: float, refill_interval: float = 1.0):
self.capacity = capacity
self.refill_rate = refill_rate
self.refill_interval = refill_interval
self.tokens = capacity
self.last_refill = time.monotonic()
self.lock = threading.Lock()
def _refill(self):
"""Refill tokens based on elapsed time."""
now = time.monotonic()
elapsed = now - self.last_refill
new_tokens = elapsed * self.refill_rate
if new_tokens > 0:
self.tokens = min(self.capacity, self.tokens + new_tokens)
self.last_refill = now
def consume(self, tokens: int = 1) -> bool:
"""Try to consume tokens. Returns True if allowed."""
with self.lock:
self._refill()
if self.tokens >= tokens:
self.tokens -= tokens
return True
return False
def get_wait_time(self) -> float:
"""Get seconds until a token becomes available."""
with self.lock:
if self.tokens > 0:
return 0
return (1.0 / self.refill_rate) if self.refill_rate > 0 else float('inf')
# Usage
bucket = TokenBucket(capacity=10, refill_rate=2) # 10 burst, 2/sec refill
for i in range(15):
allowed = bucket.consume()
print(f"Request {i + 1}: {'ALLOWED' if allowed else 'DENIED'}")
time.sleep(0.1)
Expected output:
Request 1: ALLOWED
...
Request 10: ALLOWED
Request 11: DENIED
Request 12: DENIED
(tokens refill at 2/sec, later requests may be allowed)
Distributed Rate Limiter with Redis
# Redis_rate_limiter.py
# Distributed rate limiter using Redis sorted sets
import time
import Redis
from typing import Optional
class RedisSlidingWindowLimiter:
"""Distributed Sliding Window rate limiter."""
def __init__(self, Redis_client: Redis.Redis):
self.Redis = Redis_client
def is_allowed(
self, key: str, max_requests: int, window_seconds: int
) -> bool:
"""Check if request is allowed for the given key."""
now = time.time()
window_start = now - window_seconds
# Remove old entries outside the window
self.Redis.zremrangebyscore(key, 0, window_start)
# Count current entries in window
current_count = self.Redis.zcard(key)
if current_count >= max_requests:
return False
# Add current request
self.Redis.zadd(key, {str(now): now})
self.Redis.expire(key, window_seconds)
return True
def get_remaining(self, key: str, max_requests: int) -> int:
"""Get remaining requests in current window."""
now = time.time()
window_start = now - 60
self.Redis.zremrangebyscore(key, 0, window_start)
current = self.Redis.zcard(key)
return max(0, max_requests - current)
def get_reset_time(self, key: str) -> float:
"""Get seconds until the current window resets."""
now = time.time()
oldest = self.Redis.zrange(key, 0, 0, withscores=True)
if oldest:
return oldest[0][1] + 60 - now
return 0
class RateLimiterMiddleware:
"""FastAPI middleware for Rate Limiting."""
def __init__(self, Redis_client: Redis.Redis):
self.limiter = RedisSlidingWindowLimiter(Redis_client)
self.default_limits = {
'anonymous': (10, 60), # 10 req/min
'authenticated': (100, 60), # 100 req/min
'premium': (1000, 60), # 1000 req/min
'admin': (10000, 60), # 10000 req/min
}
async def check_rate_limit(self, client_id: str, tier: str = 'anonymous'):
"""Check rate limit for a client."""
max_req, window = self.default_limits.get(tier, (10, 60))
key = f"ratelimit:{tier}:{client_id}"
if not self.limiter.is_allowed(key, max_req, window):
remaining = self.limiter.get_remaining(key, max_req)
reset_time = self.limiter.get_reset_time(key)
return {
'allowed': False,
'remaining': remaining,
'reset_after': int(reset_time),
'retry_after': max(1, int(reset_time)),
}
return {
'allowed': True,
'remaining': self.limiter.get_remaining(key, max_req),
'limit': max_req,
}
# Initialize
R = Redis.Redis(host='localhost', port=6379, db=0)
middleware = RateLimiterMiddleware(R)
Node.js: Express Rate Limiter Middleware
// rate-limiter.js
// Express middleware for distributed Rate Limiting
const Redis = require('Redis');
const client = Redis.createClient({ URL: 'Redis://localhost:6379' });
class SlidingWindowCounter {
constructor(windowSeconds, maxRequests) {
this.windowSeconds = windowSeconds;
this.maxRequests = maxRequests;
}
async isAllowed(key) {
const now = Date.now();
const windowKey = `rl:${key}:${Math.floor(now / (this.windowSeconds * 1000))}`;
const multi = client.multi();
multi.incr(windowKey);
multi.expire(windowKey, this.windowSeconds + 1);
const results = await multi.exec();
const count = results[0];
return {
allowed: count <= this.maxRequests,
remaining: Math.max(0, this.maxRequests - count),
resetIn: this.windowSeconds - ((now / 1000) % this.windowSeconds),
};
}
}
const limiter = new SlidingWindowCounter(60, 100);
function rateLimit(config = {}) {
const windowMs = config.windowMs || 60000;
const max = config.max || 100;
const keyGenerator = config.keyGenerator || ((req) => req.ip);
return async (req, res, next) => {
const key = keyGenerator(req);
const result = await limiter.isAllowed(key);
// Set standard rate limit headers
res.set('X-RateLimit-Limit', String(max));
res.set('X-RateLimit-Remaining', String(result.remaining));
res.set('X-RateLimit-Reset', String(Math.ceil(result.resetIn)));
if (!result.allowed) {
res.set('Retry-After', String(Math.ceil(result.resetIn)));
return res.status(429).JSON({
error: 'Too Many Requests',
message: `Rate limit exceeded. Retry in ${Math.ceil(result.resetIn)} seconds.`,
retryAfter: Math.ceil(result.resetIn),
});
}
next();
};
}
// Per-route rate limits
const authLimiter = rateLimit({ windowMs: 60000, max: 5 }); // 5 req/min
const apiLimiter = rateLimit({ windowMs: 60000, max: 100 }); // 100 req/min
const webhookLimiter = rateLimit({ windowMs: 60000, max: 1000 }); // 1000 req/min
module.exports = { rateLimit, authLimiter, apiLimiter, webhookLimiter };
Common Errors
1. Rate Limiting by IP Only
NAT gateways cause thousands of legitimate users to share a single IP. Rate Limiting by IP alone blocks entire organizations. Use composite keys: user_id:ip, api_key, or session_id. Fall back to IP only when no authenticated identity is available.
2. Not Returning Rate Limit Headers
Clients cannot adapt their behavior without knowing their remaining quota. Always return X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and Retry-After headers following the IETF standard draft.
3. Using Local Memory in Distributed Systems
Each server instance having its own in-memory rate counter means a client can make N requests per instance instead of N total. Use Redis for distributed Rate Limiting to ensure consistent limits across all instances.
4. Fixed Window Boundary Problem
A fixed window (e.g., 100 requests per minute) resets at minute boundaries. A client can send 100 requests at 11:59:59 and 100 requests at 12:00:00, effectively doubling the limit. Use Sliding Window algorithms to prevent this.
5. Not Rate Limiting at Multiple Layers
Application-level Rate Limiting catches sophisticated attacks, but simple DDoS floods never reach the application. Implement Rate Limiting at the CDN, load balancer, API Gateway, and application layers for defense in depth.
6. Rate Limiting Authenticated Users the Same as Anonymous
Give authenticated users higher limits than anonymous users. This rewards registration and improves the user experience. Tier-based limits (free/pro/enterprise) allow differentiated service levels.
Practice Questions
1. What is the difference between token bucket and Sliding Window counter?
Token bucket allows bursts up to a capacity and enforces an average rate by consuming tokens over time. Sliding Window counter tracks request counts within a rolling time window. Token bucket handles bursts naturally; Sliding Window provides precise time-based limits.
2. How does Redis enable distributed Rate Limiting?
Redis provides atomic operations (INCR, EXPIRE, sorted sets) that maintain consistent State across all server instances. All application servers read and write to the same Redis instance or cluster, ensuring a unified rate limit regardless of which server handles the request.
3. Why use exponential backoff with rate limit headers?
When a client receives 429, retrying immediately will also fail and waste resources. Exponential backoff (retry at 1s, 2s, 4s, 8s) spreads retries over time. The Retry-After header tells the client exactly when to retry.
4. How do you handle Rate Limiting for WebSocket connections?
Track the number of messages per second per connection, not the connection count itself. Apply rate limits on message frequency, payload size, and concurrent connections. Disconnect clients that exceed limits and implement a cooldown period.
5. Challenge: Design a multi-tier Rate Limiting system for a SaaS platform: anonymous users get 10 requests per minute (global), free-tier users get 100 requests per minute (per user), pro users get 1000 requests per minute (per user) with 200 burst capacity, and enterprise users get 10,000 requests per minute with custom burst. Use Redis Sliding Window counter. Implement Lua scripting for atomic operations. Handle tier upgrades mid-window by resetting the counter on upgrade. Return RFC-compliant rate limit headers.
Mini Project: API Rate Limiting Dashboard
# rate_limit_dashboard.py
# Rate Limiting analytics and monitoring
import time
from collections import defaultdict
from datetime import datetime
class RateLimitMonitor:
"""Monitor and report Rate Limiting activity."""
def __init__(self):
self.requests = []
self.blocked = []
def record_request(self, client_id: str, endpoint: str, allowed: bool):
"""Record a rate-limited request."""
record = {
'client_id': client_id,
'endpoint': endpoint,
'allowed': allowed,
'timestamp': datetime.utcnow().isoformat(),
}
if allowed:
self.requests.append(record)
else:
self.blocked.append(record)
def generate_report(self):
"""Print a Rate Limiting report."""
total = len(self.requests) + len(self.blocked)
blocked_count = len(self.blocked)
block_rate = (blocked_count / total * 100) if total > 0 else 0
# Top blocked clients
client_blocks = defaultdict(int)
for b in self.blocked:
client_blocks[b['client_id']] += 1
top_blocked = sorted(client_blocks.items(), key=lambda x: x[1], reverse=True)[:5]
# Top endpoints
endpoint_counts = defaultdict(int)
for R in self.requests + self.blocked:
endpoint_counts[R['endpoint']] += 1
top_endpoints = sorted(endpoint_counts.items(), key=lambda x: x[1], reverse=True)[:5]
print("=== Rate Limiting Dashboard ===")
print(f"Total requests: {total}")
print(f"Allowed: {len(self.requests)}")
print(f"Blocked: {blocked_count}")
print(f"Block rate: {block_rate:.2f}%")
print()
print("Top Blocked Clients:")
for client, count in top_blocked:
print(f" {client}: {count} blocked")
print()
print("Top Endpoints:")
for endpoint, count in top_endpoints:
print(f" {endpoint}: {count} requests")
print()
print(f"Period: {self.requests[0]['timestamp'] if self.requests else 'N/A'} "
f"to {datetime.utcnow().isoformat()}")
monitor = RateLimitMonitor()
monitor.record_request('user:42', '/API/users', True)
monitor.record_request('user:99', '/API/login', False)
monitor.record_request('attacker:1', '/API/data', False)
monitor.generate_report()
FAQ
Related Concepts
What's Next
You now understand Rate Limiting implementation patterns. Next, learn about the Circuit Breaker Pattern for protecting downstream services, then explore retry strategies for handling rate limit responses.
- Practice daily — Add Rate Limiting middleware to an existing API endpoint
- Build a project — Build a distributed rate limiter with Redis and Lua scripting that handles multi-tier limits, sliding windows, and returns RFC-compliant headers
- Explore related topics — Check out API Gateway Rate Limiting and CDN-level DDoS protection
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro