API Rate Limiting & Throttling

Protecting your API from abuse, one request at a time

Home
Hld Concepts
Communication
Rate Limiting

Why Rate Limiting?

Section titled “Why Rate Limiting?”

Without rate limiting:

❌ One client can overwhelm your API
❌ DDoS attacks possible
❌ Unfair resource usage
❌ Backend services crash

With rate limiting:

✅ Fair usage for all clients
✅ Protection from abuse
✅ Backend services protected
✅ Predictable costs

Rate Limiting Algorithms

Section titled “Rate Limiting Algorithms”

1. Token Bucket (Most Popular)

Section titled “1. Token Bucket (Most Popular)”

Bucket holds tokens. Tokens refill at fixed rate. Request consumes token.

Characteristics:

✅ Allows bursts (up to bucket capacity)
✅ Smooths to average rate over time
✅ Simple to implement
✅ Most popular algorithm

Algorithm:

Initialize bucket with capacity tokens
Refill tokens at refill_rate per second
When request arrives:
- If tokens available → consume token, allow request
- If no tokens → reject request (429)

Token Bucket Implementation

Section titled “Token Bucket Implementation”

Python
Java

"token_bucket.py

1
import time
2
from threading import Lock
3
from typing import Optional
4

5
class TokenBucket:
6
    """Token bucket rate limiter"""
7

8
    def __init__(self, capacity: int, refill_rate: float):
9
        """
10
        Args:
11
            capacity: Maximum tokens in bucket
12
            refill_rate: Tokens added per second
13
        """
14
        self.capacity = capacity
15
        self.refill_rate = refill_rate
16
        self.tokens = capacity
17
        self.last_refill = time.time()
18
        self.lock = Lock()
19

20
    def acquire(self, tokens: int = 1) -> bool:
21
        """Try to acquire tokens. Returns True if successful."""
22
        with self.lock:
23
            self._refill()
24

25
            if self.tokens >= tokens:
26
                self.tokens -= tokens
27
                return True
28
            else:
29
                return False
30

31
    def _refill(self):
32
        """Refill tokens based on elapsed time"""
33
        now = time.time()
34
        elapsed = now - self.last_refill
35

36
        # Calculate tokens to add
37
        tokens_to_add = elapsed * self.refill_rate
38

39
        # Add tokens (don't exceed capacity)
40
        self.tokens = min(self.capacity, self.tokens + tokens_to_add)
41
        self.last_refill = now
42

43
    def get_available_tokens(self) -> int:
44
        """Get current number of available tokens"""
45
        with self.lock:
46
            self._refill()
47
            return int(self.tokens)
48

49
# Usage
50
rate_limiter = TokenBucket(capacity=10, refill_rate=2.0)  # 10 tokens, refill 2/sec
51

52
def handle_request():
53
    if rate_limiter.acquire():
54
        # Process request
55
        return "Request processed"
56
    else:
57
        return "Rate limit exceeded", 429

"TokenBucket.java

1
import java.util.concurrent.locks.ReentrantLock;
2

3
public class TokenBucket {
4
    private final int capacity;
5
    private final double refillRate;
6
    private double tokens;
7
    private long lastRefill;
8
    private final ReentrantLock lock = new ReentrantLock();
9

10
    public TokenBucket(int capacity, double refillRate) {
11
        this.capacity = capacity;
12
        this.refillRate = refillRate;
13
        this.tokens = capacity;
14
        this.lastRefill = System.currentTimeMillis();
15
    }
16

17
    public boolean acquire(int tokensToConsume) {
18
        lock.lock();
19
        try {
20
            refill();
21

22
            if (tokens >= tokensToConsume) {
23
                tokens -= tokensToConsume;
24
                return true;
25
            } else {
26
                return false;
27
            }
28
        } finally {
29
            lock.unlock();
30
        }
31
    }
32

33
    private void refill() {
34
        long now = System.currentTimeMillis();
35
        double elapsed = (now - lastRefill) / 1000.0;  // Convert to seconds
36

37
        // Calculate tokens to add
38
        double tokensToAdd = elapsed * refillRate;
39

40
        // Add tokens (don't exceed capacity)
41
        tokens = Math.min(capacity, tokens + tokensToAdd);
42
        lastRefill = now;
43
    }
44

45
    public int getAvailableTokens() {
46
        lock.lock();
47
        try {
48
            refill();
49
            return (int) tokens;
50
        } finally {
51
            lock.unlock();
52
        }
53
    }
54
}
55

56
// Usage
57
TokenBucket rateLimiter = new TokenBucket(10, 2.0);  // 10 tokens, refill 2/sec
58

59
public String handleRequest() {
60
    if (rateLimiter.acquire(1)) {
61
        // Process request
62
        return "Request processed";
63
    } else {
64
        return "Rate limit exceeded";  // Return 429
65
    }
66
}

2. Leaky Bucket

Section titled “2. Leaky Bucket”

Bucket holds requests. Requests leak out at fixed rate. If full, reject.

Characteristics:

✅ Smooths traffic to constant rate
✅ No bursts allowed
✅ Predictable output rate
❌ Less flexible than token bucket

Algorithm:

Initialize bucket with capacity (queue size)
Process requests at leak_rate per second
When request arrives:
- If bucket has space → add to bucket
- If bucket full → reject (429)

Leaky Bucket Implementation

Section titled “Leaky Bucket Implementation”

Python
Java

"leaky_bucket.py

1
import time
2
import queue
3
from threading import Lock, Thread
4
from typing import Optional
5

6
class LeakyBucket:
7
    """Leaky bucket rate limiter"""
8

9
    def __init__(self, capacity: int, leak_rate: float):
10
        """
11
        Args:
12
            capacity: Maximum requests in bucket
13
            leak_rate: Requests processed per second
14
        """
15
        self.capacity = capacity
16
        self.leak_rate = leak_rate
17
        self.bucket = queue.Queue(maxsize=capacity)
18
        self.lock = Lock()
19
        self.processing = False
20

21
    def start_processing(self):
22
        """Start processing requests"""
23
        if not self.processing:
24
            self.processing = True
25
            Thread(target=self._process_requests, daemon=True).start()
26

27
    def add_request(self, request) -> bool:
28
        """Try to add request to bucket. Returns True if successful."""
29
        try:
30
            self.bucket.put_nowait(request)
31
            return True
32
        except queue.Full:
33
            return False
34

35
    def _process_requests(self):
36
        """Process requests at leak rate"""
37
        interval = 1.0 / self.leak_rate  # Time between requests
38

39
        while self.processing:
40
            try:
41
                request = self.bucket.get(timeout=interval)
42
                # Process request
43
                self._handle_request(request)
44
            except queue.Empty:
45
                continue
46

47
    def _handle_request(self, request):
48
        """Handle processed request"""
49
        # Override in subclass or pass handler
50
        pass

"LeakyBucket.java

1
import java.util.concurrent.BlockingQueue;
2
import java.util.concurrent.LinkedBlockingQueue;
3
import java.util.concurrent.Executors;
4
import java.util.concurrent.ScheduledExecutorService;
5
import java.util.concurrent.TimeUnit;
6

7
public class LeakyBucket {
8
    private final int capacity;
9
    private final double leakRate;
10
    private final BlockingQueue<Request> bucket;
11
    private final ScheduledExecutorService scheduler;
12

13
    public LeakyBucket(int capacity, double leakRate) {
14
        this.capacity = capacity;
15
        this.leakRate = leakRate;
16
        this.bucket = new LinkedBlockingQueue<>(capacity);
17
        this.scheduler = Executors.newScheduledThreadPool(1);
18

19
        // Start processing
20
        startProcessing();
21
    }
22

23
    public boolean addRequest(Request request) {
24
        // Try to add request to bucket
25
        return bucket.offer(request);
26
    }
27

28
    private void startProcessing() {
29
        long interval = (long) (1000 / leakRate);  // Milliseconds between requests
30

31
        scheduler.scheduleAtFixedRate(() -> {
32
            Request request = bucket.poll();
33
            if (request != null) {
34
                handleRequest(request);
35
            }
36
        }, 0, interval, TimeUnit.MILLISECONDS);
37
    }
38

39
    private void handleRequest(Request request) {
40
        // Process request
41
    }
42
}

3. Sliding Window

Section titled “3. Sliding Window”

Tracks requests in sliding time window. More accurate than fixed window.

Characteristics:

✅ More accurate than fixed window
✅ No burst at boundaries
❌ More memory intensive (stores timestamps)
❌ More complex to implement

Sliding Window Implementation

Section titled “Sliding Window Implementation”

Python
Java

"sliding_window.py

1
import time
2
from collections import deque
3
from threading import Lock
4

5
class SlidingWindowRateLimiter:
6
    """Sliding window rate limiter"""
7

8
    def __init__(self, limit: int, window_seconds: int):
9
        """
10
        Args:
11
            limit: Maximum requests allowed
12
            window_seconds: Time window in seconds
13
        """
14
        self.limit = limit
15
        self.window_seconds = window_seconds
16
        self.requests = deque()  # Store request timestamps
17
        self.lock = Lock()
18

19
    def is_allowed(self) -> bool:
20
        """Check if request is allowed"""
21
        with self.lock:
22
            now = time.time()
23

24
            # Remove old requests outside window
25
            while self.requests and self.requests[0] < now - self.window_seconds:
26
                self.requests.popleft()
27

28
            # Check if under limit
29
            if len(self.requests) < self.limit:
30
                self.requests.append(now)
31
                return True
32
            else:
33
                return False
34

35
    def get_remaining_requests(self) -> int:
36
        """Get remaining requests in current window"""
37
        with self.lock:
38
            now = time.time()
39

40
            # Remove old requests
41
            while self.requests and self.requests[0] < now - self.window_seconds:
42
                self.requests.popleft()
43

44
            return max(0, self.limit - len(self.requests))

"SlidingWindowRateLimiter.java

1
import java.util.concurrent.ConcurrentLinkedDeque;
2
import java.util.concurrent.locks.ReentrantLock;
3

4
public class SlidingWindowRateLimiter {
5
    private final int limit;
6
    private final long windowMillis;
7
    private final ConcurrentLinkedDeque<Long> requests;
8
    private final ReentrantLock lock = new ReentrantLock();
9

10
    public SlidingWindowRateLimiter(int limit, int windowSeconds) {
11
        this.limit = limit;
12
        this.windowMillis = windowSeconds * 1000L;
13
        this.requests = new ConcurrentLinkedDeque<>();
14
    }
15

16
    public boolean isAllowed() {
17
        lock.lock();
18
        try {
19
            long now = System.currentTimeMillis();
20

21
            // Remove old requests outside window
22
            while (!requests.isEmpty() && requests.peekFirst() < now - windowMillis) {
23
                requests.pollFirst();
24
            }
25

26
            // Check if under limit
27
            if (requests.size() < limit) {
28
                requests.addLast(now);
29
                return true;
30
            } else {
31
                return false;
32
            }
33
        } finally {
34
            lock.unlock();
35
        }
36
    }
37

38
    public int getRemainingRequests() {
39
        lock.lock();
40
        try {
41
            long now = System.currentTimeMillis();
42

43
            // Remove old requests
44
            while (!requests.isEmpty() && requests.peekFirst() < now - windowMillis) {
45
                requests.pollFirst();
46
            }
47

48
            return Math.max(0, limit - requests.size());
49
        } finally {
50
            lock.unlock();
51
        }
52
    }
53
}

4. Fixed Window

Section titled “4. Fixed Window”

Divides time into fixed windows. Simple but allows bursts.

Characteristics:

✅ Simple to implement
✅ Low memory usage
❌ Allows bursts at boundaries
❌ Less accurate

Fixed Window Implementation

Section titled “Fixed Window Implementation”

Python
Java

"fixed_window.py

1
import time
2
from threading import Lock
3

4
class FixedWindowRateLimiter:
5
    """Fixed window rate limiter"""
6

7
    def __init__(self, limit: int, window_seconds: int):
8
        """
9
        Args:
10
            limit: Maximum requests per window
11
            window_seconds: Window size in seconds
12
        """
13
        self.limit = limit
14
        self.window_seconds = window_seconds
15
        self.count = 0
16
        self.window_start = time.time()
17
        self.lock = Lock()
18

19
    def is_allowed(self) -> bool:
20
        """Check if request is allowed"""
21
        with self.lock:
22
            now = time.time()
23

24
            # Check if window expired
25
            if now - self.window_start >= self.window_seconds:
26
                # Reset window
27
                self.count = 0
28
                self.window_start = now
29

30
            # Check limit
31
            if self.count < self.limit:
32
                self.count += 1
33
                return True
34
            else:
35
                return False

"FixedWindowRateLimiter.java

1
import java.util.concurrent.atomic.AtomicInteger;
2
import java.util.concurrent.atomic.AtomicLong;
3
import java.util.concurrent.locks.ReentrantLock;
4

5
public class FixedWindowRateLimiter {
6
    private final int limit;
7
    private final long windowMillis;
8
    private final AtomicInteger count = new AtomicInteger(0);
9
    private final AtomicLong windowStart = new AtomicLong(System.currentTimeMillis());
10
    private final ReentrantLock lock = new ReentrantLock();
11

12
    public FixedWindowRateLimiter(int limit, int windowSeconds) {
13
        this.limit = limit;
14
        this.windowMillis = windowSeconds * 1000L;
15
    }
16

17
    public boolean isAllowed() {
18
        lock.lock();
19
        try {
20
            long now = System.currentTimeMillis();
21

22
            // Check if window expired
23
            if (now - windowStart.get() >= windowMillis) {
24
                // Reset window
25
                count.set(0);
26
                windowStart.set(now);
27
            }
28

29
            // Check limit
30
            if (count.get() < limit) {
31
                count.incrementAndGet();
32
                return true;
33
            } else {
34
                return false;
35
            }
36
        } finally {
37
            lock.unlock();
38
        }
39
    }
40
}

Distributed Rate Limiting

Section titled “Distributed Rate Limiting”

For multiple servers, use Redis:

Python
Java

"distributed_rate_limiter.py

1
import redis
2
import time
3

4
class DistributedTokenBucket:
5
    """Distributed token bucket using Redis"""
6

7
    def __init__(self, redis_client: redis.Redis, key_prefix: str,
8
                 capacity: int, refill_rate: float):
9
        self.redis = redis_client
10
        self.key_prefix = key_prefix
11
        self.capacity = capacity
12
        self.refill_rate = refill_rate
13

14
    def is_allowed(self, identifier: str) -> bool:
15
        """Check if request is allowed for identifier"""
16
        key = f"{self.key_prefix}:{identifier}"
17
        now = time.time()
18

19
        # Use Lua script for atomic operations
20
        lua_script = """
21
        local key = KEYS[1]
22
        local capacity = tonumber(ARGV[1])
23
        local refill_rate = tonumber(ARGV[2])
24
        local now = tonumber(ARGV[3])
25

26
        local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
27
        local tokens = tonumber(bucket[1]) or capacity
28
        local last_refill = tonumber(bucket[2]) or now
29

30
        -- Refill tokens
31
        local elapsed = now - last_refill
32
        local tokens_to_add = elapsed * refill_rate
33
        tokens = math.min(capacity, tokens + tokens_to_add)
34

35
        -- Check if can consume token
36
        if tokens >= 1 then
37
            tokens = tokens - 1
38
            redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
39
            redis.call('EXPIRE', key, 3600)  -- Expire after 1 hour
40
            return 1
41
        else
42
            redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
43
            redis.call('EXPIRE', key, 3600)
44
            return 0
45
        end
46
        """
47

48
        result = self.redis.eval(lua_script, 1, key,
49
                                 self.capacity, self.refill_rate, now)
50
        return bool(result)

"DistributedRateLimiter.java

1
import redis.clients.jedis.Jedis;
2
import redis.clients.jedis.JedisPool;
3

4
public class DistributedTokenBucket {
5
    private final JedisPool jedisPool;
6
    private final String keyPrefix;
7
    private final int capacity;
8
    private final double refillRate;
9

10
    private static final String LUA_SCRIPT =
11
        "local key = KEYS[1]\n" +
12
        "local capacity = tonumber(ARGV[1])\n" +
13
        "local refill_rate = tonumber(ARGV[2])\n" +
14
        "local now = tonumber(ARGV[3])\n" +
15
        "local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')\n" +
16
        "local tokens = tonumber(bucket[1]) or capacity\n" +
17
        "local last_refill = tonumber(bucket[2]) or now\n" +
18
        "local elapsed = now - last_refill\n" +
19
        "local tokens_to_add = elapsed * refill_rate\n" +
20
        "tokens = math.min(capacity, tokens + tokens_to_add)\n" +
21
        "if tokens >= 1 then\n" +
22
        "  tokens = tokens - 1\n" +
23
        "  redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)\n" +
24
        "  redis.call('EXPIRE', key, 3600)\n" +
25
        "  return 1\n" +
26
        "else\n" +
27
        "  redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)\n" +
28
        "  redis.call('EXPIRE', key, 3600)\n" +
29
        "  return 0\n" +
30
        "end";
31

32
    public boolean isAllowed(String identifier) {
33
        try (Jedis jedis = jedisPool.getResource()) {
34
            String key = keyPrefix + ":" + identifier;
35
            long now = System.currentTimeMillis() / 1000;
36

37
            Object result = jedis.eval(LUA_SCRIPT,
38
                Collections.singletonList(key),
39
                Arrays.asList(
40
                    String.valueOf(capacity),
41
                    String.valueOf(refillRate),
42
                    String.valueOf(now)
43
                ));
44

45
            return ((Long) result) == 1L;
46
        }
47
    }
48
}

Rate Limiting Strategies

Section titled “Rate Limiting Strategies”

By Client IP

Section titled “By Client IP”

1
rate_limiter = TokenBucket(capacity=100, refill_rate=10)
2
client_ip = request.remote_addr
3

4
if not rate_limiter.is_allowed(client_ip):
5
    return "Rate limit exceeded", 429

By User ID

Section titled “By User ID”

1
user_id = get_user_id(request)
2
if not rate_limiter.is_allowed(f"user:{user_id}"):
3
    return "Rate limit exceeded", 429

By API Key

Section titled “By API Key”

1
api_key = request.headers.get('X-API-Key')
2
if not rate_limiter.is_allowed(f"key:{api_key}"):
3
    return "Rate limit exceeded", 429

Tiered Limits

Section titled “Tiered Limits”

1
# Different limits for different tiers
2
limits = {
3
    'free': TokenBucket(100, 10),
4
    'premium': TokenBucket(1000, 100),
5
    'enterprise': TokenBucket(10000, 1000),
6
}
7

8
tier = get_user_tier(user_id)
9
if not limits[tier].is_allowed(user_id):
10
    return "Rate limit exceeded", 429

HTTP Response Headers

Section titled “HTTP Response Headers”

Inform clients about rate limits:

1
HTTP/1.1 200 OK
2
X-RateLimit-Limit: 100
3
X-RateLimit-Remaining: 95
4
X-RateLimit-Reset: 1640995200

When limit exceeded:

1
HTTP/1.1 429 Too Many Requests
2
X-RateLimit-Limit: 100
3
X-RateLimit-Remaining: 0
4
X-RateLimit-Reset: 1640995200
5
Retry-After: 60

Algorithm Comparison

Section titled “Algorithm Comparison”

Algorithm	Bursts	Accuracy	Memory	Complexity
Token Bucket	✅ Yes	High	Low	Medium
Leaky Bucket	❌ No	High	Medium	Medium
Sliding Window	❌ No	Very High	High	High
Fixed Window	✅ Yes	Medium	Low	Low

Recommendation: Use Token Bucket for most cases. It’s simple, accurate, and allows bursts.

Key Takeaways

Section titled “Key Takeaways”

🪣 Token Bucket: Most Popular

Token bucket allows bursts, smooths to average rate. Most widely used algorithm.

📊 Sliding Window: Most Accurate

Sliding window is most accurate but uses more memory. Use when accuracy is critical.

🌐 Distributed: Use Redis

For multiple servers, use Redis with Lua scripts for atomic operations.

🔢 Return 429

When rate limit exceeded, return HTTP 429 with Retry-After header. Inform clients about limits.

Next Steps

Section titled “Next Steps”

Review API Gateway - rate limiting is often implemented in gateways
Learn Caching - cache rate limit data for performance
Understand Distributed Systems - distributed rate limiting challenges

📚 Related Articles

Continue learning with these related topics

Design a Payment Processor

Design a payment processor system that supports multiple payment methods (credit card, PayPal, bank transfer), manages transaction states, ensures idempotency to prevent duplicate charges, and implements retry logic for handling transient failures. Master this hard Low Level Design interview problem with step-by-step guidance, interactive UML class diagram builder, and complete solutions in Python, Java, C++, TypeScript, JavaScript, and C#.

→

Design an Elevator System

Design a smart elevator control system for a high-rise building. It needs to handle multiple elevator cars, different request strategies (SCAN, FCFS), and emergency states. Master this easy Low Level Design interview problem with step-by-step guidance, interactive UML class diagram builder, and complete solutions in Python, Java, C++, TypeScript, JavaScript, and C#.

→

Design a Feed Generator

Design a feed generator system that aggregates content from multiple sources, ranks posts based on various algorithms, and delivers personalized feeds to users. The system should support different ranking strategies, caching mechanisms, real-time feed updates, and content filtering. Master this hard Low Level Design interview problem with step-by-step guidance, interactive UML class diagram builder, and complete solutions in Python, Java, C++, TypeScript, JavaScript, and C#.

→

Design a Parking Lot - Multi Threaded

Design a multi-threaded parking lot system that handles concurrent vehicle entries and exits from multiple gates. The system must ensure thread-safe operations, prevent deadlocks, implement fair allocation mechanisms, and maintain real-time availability updates across all threads. Master this hard Low Level Design interview problem with step-by-step guidance, interactive UML class diagram builder, and complete solutions in Python, Java, C++, TypeScript, JavaScript, and C#.

→

Design a Simple Elevator System - Multiple Lifts

Design a simple elevator system that manages 2-4 elevators in a building. The system should efficiently distribute requests across elevators, balance load to minimize wait times, optimize elevator direction based on requests, and handle concurrent requests from different floors. Master this medium Low Level Design interview problem with step-by-step guidance, interactive UML class diagram builder, and complete solutions in Python, Java, C++, TypeScript, JavaScript, and C#.

→

Design a Rate Limiter

Design a rate limiter system that restricts the number of requests a user or client can make to an API within a specified time window. The system should support multiple rate limiting algorithms, handle concurrent requests efficiently, and be scalable across distributed environments. Master this medium Low Level Design interview problem with step-by-step guidance, interactive UML class diagram builder, and complete solutions in Python, Java, C++, TypeScript, JavaScript, and C#.

→

Last updated: Jan 6, 2026

Previous
Real-time Communication Next
Message Queues Fundamentals