Distributed Caching

The Single Server Problem

When your application runs on one server, caching is simple - just use local memory. But what happens when you scale to multiple servers?

The problem: Each server has its own cache. Data cached on Server 1 isn’t available on Server 2. You’re wasting memory and getting inconsistent results.

The solution: Distributed caching - a shared cache accessible by all servers.

---

What is Distributed Caching?

Distributed caching means using a cache that runs on separate servers and is shared by all your application servers.

Benefits

Benefit	Description
Shared Cache	All servers see the same cached data
Larger Capacity	Sum of all cache servers, not limited to one machine
High Availability	Cache survives individual server failures
Consistency	Updates visible to all servers immediately
Memory Efficiency	One copy instead of N copies

---

Redis vs Memcached

The two most popular distributed caching solutions:

Redis (Remote Dictionary Server)

Feature-rich in-memory data store. More than just a cache.

Redis Strengths:

• Rich data structures (lists, sets, sorted sets, hashes)
• Persistence options (RDB, AOF)
• Advanced features (pub/sub, transactions, Lua scripting)
• Atomic operations
• Built-in replication and clustering

Redis Use Cases:

• Caching
• Session storage
• Real-time leaderboards
• Message queues
• Rate limiting
• Distributed locks

Memcached

Simple, fast key-value store. Pure caching solution.

Memcached Strengths:

• Simpler than Redis
• Lower memory overhead
• Faster for simple key-value operations
• Good for pure caching needs

Memcached Use Cases:

• Simple caching
• Session storage (if persistence not needed)
• When you only need key-value

Comparison

Feature	Redis	Memcached
Data Structures	Rich (strings, lists, sets, etc.)	Key-value only
Persistence	Yes (RDB, AOF)	No
Performance	Fast	Faster (simpler)
Memory Efficiency	Higher overhead	Lower overhead
Replication	Built-in	Client-side sharding
Use Case	Feature-rich caching	Simple caching

Which to Choose?

• Choose Redis if you need rich data structures, persistence, or advanced features
• Choose Memcached if you only need simple key-value caching and want maximum performance

Most modern applications choose Redis for its flexibility and feature set.

---

Real-World Examples

Major companies use distributed caching to scale their applications:

Redis: Twitter’s Timeline Cache

The Challenge: Twitter serves billions of timeline requests per day. Each user’s timeline is unique and requires complex queries.

The Solution: Twitter uses Redis clusters:

• Scale: 100+ Redis clusters, millions of keys
• Data: User timelines, tweet metadata, follower lists
• Pattern: Cache-aside with Redis
• TTL: 5 minutes for timelines, 1 hour for user profiles

Why Redis? Twitter needs rich data structures (lists for timelines, sets for followers, sorted sets for trending). Redis provides these natively.

Architecture:

• Application servers → Redis cluster (sharded by user ID)
• Cache miss → Query database → Store in Redis
• Replication: Each Redis node has replicas for high availability

Impact: 70% of timeline requests served from Redis. Reduced database load by 80%. Timeline load time: 50ms (Redis) vs 500ms (database).

Memcached: Facebook’s Feed Cache

The Challenge: Facebook’s news feed serves personalized content to billions of users. Simple key-value caching needed.

The Solution: Facebook uses Memcached extensively:

• Scale: Thousands of Memcached servers, petabytes of cached data
• Data: Feed content, user profiles, friend lists
• Pattern: Cache-aside with Memcached
• Sharding: Consistent hashing across Memcached servers

Why Memcached? Facebook needs simple, fast key-value storage. Memcached is optimized for this use case - simpler than Redis, faster for pure caching.

Architecture:

• Application servers → Memcached pool (consistent hashing)
• Cache miss → Query database → Store in Memcached
• No persistence (acceptable - data can be recomputed)

Impact: 99% of feed requests served from Memcached. Database queries reduced by 99%. Feed load time: 10ms (Memcached) vs 200ms (database).

Redis vs Memcached: Instagram’s Use Case

The Challenge: Instagram needs both simple caching and complex data structures for different features.

The Solution: Instagram uses both:

• Memcached: Simple caching (user sessions, simple counters)
• Redis: Complex data structures (feed timelines, follower lists, real-time features)

Why Both? Different features have different needs:

• User sessions: Simple key-value → Memcached (faster, simpler)
• Feed timelines: Need lists, sorted sets → Redis (richer features)
• Real-time features: Need pub/sub → Redis (Memcached doesn’t support)

Example:

• User login: Session stored in Memcached (simple, fast)
• Feed generation: Timeline stored in Redis (needs list operations)
• Real-time notifications: Pub/sub via Redis (Memcached can’t do this)

Impact: Right tool for each job. Optimized performance and cost.

Amazon ElastiCache: E-commerce Platform

The Challenge: E-commerce platform needs distributed caching but doesn’t want to manage infrastructure.

The Solution: Uses Amazon ElastiCache (managed Redis):

• Scale: Auto-scales based on load
• Data: Product catalog, shopping carts, session data
• Pattern: Read-through with ElastiCache
• High Availability: Multi-AZ deployment with automatic failover

Why Managed Service? Focus on application logic, not infrastructure. ElastiCache handles scaling, backups, monitoring.

Architecture:

• Application → ElastiCache (managed Redis cluster)
• Automatic failover if primary fails
• Automatic backups and point-in-time recovery

Impact: Reduced operational overhead by 90%. Product pages load 5x faster. Handles Black Friday traffic spikes automatically.

Netflix: Multi-Region Caching

The Challenge: Netflix serves content globally. Users in different regions need low latency.

The Solution: Netflix uses distributed caching across regions:

• Scale: Redis clusters in each AWS region
• Data: Video metadata, user preferences, recommendations
• Pattern: Cache-aside with regional Redis clusters
• Consistency: Eventual consistency across regions (acceptable for metadata)

Why Multi-Region? Latency matters for video streaming. Users in Asia get data from Asian Redis cluster (20ms) instead of US cluster (200ms).

Architecture:

• Regional application servers → Regional Redis cluster
• Cache miss → Regional database → Store in regional Redis
• Cross-region replication for popular content

Impact: Global latency reduced from 200ms to 20ms average. 95% cache hit rate. Reduced bandwidth costs by 80%.

---

Key Takeaways from Real Examples

1. Redis for rich data - When you need lists, sets, sorted sets, pub/sub
2. Memcached for simplicity - When you only need simple key-value caching
3. Managed services - Use ElastiCache, Azure Cache, etc. to reduce operational overhead
4. Multi-region - Deploy caches in multiple regions for global applications
5. Hybrid approaches - Many companies use both Redis and Memcached for different use cases

---

Distributed Cache Architecture

Basic Setup

Consistency Challenges

Problem: When Server 1 updates cache, how do Servers 2 and 3 know?

Solutions:

1. Write-Through to Shared Cache

   • All writes go to distributed cache
   • All servers read from same cache
   • Ensures consistency

2. Cache Invalidation

   • When data updated, invalidate cache
   • Next read fetches fresh data
   • More on this in next lesson

3. Short TTL

   • Use short expiration times
   • Accepts eventual consistency
   • Simple but may have stale data

---

LLD Connection: Designing Cache Clients

At the code level, you need to design cache client wrappers that abstract Redis/Memcached:

Cache Interface Abstraction

Python

cache_client.py

from abc import ABC, abstractmethod
from typing import Optional, Any
import redis
import memcache

class CacheClient(ABC):
    @abstractmethod
    def get(self, key: str) -> Optional[Any]:
        pass

    @abstractmethod
    def set(self, key: str, value: Any, ttl: int = 300) -> bool:
        pass

    @abstractmethod
    def delete(self, key: str) -> bool:
        pass

class RedisCacheClient(CacheClient):
    def __init__(self, host: str = 'localhost', port: int = 6379):
        self.client = redis.Redis(host=host, port=port, decode_responses=True)

    def get(self, key: str) -> Optional[Any]:
        try:
            return self.client.get(key)
        except redis.RedisError:
            # Handle failure gracefully
            return None

    def set(self, key: str, value: Any, ttl: int = 300) -> bool:
        try:
            return self.client.setex(key, ttl, value)
        except redis.RedisError:
            return False

    def delete(self, key: str) -> bool:
        try:
            return bool(self.client.delete(key))
        except redis.RedisError:
            return False

class MemcachedCacheClient(CacheClient):
    def __init__(self, servers: list = None):
        self.client = memcache.Client(servers or ['127.0.0.1:11211'])

    def get(self, key: str) -> Optional[Any]:
        try:
            return self.client.get(key)
        except Exception:
            return None

    def set(self, key: str, value: Any, ttl: int = 300) -> bool:
        try:
            return self.client.set(key, value, time=ttl)
        except Exception:
            return False

    def delete(self, key: str) -> bool:
        try:
            return self.client.delete(key)
        except Exception:
            return False

# Usage - application code doesn't care about implementation
class UserService:
    def __init__(self, cache: CacheClient):
        self.cache = cache

    def get_user(self, user_id: int):
        # Cache-aside pattern
        cache_key = f"user:{user_id}"
        user = self.cache.get(cache_key)

        if user:
            return user

        # Cache miss - fetch from DB
        user = self._fetch_from_db(user_id)

        # Store in cache
        if user:
            self.cache.set(cache_key, user, ttl=300)

        return user

Java

CacheClient.java

import java.util.Optional;

interface CacheClient {
    Optional<String> get(String key);
    boolean set(String key, String value, int ttlSeconds);
    boolean delete(String key);
}

class RedisCacheClient implements CacheClient {
    private final Jedis jedis;

    public RedisCacheClient(String host, int port) {
        this.jedis = new Jedis(host, port);
    }

    public Optional<String> get(String key) {
        try {
            String value = jedis.get(key);
            return Optional.ofNullable(value);
        } catch (Exception e) {
            // Handle failure gracefully
            return Optional.empty();
        }
    }

    public boolean set(String key, String value, int ttlSeconds) {
        try {
            return "OK".equals(jedis.setex(key, ttlSeconds, value));
        } catch (Exception e) {
            return false;
        }
    }

    public boolean delete(String key) {
        try {
            return jedis.del(key) > 0;
        } catch (Exception e) {
            return false;
        }
    }
}

class MemcachedCacheClient implements CacheClient {
    private final MemcachedClient client;

    public MemcachedCacheClient(String host, int port) {
        this.client = new MemcachedClient(
            new InetSocketAddress(host, port));
    }

    public Optional<String> get(String key) {
        try {
            return Optional.ofNullable((String) client.get(key));
        } catch (Exception e) {
            return Optional.empty();
        }
    }

    public boolean set(String key, String value, int ttlSeconds) {
        try {
            return client.set(key, ttlSeconds, value).get();
        } catch (Exception e) {
            return false;
        }
    }

    public boolean delete(String key) {
        try {
            return client.delete(key).get();
        } catch (Exception e) {
            return false;
        }
    }
}

// Usage - application code doesn't care about implementation
class UserService {
    private final CacheClient cache;

    public UserService(CacheClient cache) {
        this.cache = cache;
    }

    public Optional<User> getUser(int userId) {
        // Cache-aside pattern
        String cacheKey = "user:" + userId;
        Optional<String> cached = cache.get(cacheKey);

        if (cached.isPresent()) {
            return Optional.of(deserialize(cached.get()));
        }

        // Cache miss - fetch from DB
        Optional<User> user = fetchFromDb(userId);

        // Store in cache
        user.ifPresent(u -> cache.set(cacheKey, serialize(u), 300));

        return user;
    }
}

Connection Pooling

Important: Don’t create new connections for each request. Use connection pooling:

Python

connection_pool.py

import redis
from redis.connection import ConnectionPool

# Create connection pool
pool = ConnectionPool(
    host='localhost',
    port=6379,
    max_connections=50,  # Max connections in pool
    decode_responses=True
)

# Reuse pool across requests
class CacheService:
    def __init__(self):
        self.redis = redis.Redis(connection_pool=pool)

    def get(self, key: str):
        return self.redis.get(key)

Java

ConnectionPool.java

import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;

// Create connection pool
JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(50);  // Max connections
poolConfig.setMaxIdle(10);   // Max idle connections

JedisPool pool = new JedisPool(
    poolConfig, "localhost", 6379);

// Reuse pool across requests
class CacheService {
    private final JedisPool pool;

    public CacheService(JedisPool pool) {
        this.pool = pool;
    }

    public String get(String key) {
        try (Jedis jedis = pool.getResource()) {
            return jedis.get(key);
        }
    }
}

Retry Logic and Circuit Breaker

Handle cache failures gracefully:

Python

cache_with_retry.py

import time
from typing import Callable, Optional, Any

class CacheWithRetry:
    def __init__(self, cache: CacheClient, max_retries: int = 3):
        self.cache = cache
        self.max_retries = max_retries

    def get_with_retry(self, key: str) -> Optional[Any]:
        for attempt in range(self.max_retries):
            try:
                return self.cache.get(key)
            except Exception as e:
                if attempt == self.max_retries - 1:
                    # Last attempt failed - return None (cache miss)
                    return None
                # Exponential backoff
                time.sleep(2 ** attempt)
        return None

Java

CacheWithRetry.java

import java.util.Optional;
import java.util.function.Supplier;

class CacheWithRetry {
    private final CacheClient cache;
    private final int maxRetries;

    public CacheWithRetry(CacheClient cache, int maxRetries) {
        this.cache = cache;
        this.maxRetries = maxRetries;
    }

    public Optional<String> getWithRetry(String key) {
        for (int attempt = 0; attempt < maxRetries; attempt++) {
            try {
                return cache.get(key);
            } catch (Exception e) {
                if (attempt == maxRetries - 1) {
                    // Last attempt failed - return empty (cache miss)
                    return Optional.empty();
                }
                // Exponential backoff
                try {
                    Thread.sleep((long) Math.pow(2, attempt) * 1000);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    return Optional.empty();
                }
            }
        }
        return Optional.empty();
    }
}

---

Sharding and Partitioning

For very large caches, shard data across multiple cache nodes:

Sharding Strategy:

• Hash key to determine which shard
• Distribute load across nodes
• Each shard handles subset of keys

---

Key Takeaways

🌐 Shared Cache

Distributed caching provides shared cache accessible by all application servers.

🔴 Redis vs Memcached

Redis = feature-rich, Memcached = simple and fast. Choose based on needs.

🏗️ Abstract Implementation

Design cache interfaces that abstract Redis/Memcached. Makes switching easier.

🔌 Connection Pooling

Always use connection pooling. Don’t create connections per request.

---

Next Steps

• Master Cache Invalidation - the hardest problem in computer science
• Understand CDN & Edge Caching - caching at the edge for global users