Scalability Fundamentals

What is Scalability?

Scalability is a system’s ability to handle increased load by adding resources. A scalable system can grow to accommodate more users, more data, or more transactions without degrading performance.

Key Insight

A system is scalable if performance remains acceptable as load increases. The goal isn’t just to handle more load—it’s to handle more load efficiently.

---

Two Approaches to Scaling

Vertical Scaling (Scale Up)

Add more power to existing machines - bigger CPU, more RAM, faster disks.

Pros:

• Simple - no code changes needed
• No distributed system complexity
• Strong consistency is easy

Cons:

• Hardware limits (can’t add infinite CPU)
• Expensive at high end
• Single point of failure
• Downtime during upgrades

Horizontal Scaling (Scale Out)

Add more machines - distribute the load across multiple servers.

Pros:

• No hardware limits (add infinite machines)
• Cost-effective (use commodity hardware)
• Built-in redundancy
• Gradual scaling

Cons:

• Distributed system complexity
• Data consistency challenges
• Code must be designed for it
• Network overhead

---

Comparison: Vertical vs Horizontal

Aspect	Vertical Scaling	Horizontal Scaling
Approach	Bigger machine	More machines
Limit	Hardware ceiling	Theoretically unlimited
Cost	Expensive at scale	Cost-effective
Complexity	Simple	Complex
Downtime	Required for upgrades	Zero-downtime possible
Failure	Single point of failure	Redundancy built-in
Code changes	Usually none	May require redesign

Industry Trend

Most large-scale systems use horizontal scaling. Companies like Google, Amazon, and Netflix run on thousands of commodity servers rather than a few supercomputers.

---

What Makes Code Horizontally Scalable?

This is where LLD meets HLD. Your class design determines whether your system can scale horizontally.

The Problem: Stateful Services

💡 Tip: Click dropdown to switch between languages

Python

Python

stateful_service.py

class ShoppingCartService:
    """NOT horizontally scalable - stores state in memory"""

    def __init__(self):
        self.carts = {}  # user_id -> cart items

    def add_item(self, user_id: str, item: str):
        if user_id not in self.carts:
            self.carts[user_id] = []
        self.carts[user_id].append(item)

    def get_cart(self, user_id: str) -> list:
        return self.carts.get(user_id, [])

# Problem: If user's next request goes to a different server,
# their cart is empty!

Java

StatefulService.java

import java.util.*;

public class ShoppingCartService {
    // NOT horizontally scalable - stores state in memory
    private Map<String, List<String>> carts = new HashMap<>();

    public void addItem(String userId, String item) {
        carts.computeIfAbsent(userId, k -> new ArrayList<>()).add(item);
    }

    public List<String> getCart(String userId) {
        return carts.getOrDefault(userId, Collections.emptyList());
    }
}

// Problem: If user's next request goes to a different server,
// their cart is empty!

TypeScript

stateful_service.ts

class ShoppingCartService {
    // NOT horizontally scalable - stores state in memory
    private carts: Map<string, string[]> = new Map();

    addItem(userId: string, item: string): void {
        if (!this.carts.has(userId)) {
            this.carts.set(userId, []);
        }
        this.carts.get(userId)!.push(item);
    }

    getCart(userId: string): string[] {
        return this.carts.get(userId) || [];
    }
}

// Problem: If user's next request goes to a different server,
// their cart is empty!

C++

stateful_service.cpp

#include <unordered_map>
#include <vector>
#include <string>

class ShoppingCartService {
private:
    // NOT horizontally scalable - stores state in memory
    std::unordered_map<std::string, std::vector<std::string>> carts;

public:
    void addItem(const std::string& userId, const std::string& item) {
        carts[userId].push_back(item);
    }

    std::vector<std::string> getCart(const std::string& userId) const {
        auto it = carts.find(userId);
        return it != carts.end() ? it->second : std::vector<std::string>();
    }
};

// Problem: If user's next request goes to a different server,
// their cart is empty!

C#

StatefulService.cs

using System;
using System.Collections.Generic;

public class ShoppingCartService {
    // NOT horizontally scalable - stores state in memory
    private Dictionary<string, List<string>> carts = new Dictionary<string, List<string>>();

    public void AddItem(string userId, string item) {
        if (!carts.ContainsKey(userId)) {
            carts[userId] = new List<string>();
        }
        carts[userId].Add(item);
    }

    public List<string> GetCart(string userId) {
        return carts.ContainsKey(userId) ? carts[userId] : new List<string>();
    }
}

// Problem: If user's next request goes to a different server,
// their cart is empty!

The Solution: Stateless Services

The solution is to externalize all state to a shared store (Redis, database, etc.):

• No instance variables holding user/session data
• All state lives externally in Redis, database, or similar
• Any server can handle any request because they all access the same shared state

💡 Tip: Click dropdown to switch between languages

Python

Python

stateless_service.py

import redis

class ShoppingCartService:
    """Horizontally scalable - no local state"""

    def __init__(self, redis_client):
        self.redis = redis_client  # External storage

    def add_item(self, user_id: str, item: str) -> None:
        self.redis.rpush(f"cart:{user_id}", item)

    def get_cart(self, user_id: str) -> list:
        return self.redis.lrange(f"cart:{user_id}", 0, -1)

# Any server can handle any request!

Java

StatelessService.java

import redis.clients.jedis.Jedis;
import java.util.List;

public class ShoppingCartService {
    private final Jedis redis;  // External storage

    public ShoppingCartService(Jedis redis) {
        this.redis = redis;
    }

    public void addItem(String userId, String item) {
        redis.rpush("cart:" + userId, item);
    }

    public List<String> getCart(String userId) {
        return redis.lrange("cart:" + userId, 0, -1);
    }
}

// Any server can handle any request!

TypeScript

stateless_service.ts

import Redis from 'ioredis';

class ShoppingCartService {
    private redis: Redis;  // External storage

    constructor(redis: Redis) {
        this.redis = redis;
    }

    async addItem(userId: string, item: string): Promise<void> {
        await this.redis.rpush(`cart:${userId}`, item);
    }

    async getCart(userId: string): Promise<string[]> {
        return await this.redis.lrange(`cart:${userId}`, 0, -1);
    }
}

// Any server can handle any request!

C++

stateless_service.cpp

#include <hiredis/hiredis.h>
#include <vector>
#include <string>

class ShoppingCartService {
private:
    redisContext* redis;  // External storage

public:
    ShoppingCartService(redisContext* redis) : redis(redis) {}

    void addItem(const std::string& userId, const std::string& item) {
        std::string key = "cart:" + userId;
        redisCommand(redis, "RPUSH %s %s", key.c_str(), item.c_str());
    }

    std::vector<std::string> getCart(const std::string& userId) {
        std::string key = "cart:" + userId;
        redisReply* reply = (redisReply*)redisCommand(redis, "LRANGE %s 0 -1", key.c_str());
        std::vector<std::string> items;
        for (size_t i = 0; i < reply->elements; i++) {
            items.push_back(reply->element[i]->str);
        }
        freeReplyObject(reply);
        return items;
    }
};

// Any server can handle any request!

C#

StatelessService.cs

using StackExchange.Redis;
using System.Collections.Generic;
using System.Linq;

public class ShoppingCartService {
    private IDatabase redis;  // External storage

    public ShoppingCartService(IDatabase redis) {
        this.redis = redis;
    }

    public void AddItem(string userId, string item) {
        redis.ListRightPush($"cart:{userId}", item);
    }

    public List<string> GetCart(string userId) {
        return redis.ListRange($"cart:{userId}", 0, -1)
            .Select(v => v.ToString())
            .ToList();
    }
}

// Any server can handle any request!

---

Scalability Dimensions

Systems can be scaled along different dimensions:

1. Load Scalability

Handle more requests per second

2. Data Scalability

Handle more data

3. Geographic Scalability

Serve users globally with low latency

---

Design Patterns for Scalability

Pattern 1: Stateless Services

Key Principle: All state lives externally (database, cache, message queue). The service itself stores nothing between requests.

This allows you to run any number of service instances and route requests to any of them.

Pattern 2: Idempotent Operations

Key Principle: Operations that can be safely retried without side effects. Critical for distributed systems where network failures cause retries.

Implementation: Store results keyed by a unique idempotency key. Before processing, check if the key exists and return the cached result.

Pattern 3: Async Processing

Key Principle: Move slow, non-critical operations out of the request path using message queues.

Result: Users get fast responses. Slow operations (email, analytics, notifications) happen in the background without blocking.

---

Real-World Examples

Example 1: Netflix Horizontal Scaling

Company: Netflix

Scenario: Netflix streams content to 200+ million subscribers globally. The system must handle massive traffic spikes during peak hours and popular show releases.

Implementation: Uses horizontal scaling with stateless microservices:

Why Horizontal Scaling?

• Traffic Spikes: Handle 10x traffic during popular releases
• Global Distribution: Serve users from nearest data center
• Cost-Effective: Use commodity hardware, scale down during off-peak
• Result: Handles billions of requests daily with 99.99% availability

Real-World Impact:

• Scale: 200+ million subscribers, 15% of global internet bandwidth
• Instances: Thousands of microservice instances, auto-scaling
• Availability: 99.99% uptime despite massive scale

---

Example 2: Amazon E-Commerce Platform

Company: Amazon

Scenario: Amazon handles millions of orders daily with massive traffic spikes during events like Prime Day and Black Friday.

Implementation: Uses horizontal scaling with stateless services and externalized state:

Why Stateless Design?

• Auto-Scaling: Add instances instantly during traffic spikes
• Fault Tolerance: Failed instances don’t lose data
• Load Distribution: Any instance can handle any request
• Result: Handles 10x traffic spikes without downtime

Real-World Impact:

• Scale: Millions of orders per day, billions of page views
• Spike Handling: 10x traffic increase during Prime Day
• Availability: 99.99% uptime during peak events

---

Example 3: Google Search Horizontal Scaling

Company: Google

Scenario: Google Search handles billions of queries daily. The system must respond in milliseconds while handling massive concurrent load.

Implementation: Uses massive horizontal scaling with stateless search clusters:

Why Massive Horizontal Scaling?

• Query Volume: Billions of queries require massive capacity
• Low Latency: Geographic distribution reduces latency
• Redundancy: Multiple clusters ensure availability
• Result: Sub-100ms response times at massive scale

Real-World Impact:

• Scale: 8.5+ billion queries per day
• Performance: < 100ms average response time
• Availability: 99.99% uptime globally

---

Example 4: Uber’s Ride Matching System

Company: Uber

Scenario: Uber matches riders with drivers in real-time across hundreds of cities. The system must handle millions of concurrent requests.

Implementation: Uses horizontal scaling with stateless matching services:

Why Stateless Design?

• Real-Time Matching: Low latency required for good UX
• Geographic Distribution: Serve users from nearest region
• Fault Tolerance: Instance failures don’t affect matching
• Result: Sub-second matching times at massive scale

Real-World Impact:

• Scale: Millions of rides per day globally
• Latency: < 1 second matching time
• Availability: 99.9% uptime despite massive concurrent load

---

Scalability Checklist for LLD

When designing classes, ask yourself:

Question	Why It Matters
Does this class store state in instance variables?	Prevents horizontal scaling
Can multiple instances run simultaneously?	Required for scaling out
Are operations idempotent?	Enables safe retries
What happens if this operation is slow?	May need async processing
Does this depend on local resources (files, memory)?	Won’t work across servers
How does this handle concurrent requests?	Thread safety concerns

---

Key Takeaways

Remember

• Scalability = handling growth efficiently
• Vertical scaling = bigger machines (simple but limited)
• Horizontal scaling = more machines (complex but unlimited)
• Stateless services are key to horizontal scaling
• External state (Redis, databases) enables shared state
• Idempotent operations enable safe retries
• Async processing offloads non-critical work

---

What’s Next?

Understanding scalability is just the beginning. Next, we’ll dive into measuring system performance:

Next up: Latency and Throughput - Learn the key metrics that define system performance.

Design Exercise

Take one of your existing classes and evaluate: Is it horizontally scalable? What changes would make it scalable?