Understanding Bottlenecks

What is a Bottleneck?

A bottleneck is the component that limits your system’s overall performance. No matter how fast other parts are, the system can only go as fast as its slowest component.

Amdahl's Law

If 80% of your processing time is in one component, making everything else infinitely fast only gives you a 5x improvement. Focus on the bottleneck!

---

Types of Bottlenecks

1. CPU Bottleneck

Symptoms: High CPU usage, slow computations

💡 Tip: Click dropdown to switch between languages

Python

Python

cpu_bottleneck.py

# CPU-bound operation blocking the event loop
def calculate_fibonacci(n: int) -> int:
    if n <= 1:
        return n
    return calculate_fibonacci(n-1) + calculate_fibonacci(n-2)

# Solution: Use efficient algorithm or offload to worker
from functools import lru_cache

@lru_cache(maxsize=1000)
def calculate_fibonacci_cached(n: int) -> int:
    if n <= 1:
        return n
    return calculate_fibonacci_cached(n-1) + calculate_fibonacci_cached(n-2)

Java

CpuBottleneck.java

// CPU-bound operation
public long calculateFibonacci(int n) {
    if (n <= 1) return n;
    return calculateFibonacci(n-1) + calculateFibonacci(n-2);
}

// Solution: Use efficient algorithm with memoization
private Map<Integer, Long> cache = new ConcurrentHashMap<>();

public long calculateFibonacciCached(int n) {
    if (n <= 1) return n;
    return cache.computeIfAbsent(n, key ->
        calculateFibonacciCached(key-1) + calculateFibonacciCached(key-2)
    );
}

TypeScript

cpu_bottleneck.ts

// CPU-bound operation blocking the event loop
function calculateFibonacci(n: number): number {
    if (n <= 1) return n;
    return calculateFibonacci(n-1) + calculateFibonacci(n-2);
}

// Solution: Use efficient algorithm with memoization
const cache = new Map<number, number>();

function calculateFibonacciCached(n: number): number {
    if (n <= 1) return n;
    if (cache.has(n)) return cache.get(n)!;
    const result = calculateFibonacciCached(n-1) + calculateFibonacciCached(n-2);
    cache.set(n, result);
    return result;
}

C++

cpu_bottleneck.cpp

#include <unordered_map>

// CPU-bound operation
long calculateFibonacci(int n) {
    if (n <= 1) return n;
    return calculateFibonacci(n-1) + calculateFibonacci(n-2);
}

// Solution: Use efficient algorithm with memoization
std::unordered_map<int, long> cache;

long calculateFibonacciCached(int n) {
    if (n <= 1) return n;
    if (cache.find(n) != cache.end()) return cache[n];
    long result = calculateFibonacciCached(n-1) + calculateFibonacciCached(n-2);
    cache[n] = result;
    return result;
}

C#

CpuBottleneck.cs

using System;
using System.Collections.Generic;

// CPU-bound operation
long CalculateFibonacci(int n) {
    if (n <= 1) return n;
    return CalculateFibonacci(n-1) + CalculateFibonacci(n-2);
}

// Solution: Use efficient algorithm with memoization
private Dictionary<int, long> cache = new Dictionary<int, long>();

long CalculateFibonacciCached(int n) {
    if (n <= 1) return n;
    if (cache.ContainsKey(n)) return cache[n];
    long result = CalculateFibonacciCached(n-1) + CalculateFibonacciCached(n-2);
    cache[n] = result;
    return result;
}

2. Memory Bottleneck

Symptoms: High memory usage, OOM errors, GC pauses

💡 Tip: Click dropdown to switch between languages

Python

Python

memory_bottleneck.py

# Loading everything into memory
def process_large_file(filename: str) -> list:
    with open(filename) as f:
        data = f.readlines()  # Loads entire file into memory!
    return [process(line) for line in data]

# Solution: Stream processing
def process_large_file_streaming(filename: str):
    with open(filename) as f:
        for line in f:  # Reads one line at a time
            yield process(line)

Java

MemoryBottleneck.java

// Loading everything into memory
public List<String> processLargeFile(String filename) throws IOException {
    List<String> lines = Files.readAllLines(Path.of(filename)); // Loads all!
    return lines.stream().map(this::process).collect(Collectors.toList());
}

// Solution: Stream processing
public Stream<String> processLargeFileStreaming(String filename) throws IOException {
    return Files.lines(Path.of(filename))  // Streams line by line
        .map(this::process);
}

TypeScript

memory_bottleneck.ts

import * as fs from 'fs';

// Loading everything into memory
function processLargeFile(filename: string): string[] {
    const data = fs.readFileSync(filename, 'utf-8').split('\n');  // Loads all!
    return data.map(process);
}

// Solution: Stream processing
import * as readline from 'readline';

async function* processLargeFileStreaming(filename: string) {
    const fileStream = fs.createReadStream(filename);
    const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity
    });

    for await (const line of rl) {
        yield process(line);  // Processes one line at a time
    }
}

C++

memory_bottleneck.cpp

#include <fstream>
#include <vector>
#include <string>

// Loading everything into memory
std::vector<std::string> processLargeFile(const std::string& filename) {
    std::ifstream file(filename);
    std::vector<std::string> lines;
    std::string line;
    while (std::getline(file, line)) {
        lines.push_back(line);  // Loads all into memory!
    }
    return lines;
}

// Solution: Stream processing
#include <fstream>
#include <string>

void processLargeFileStreaming(const std::string& filename) {
    std::ifstream file(filename);
    std::string line;
    while (std::getline(file, line)) {
        process(line);  // Processes one line at a time
    }
}

C#

MemoryBottleneck.cs

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

// Loading everything into memory
List<string> ProcessLargeFile(string filename) {
    var lines = File.ReadAllLines(filename);  // Loads all!
    return lines.Select(Process).ToList();
}

// Solution: Stream processing
IEnumerable<string> ProcessLargeFileStreaming(string filename) {
    using var reader = new StreamReader(filename);
    string line;
    while ((line = reader.ReadLine()) != null) {
        yield return Process(line);  // Processes one line at a time
    }
}

3. Database Bottleneck

Symptoms: Slow queries, connection pool exhaustion, high DB CPU

💡 Tip: Click dropdown to switch between languages

Python

Python

database_bottleneck.py

# N+1 query problem
def get_orders_with_items(user_id: str) -> list:
    orders = db.query("SELECT * FROM orders WHERE user_id = ?", user_id)
    for order in orders:
        # This runs a query for EACH order!
        order.items = db.query("SELECT * FROM items WHERE order_id = ?", order.id)
    return orders

# Solution: Use JOIN or batch query
def get_orders_with_items_optimized(user_id: str) -> list:
    return db.query("""
        SELECT o.*, i.*
        FROM orders o
        LEFT JOIN items i ON o.id = i.order_id
        WHERE o.user_id = ?
    """, user_id)

Java

DatabaseBottleneck.java

// N+1 query problem
public List<Order> getOrdersWithItems(String userId) {
    List<Order> orders = db.query("SELECT * FROM orders WHERE user_id = ?", userId);
    for (Order order : orders) {
        // This runs a query for EACH order!
        order.setItems(db.query("SELECT * FROM items WHERE order_id = ?", order.getId()));
    }
    return orders;
}

// Solution: Use JOIN or batch query
public List<Order> getOrdersWithItemsOptimized(String userId) {
    return db.query("""
        SELECT o.*, i.*
        FROM orders o
        LEFT JOIN items i ON o.id = i.order_id
        WHERE o.user_id = ?
        """, userId);
}

TypeScript

database_bottleneck.ts

// N+1 query problem
async function getOrdersWithItems(userId: string): Promise<Order[]> {
    const orders = await db.query("SELECT * FROM orders WHERE user_id = ?", userId);
    for (const order of orders) {
        // This runs a query for EACH order!
        order.items = await db.query("SELECT * FROM items WHERE order_id = ?", order.id);
    }
    return orders;
}

// Solution: Use JOIN or batch query
async function getOrdersWithItemsOptimized(userId: string): Promise<Order[]> {
    return await db.query(`
        SELECT o.*, i.*
        FROM orders o
        LEFT JOIN items i ON o.id = i.order_id
        WHERE o.user_id = ?
    `, userId);
}

C++

database_bottleneck.cpp

#include <vector>
#include <string>

// N+1 query problem
std::vector<Order> getOrdersWithItems(const std::string& userId) {
    auto orders = db->query("SELECT * FROM orders WHERE user_id = ?", userId);
    for (auto& order : orders) {
        // This runs a query for EACH order!
        order.items = db->query("SELECT * FROM items WHERE order_id = ?", order.id);
    }
    return orders;
}

// Solution: Use JOIN or batch query
std::vector<Order> getOrdersWithItemsOptimized(const std::string& userId) {
    return db->query(R"(
        SELECT o.*, i.*
        FROM orders o
        LEFT JOIN items i ON o.id = i.order_id
        WHERE o.user_id = ?
    )", userId);
}

C#

DatabaseBottleneck.cs

using System.Collections.Generic;

// N+1 query problem
List<Order> GetOrdersWithItems(string userId) {
    var orders = db.Query("SELECT * FROM orders WHERE user_id = @userId",
        new { userId });
    foreach (var order in orders) {
        // This runs a query for EACH order!
        order.Items = db.Query("SELECT * FROM items WHERE order_id = @orderId",
            new { orderId = order.Id });
    }
    return orders;
}

// Solution: Use JOIN or batch query
List<Order> GetOrdersWithItemsOptimized(string userId) {
    return db.Query<Order>(@"
        SELECT o.*, i.*
        FROM orders o
        LEFT JOIN items i ON o.id = i.order_id
        WHERE o.user_id = @userId
    ", new { userId });
}

4. Network/I/O Bottleneck

Symptoms: High network latency, waiting on external services

---

Finding Bottlenecks

Step 1: Monitor Resource Utilization

Resource	Tool	Warning Signs
CPU	top, htop, metrics	>80% sustained
Memory	free, vmstat	>90%, frequent GC
Disk	iostat, iotop	High wait times
Network	netstat, ss	Packet loss, high latency

Step 2: Profile Your Code

💡 Tip: Click dropdown to switch between languages

Python

Python

profiling.py

import cProfile
import pstats

def profile_function(func):
    """Decorator to profile a function"""
    def wrapper(*args, **kwargs):
        profiler = cProfile.Profile()
        profiler.enable()
        result = func(*args, **kwargs)
        profiler.disable()

        stats = pstats.Stats(profiler)
        stats.sort_stats('cumulative')
        stats.print_stats(10)  # Top 10 slowest

        return result
    return wrapper

@profile_function
def my_slow_function():
    # Your code here
    pass

Java

Profiling.java

// Use JVM profilers: JProfiler, YourKit, or async-profiler
// Or simple timing:
public class SimpleProfiler {
    public static <T> T profile(String name, java.util.function.Supplier<T> operation) {
        long start = System.nanoTime();
        T result = operation.get();
        long duration = System.nanoTime() - start;
        System.out.printf("%s took %.2fms%n", name, duration / 1_000_000.0);
        return result;
    }
}

// Usage
User user = SimpleProfiler.profile("getUser", () -> userService.get(id));

TypeScript

profiling.ts

function profileFunction<T extends (...args: any[]) => any>(
    func: T,
    name: string
): T {
    return ((...args: Parameters<T>) => {
        const start = performance.now();
        const result = func(...args);
        const duration = performance.now() - start;
        console.log(`${name} took ${duration.toFixed(2)}ms`);
        return result;
    }) as T;
}

// Usage
const profiledGetUser = profileFunction(userService.get.bind(userService), "getUser");
const user = profiledGetUser(id);

C++

profiling.cpp

#include <chrono>
#include <iostream>

template<typename Func>
auto profile(const std::string& name, Func func) -> decltype(func()) {
    auto start = std::chrono::high_resolution_clock::now();
    auto result = func();
    auto end = std::chrono::high_resolution_clock::now();
    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    std::cout << name << " took " << duration.count() << "ms" << std::endl;
    return result;
}

// Usage
auto user = profile("getUser", [&]() { return userService->get(id); });

C#

Profiling.cs

using System;
using System.Diagnostics;

public static class SimpleProfiler {
    public static T Profile<T>(string name, Func<T> operation) {
        var stopwatch = Stopwatch.StartNew();
        T result = operation();
        stopwatch.Stop();
        Console.WriteLine($"{name} took {stopwatch.ElapsedMilliseconds}ms");
        return result;
    }
}

// Usage
var user = SimpleProfiler.Profile("getUser", () => userService.Get(id));

Step 3: Trace Requests End-to-End

---

Common Solutions

Bottleneck Type	Solutions
CPU	Optimize algorithms, caching, horizontal scaling
Memory	Streaming, pagination, efficient data structures
Database	Indexing, query optimization, caching, read replicas
Network	Caching, compression, connection pooling
External APIs	Caching, async calls, circuit breakers, timeouts

---

Advanced: Latency vs Throughput Bottlenecks

Understanding the difference is crucial for senior engineers:

Type	Symptom	Diagnosis	Solution
Latency	High response times	Profile shows slow operations	Optimize the slow code
Throughput	Requests queue up	Resources saturated	Add capacity or optimize resource usage
Both	Slow AND queueing	Everything is red	Triage: fix biggest impact first

---

Deep Dive: Production Bottleneck Investigation

Here’s how senior engineers approach bottleneck investigation in production:

Step 1: Establish Baseline Metrics

Before optimizing, you need to know what “normal” looks like. Track these key metrics:

Category	Metrics to Track
Request Latency	P50, P95, P99 response times
Database	Query times by operation, connection pool usage
External APIs	Call durations by service, error rates
Resources	CPU, memory, disk I/O, network

Tools:

• APM (Application Performance Monitoring): Datadog, New Relic, Dynatrace
• Metrics: Prometheus + Grafana
• Distributed Tracing: Jaeger, Zipkin, AWS X-Ray

Key Principle

Instrument your code at boundaries: incoming requests, database calls, external service calls, and cache operations. This gives you visibility into where time is spent.

Step 2: Identify the Hot Path

The hot path is the code that runs most frequently or consumes most resources:

Pareto Principle (80/20 Rule)

80% of performance problems come from 20% of the code. Don’t waste time optimizing code that rarely runs. Profile first, then optimize the hot path.

---

Real-World Examples

Example 1: Facebook News Feed Database Bottleneck

Company: Meta (Facebook)

Scenario: Facebook News Feed was loading slowly for users. Investigation revealed a database bottleneck causing high latency.

Implementation: Identified and fixed N+1 query problem:

Why This Matters:

• Scale: Billions of users, trillions of posts
• Impact: 48x latency reduction
• User Experience: Faster feed loading increases engagement
• Result: Reduced database load by 98%

Real-World Impact:

• Queries: Reduced from 81 to 1 query per feed load
• Latency: 2.4 seconds → 50ms (48x improvement)
• Database Load: 98% reduction in queries

---

Example 2: Twitter Timeline CPU Bottleneck

Company: Twitter (now X)

Scenario: Timeline generation was slow during peak hours. CPU usage was maxed out, causing high latency.

Implementation: Identified CPU bottleneck in ranking algorithm:

Why This Matters:

• Scale: Millions of timeline requests per second
• Impact: 20-50x latency reduction
• Resource Usage: CPU reduced from 100% to 30%
• Result: Handles 10x more traffic with same hardware

Real-World Impact:

• Latency: 2-5 seconds → 100-200ms (20-50x improvement)
• CPU Usage: 100% → 30% (70% reduction)
• Capacity: 10x more traffic handled

---

Example 3: Netflix Streaming Memory Bottleneck

Company: Netflix

Scenario: Video encoding service was running out of memory when processing large video files. OOM errors caused encoding failures.

Implementation: Fixed memory bottleneck with streaming:

Why This Matters:

• Scale: Thousands of videos encoded daily
• Impact: Eliminated OOM errors, 2x throughput increase
• Reliability: Encoding no longer fails due to memory
• Result: Can process larger videos with same hardware

Real-World Impact:

• Memory Usage: 100% → 3% (97% reduction)
• Reliability: OOM errors eliminated
• Throughput: 2x increase in encoding speed

---

Example 4: Amazon Checkout Network Bottleneck

Company: Amazon

Scenario: Checkout page was slow during Prime Day. Investigation revealed external payment API was the bottleneck.

Implementation: Fixed network bottleneck with caching and async processing:

Why This Matters:

• Scale: 100K checkout requests per second during Prime Day
• Impact: 10x latency reduction, 15% conversion increase
• User Experience: Faster checkout increases conversions
• Result: Handles traffic spikes without degradation

Real-World Impact:

• Latency: 500ms → 50ms (10x improvement)
• Conversion Rate: +15% increase
• Revenue: Millions in additional revenue during Prime Day

---

Real-World Case Study: E-Commerce Checkout Bottleneck

Situation: Checkout page takes 8 seconds to load during sales events.

Investigation Process

Step 1: Add timing instrumentation to each component

Step 2: Diagnose the root cause

The inventory service was making one database query per item:

• Cart with 100 items = 100 database queries
• Each query ~60ms = 6 seconds total

Step 3: Fix with batch query

-- BEFORE: N+1 queries (100 queries for 100 items)
SELECT * FROM inventory WHERE sku = 'SKU001';
SELECT * FROM inventory WHERE sku = 'SKU002';
-- ... 98 more queries

-- AFTER: Single batch query
SELECT * FROM inventory WHERE sku IN ('SKU001', 'SKU002', ...);

Results

Metric	Before	After	Improvement
P50 Latency	6.2s	0.4s	93% reduction
P99 Latency	12s	0.8s	93% reduction
DB Queries	103	5	95% reduction
Conversion Rate	2.1%	3.8%	81% increase

Business Impact

This is why bottleneck optimization matters: A 93% latency reduction led to 81% more revenue from the checkout flow. Performance is a feature.

---

Key Takeaways

Remember

• Bottleneck = the component limiting overall throughput
• Amdahl’s Law = optimize the bottleneck first for maximum impact
• Common bottlenecks: CPU, Memory, Database, Network, External APIs
• Finding bottlenecks: Monitor → Profile → Trace
• Measure before and after to validate improvements

---

What’s Next?

You’ve completed the Foundations section! You now understand:

• Why system design matters for LLD
• Scalability fundamentals
• Latency and throughput metrics
• How to find and fix bottlenecks

Continue your journey: Explore other HLD Concepts sections to deepen your understanding of distributed systems.

Practice

Profile one of your existing services. Find the bottleneck and see if you can improve it by at least 50%!