Latency and Throughput

The Two Pillars of Performance

When measuring system performance, two metrics matter most:

Simple Analogy

Think of a highway:

• Latency = How long it takes ONE car to travel from A to B
• Throughput = How many cars pass through per hour

A highway can have low latency (cars go fast) but low throughput (only one lane). Or high throughput (many lanes) but high latency (traffic jams).

---

Understanding Latency

Latency is the time between sending a request and receiving a response.

Latency Breakdown

Every request goes through multiple stages:

Types of Latency

Type	Description	Typical Values
Network Latency	Time for data to travel over network	1-100ms
Processing Latency	Time for server to process request	1-50ms
Database Latency	Time for database queries	1-10ms
Queue Latency	Time spent waiting in queues	0-1000ms+

---

Percentile Latencies: P50, P95, P99

Average latency can be misleading. Percentiles give a better picture:

Why P99 Matters

Scale Changes Everything

At scale, even small percentages affect many users:

Daily Requests	P99 Affected Users
100,000	1,000 users
1,000,000	10,000 users
100,000,000	1,000,000 users

1 million users experiencing slow responses is a big problem!

Real-World Example: E-Commerce Checkout Latency

Company: Amazon, eBay, Shopify

Scenario: Checkout pages must load quickly to prevent cart abandonment. Even small latency increases can significantly impact conversion rates.

Implementation: Uses parallel API calls and caching:

---

Understanding Throughput

Throughput is the amount of work done per unit of time.

Common Throughput Metrics

Metric	Description	Example
RPS	Requests per second	10,000 RPS
TPS	Transactions per second	5,000 TPS
QPS	Queries per second	50,000 QPS
Bandwidth	Data transferred per second	1 Gbps

Throughput Calculation

Throughput is calculated as:

Throughput = Total Requests / Time Period

Example: If your server handles 10,000 requests in 10 seconds, your throughput is 1,000 RPS.

Key considerations:

• Measure over time - instantaneous measurements fluctuate
• Track success rate - failed requests count against effective throughput
• Monitor under load - throughput often drops when the system is stressed

---

Latency vs Throughput: The Trade-off

They’re related but independent:

Little’s Law

A fundamental relationship:

Average Concurrent Requests = Throughput × Average Latency

Example:

• Throughput: 1,000 RPS
• Average Latency: 100ms = 0.1 seconds
• Concurrent Requests: 1,000 × 0.1 = 100 requests in flight

Why This Matters

If you need 1,000 RPS with 100ms latency, your server needs to handle 100 concurrent connections. This affects thread pool sizes, connection limits, and resource planning.

---

How Method Design Impacts Performance

Your code decisions directly affect latency and throughput:

Bad: Sequential Operations

💡 Tip: Click dropdown to switch between languages

Python

Python

sequential_bad.py

class OrderService:
    """Sequential calls - high latency"""

    def get_order_details(self, order_id: str) -> OrderDetails:
        # Each call waits for the previous one
        order = self.db.get_order(order_id)        # 10ms
        user = self.user_service.get(order.user_id)  # 20ms
        items = self.inventory.get_items(order.items)  # 15ms
        shipping = self.shipping.get_status(order_id)  # 25ms

        # Total: 10 + 20 + 15 + 25 = 70ms
        return OrderDetails(order, user, items, shipping)

Java

SequentialBad.java

public class OrderService {
    // Sequential calls - high latency

    public OrderDetails getOrderDetails(String orderId) {
        // Each call waits for the previous one
        Order order = db.getOrder(orderId);           // 10ms
        User user = userService.get(order.getUserId()); // 20ms
        List<Item> items = inventory.getItems(order.getItems()); // 15ms
        ShippingStatus shipping = shippingService.getStatus(orderId); // 25ms

        // Total: 10 + 20 + 15 + 25 = 70ms
        return new OrderDetails(order, user, items, shipping);
    }
}

TypeScript

sequential_bad.ts

class OrderService {
    // Sequential calls - high latency

    async getOrderDetails(orderId: string): Promise<OrderDetails> {
        // Each call waits for the previous one
        const order = await this.db.getOrder(orderId);        // 10ms
        const user = await this.userService.get(order.userId);  // 20ms
        const items = await this.inventory.getItems(order.items);  // 15ms
        const shipping = await this.shipping.getStatus(orderId);  // 25ms

        // Total: 10 + 20 + 15 + 25 = 70ms
        return new OrderDetails(order, user, items, shipping);
    }
}

C++

sequential_bad.cpp

class OrderService {
public:
    OrderDetails getOrderDetails(const std::string& orderId) {
        // Each call waits for the previous one
        Order order = db->getOrder(orderId);        // 10ms
        User user = userService->get(order.getUserId());  // 20ms
        std::vector<Item> items = inventory->getItems(order.getItems());  // 15ms
        ShippingStatus shipping = shippingService->getStatus(orderId);  // 25ms

        // Total: 10 + 20 + 15 + 25 = 70ms
        return OrderDetails(order, user, items, shipping);
    }
};

C#

SequentialBad.cs

public class OrderService {
    // Sequential calls - high latency

    public async Task<OrderDetails> GetOrderDetails(string orderId) {
        // Each call waits for the previous one
        var order = await db.GetOrder(orderId);        // 10ms
        var user = await userService.Get(order.UserId);  // 20ms
        var items = await inventory.GetItems(order.Items);  // 15ms
        var shipping = await shippingService.GetStatus(orderId);  // 25ms

        // Total: 10 + 20 + 15 + 25 = 70ms
        return new OrderDetails(order, user, items, shipping);
    }
}

Good: Parallel Operations

💡 Tip: Click dropdown to switch between languages

Python

Python

parallel_good.py

import asyncio

class OrderService:
    """Parallel calls - lower latency"""

    async def get_order_details(self, order_id: str) -> OrderDetails:
        # First, get the order (we need it for user_id)
        order = await self.db.get_order(order_id)  # 10ms

        # Then fetch everything else in parallel
        user, items, shipping = await asyncio.gather(
            self.user_service.get(order.user_id),  # 20ms
            self.inventory.get_items(order.items),  # 15ms  } All run
            self.shipping.get_status(order_id)      # 25ms  } in parallel
        )

        # Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        # Saved 35ms (50% reduction!)
        return OrderDetails(order, user, items, shipping)

Java

ParallelGood.java

import java.util.concurrent.CompletableFuture;

public class OrderService {
    // Parallel calls - lower latency

    public CompletableFuture<OrderDetails> getOrderDetails(String orderId) {
        // First, get the order (we need it for user_id)
        return db.getOrderAsync(orderId)  // 10ms
            .thenCompose(order -> {
                // Then fetch everything else in parallel
                CompletableFuture<User> userFuture =
                    userService.getAsync(order.getUserId());  // 20ms
                CompletableFuture<List<Item>> itemsFuture =
                    inventory.getItemsAsync(order.getItems()); // 15ms  } All run
                CompletableFuture<ShippingStatus> shippingFuture =
                    shippingService.getStatusAsync(orderId);   // 25ms  } in parallel

                return CompletableFuture.allOf(userFuture, itemsFuture, shippingFuture)
                    .thenApply(v -> new OrderDetails(
                        order,
                        userFuture.join(),
                        itemsFuture.join(),
                        shippingFuture.join()
                    ));
            });

        // Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        // Saved 35ms (50% reduction!)
    }
}

TypeScript

parallel_good.ts

class OrderService {
    // Parallel calls - lower latency

    async getOrderDetails(orderId: string): Promise<OrderDetails> {
        // First, get the order (we need it for user_id)
        const order = await this.db.getOrder(orderId);  // 10ms

        // Then fetch everything else in parallel
        const [user, items, shipping] = await Promise.all([
            this.userService.get(order.userId),  // 20ms
            this.inventory.getItems(order.items),  // 15ms  } All run
            this.shipping.getStatus(orderId)      // 25ms  } in parallel
        ]);

        // Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        // Saved 35ms (50% reduction!)
        return new OrderDetails(order, user, items, shipping);
    }
}

C++

parallel_good.cpp

#include <future>
#include <vector>

class OrderService {
public:
    OrderDetails getOrderDetails(const std::string& orderId) {
        // First, get the order (we need it for user_id)
        Order order = db->getOrder(orderId);  // 10ms

        // Then fetch everything else in parallel
        auto userFuture = std::async(std::launch::async,
            [this, &order]() { return userService->get(order.getUserId()); });  // 20ms
        auto itemsFuture = std::async(std::launch::async,
            [this, &order]() { return inventory->getItems(order.getItems()); });  // 15ms
        auto shippingFuture = std::async(std::launch::async,
            [this, &orderId]() { return shippingService->getStatus(orderId); });  // 25ms

        // Wait for all parallel operations
        User user = userFuture.get();
        std::vector<Item> items = itemsFuture.get();
        ShippingStatus shipping = shippingFuture.get();

        // Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        // Saved 35ms (50% reduction!)
        return OrderDetails(order, user, items, shipping);
    }
};

C#

ParallelGood.cs

using System;
using System.Threading.Tasks;

public class OrderService {
    // Parallel calls - lower latency

    public async Task<OrderDetails> GetOrderDetails(string orderId) {
        // First, get the order (we need it for user_id)
        var order = await db.GetOrder(orderId);  // 10ms

        // Then fetch everything else in parallel
        var userTask = userService.Get(order.UserId);  // 20ms
        var itemsTask = inventory.GetItems(order.Items);  // 15ms  } All run
        var shippingTask = shippingService.GetStatus(orderId);  // 25ms  } in parallel

        await Task.WhenAll(userTask, itemsTask, shippingTask);

        // Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        // Saved 35ms (50% reduction!)
        return new OrderDetails(order, await userTask, await itemsTask, await shippingTask);
    }
}

---

Caching: The Latency Killer

Caching is the most effective way to reduce latency. The idea is simple: store frequently accessed data closer to where it’s needed.

Multi-Level Cache Architecture

Latency Math

With this caching strategy:

Level	Latency	Hit Rate
L1 (Local)	0.01ms	90%
L2 (Redis)	2ms	9%
Database	30ms	1%

Average latency = 0.9(0.01) + 0.09(2) + 0.01(30) = 0.49ms

That’s a 60x improvement over hitting the database every time!

---

Measuring Performance

Key Metrics to Track

Metric	What It Tells You	Action Threshold
P50	Typical user experience	Baseline for “normal”
P95	Most users’ worst case	Watch for drift
P99	Outlier experience	Investigate if > 3x P50
Error Rate	System health	Alert if > 1%

Tools for Measurement

In Production:

• APM Tools: Datadog, New Relic, Dynatrace
• Metrics: Prometheus + Grafana
• Distributed Tracing: Jaeger, Zipkin

In Development:

• Profilers: cProfile (Python), JProfiler (Java)
• Benchmarking: pytest-benchmark, JMH

Golden Rule

Measure in production, not just in tests. Real-world traffic patterns, data sizes, and concurrent users create performance characteristics you can’t simulate.

---

Real-World Examples

Example 1: Google Search Latency Optimization

Company: Google

Scenario: Google Search must return results in milliseconds. Even 100ms delay can reduce user satisfaction and search volume.

Implementation: Uses parallel processing and caching:

Why This Matters:

• Scale: Billions of queries per day
• Latency Impact: 100ms delay = 0.2% search volume reduction
• Performance: Parallel processing reduces latency by 60%
• Result: Sub-100ms average response time

Real-World Impact:

• Queries: 8.5+ billion searches per day
• Latency: Average 50ms, P99 200ms
• Revenue Impact: Every 100ms delay costs millions in ad revenue

---

Example 2: Amazon Product Page Latency

Company: Amazon

Scenario: Product pages must load quickly. Amazon found that every 100ms delay reduces sales by 1%.

Implementation: Uses parallel API calls and edge caching:

Why Parallel Processing?

• User Experience: Fast page loads increase conversions
• Revenue Impact: 100ms delay = 1% sales reduction
• Performance: Parallel calls reduce latency by 70%
• Result: 200ms average page load time

Real-World Impact:

• Scale: Billions of product page views daily
• Latency: 200ms average, P99 500ms
• Revenue: Every 100ms optimization worth millions annually

---

Example 3: Netflix Video Streaming Latency

Company: Netflix

Scenario: Video playback must start quickly. Users expect playback to begin within 2 seconds of clicking play.

Implementation: Uses CDN distribution and adaptive bitrate streaming:

Why Low Latency Matters:

• User Experience: Fast playback start increases engagement
• Retention: Slow start causes users to abandon
• Performance: CDN reduces latency by 90%
• Result: < 2 seconds time to first frame

Real-World Impact:

• Scale: 200+ million subscribers, billions of plays daily
• Latency: < 2 seconds time to first frame
• Engagement: Fast playback increases watch time by 20%

---

Key Takeaways

Remember

• Latency = time for one operation (lower is better)
• Throughput = operations per second (higher is better)
• P99 matters - at scale, 1% affects millions of users
• Little’s Law: Concurrent = Throughput × Latency
• Parallel operations reduce latency
• Caching is the most effective latency reducer
• Measure everything - you can’t improve what you don’t measure

---

What’s Next?

Now that you understand performance metrics, let’s learn how to identify and fix performance issues:

Next up: Understanding Bottlenecks - Learn to find and eliminate performance bottlenecks.

Action Item

Add latency tracking to one of your critical methods. You might be surprised by what you find!