Latency and Throughput

The Two Pillars of Performance

When measuring system performance, two metrics matter most:

Simple Analogy

Think of a highway:

• Latency = How long it takes ONE car to travel from A to B
• Throughput = How many cars pass through per hour

A highway can have low latency (cars go fast) but low throughput (only one lane). Or high throughput (many lanes) but high latency (traffic jams).

---

Understanding Latency

Latency is the time between sending a request and receiving a response.

Latency Breakdown

Every request goes through multiple stages:

Types of Latency

Type	Description	Typical Values
Network Latency	Time for data to travel over network	1-100ms
Processing Latency	Time for server to process request	1-50ms
Database Latency	Time for database queries	1-10ms
Queue Latency	Time spent waiting in queues	0-1000ms+

---

Percentile Latencies: P50, P95, P99

Average latency can be misleading. Percentiles give a better picture:

Why P99 Matters

Scale Changes Everything

At scale, even small percentages affect many users:

Daily Requests	P99 Affected Users
100,000	1,000 users
1,000,000	10,000 users
100,000,000	1,000,000 users

1 million users experiencing slow responses is a big problem!

Real-World Example

---

Understanding Throughput

Throughput is the amount of work done per unit of time.

Common Throughput Metrics

Metric	Description	Example
RPS	Requests per second	10,000 RPS
TPS	Transactions per second	5,000 TPS
QPS	Queries per second	50,000 QPS
Bandwidth	Data transferred per second	1 Gbps

Throughput Calculation

Throughput is calculated as:

Throughput = Total Requests / Time Period

Example: If your server handles 10,000 requests in 10 seconds, your throughput is 1,000 RPS.

Key considerations:

• Measure over time - instantaneous measurements fluctuate
• Track success rate - failed requests count against effective throughput
• Monitor under load - throughput often drops when the system is stressed

---

Latency vs Throughput: The Trade-off

They’re related but independent:

Little’s Law

A fundamental relationship:

Average Concurrent Requests = Throughput × Average Latency

Example:

• Throughput: 1,000 RPS
• Average Latency: 100ms = 0.1 seconds
• Concurrent Requests: 1,000 × 0.1 = 100 requests in flight

Why This Matters

If you need 1,000 RPS with 100ms latency, your server needs to handle 100 concurrent connections. This affects thread pool sizes, connection limits, and resource planning.

---

How Method Design Impacts Performance

Your code decisions directly affect latency and throughput:

Bad: Sequential Operations

Python

sequential_bad.py

class OrderService:
    """❌ Sequential calls - high latency"""

    def get_order_details(self, order_id: str) -> OrderDetails:
        # Each call waits for the previous one
        order = self.db.get_order(order_id)        # 10ms
        user = self.user_service.get(order.user_id)  # 20ms
        items = self.inventory.get_items(order.items)  # 15ms
        shipping = self.shipping.get_status(order_id)  # 25ms

        # Total: 10 + 20 + 15 + 25 = 70ms
        return OrderDetails(order, user, items, shipping)

Java

SequentialBad.java

public class OrderService {
    // ❌ Sequential calls - high latency

    public OrderDetails getOrderDetails(String orderId) {
        // Each call waits for the previous one
        Order order = db.getOrder(orderId);           // 10ms
        User user = userService.get(order.getUserId()); // 20ms
        List<Item> items = inventory.getItems(order.getItems()); // 15ms
        ShippingStatus shipping = shippingService.getStatus(orderId); // 25ms

        // Total: 10 + 20 + 15 + 25 = 70ms
        return new OrderDetails(order, user, items, shipping);
    }
}

Good: Parallel Operations

Python

parallel_good.py

import asyncio
from concurrent.futures import ThreadPoolExecutor

class OrderService:
    """✅ Parallel calls - lower latency"""

    async def get_order_details(self, order_id: str) -> OrderDetails:
        # First, get the order (we need it for user_id)
        order = await self.db.get_order(order_id)  # 10ms

        # Then fetch everything else in parallel
        user, items, shipping = await asyncio.gather(
            self.user_service.get(order.user_id),  # 20ms
            self.inventory.get_items(order.items),  # 15ms  } All run
            self.shipping.get_status(order_id)      # 25ms  } in parallel
        )

        # Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        # Saved 35ms (50% reduction!)
        return OrderDetails(order, user, items, shipping)

Java

ParallelGood.java

import java.util.concurrent.CompletableFuture;

public class OrderService {
    // ✅ Parallel calls - lower latency

    public CompletableFuture<OrderDetails> getOrderDetails(String orderId) {
        // First, get the order (we need it for user_id)
        return db.getOrderAsync(orderId)  // 10ms
            .thenCompose(order -> {
                // Then fetch everything else in parallel
                CompletableFuture<User> userFuture =
                    userService.getAsync(order.getUserId());  // 20ms
                CompletableFuture<List<Item>> itemsFuture =
                    inventory.getItemsAsync(order.getItems()); // 15ms  } All run
                CompletableFuture<ShippingStatus> shippingFuture =
                    shippingService.getStatusAsync(orderId);   // 25ms  } in parallel

                return CompletableFuture.allOf(userFuture, itemsFuture, shippingFuture)
                    .thenApply(v -> new OrderDetails(
                        order,
                        userFuture.join(),
                        itemsFuture.join(),
                        shippingFuture.join()
                    ));
            });

        // Total: 10 + max(20, 15, 25) = 10 + 25 = 35ms
        // Saved 35ms (50% reduction!)
    }
}

---

Caching: The Latency Killer

Caching is the most effective way to reduce latency. The idea is simple: store frequently accessed data closer to where it’s needed.

Multi-Level Cache Architecture

Latency Math

With this caching strategy:

Level	Latency	Hit Rate
L1 (Local)	0.01ms	90%
L2 (Redis)	2ms	9%
Database	30ms	1%

Average latency = 0.9(0.01) + 0.09(2) + 0.01(30) = 0.49ms

That’s a 60x improvement over hitting the database every time!

---

Measuring Performance

Key Metrics to Track

Metric	What It Tells You	Action Threshold
P50	Typical user experience	Baseline for “normal”
P95	Most users’ worst case	Watch for drift
P99	Outlier experience	Investigate if > 3x P50
Error Rate	System health	Alert if > 1%

Tools for Measurement

In Production:

• APM Tools: Datadog, New Relic, Dynatrace
• Metrics: Prometheus + Grafana
• Distributed Tracing: Jaeger, Zipkin

In Development:

• Profilers: cProfile (Python), JProfiler (Java)
• Benchmarking: pytest-benchmark, JMH

Golden Rule

Measure in production, not just in tests. Real-world traffic patterns, data sizes, and concurrent users create performance characteristics you can’t simulate.

---

Key Takeaways

Remember

• Latency = time for one operation (lower is better)
• Throughput = operations per second (higher is better)
• P99 matters - at scale, 1% affects millions of users
• Little’s Law: Concurrent = Throughput × Latency
• Parallel operations reduce latency
• Caching is the most effective latency reducer
• Measure everything - you can’t improve what you don’t measure

---

What’s Next?

Now that you understand performance metrics, let’s learn how to identify and fix performance issues:

Next up: Understanding Bottlenecks - Learn to find and eliminate performance bottlenecks.

Action Item

Add latency tracking to one of your critical methods. You might be surprised by what you find!