Idempotency & Exactly-Once Processing

The Duplicate Problem

In distributed systems, messages/requests can be duplicated:

Problem: Without idempotency, retries create duplicates!

Solution: Idempotency - Make operations safe to retry!

---

What is Idempotency?

An idempotent operation is one that produces the same result when executed multiple times with the same input. In other words, calling it once has the same effect as calling it N times.

Why Idempotency Matters

In distributed systems, operations can be retried due to network failures, timeouts, or system crashes. Without idempotency, retries can cause:

• Duplicate operations (creating the same order twice)
• Incorrect state (adding balance multiple times)
• Side effects (sending multiple emails)

Idempotency makes operations safe to retry - if a request is retried, it produces the same result as the first attempt, preventing duplicates and ensuring consistency.

Mathematical Definition

f(f(x)) = f(x)

Calling the function twice = calling once.

Examples

Idempotent Operations:

• GET /users/123 - Always returns same user
• PUT /users/123 - Replaces user (same result)
• DELETE /users/123 - Deletes user (same result if already deleted)
• SET balance = 100 - Sets to 100 (same result)

Non-Idempotent Operations:

• POST /orders - Creates new order each time
• balance = balance + 50 - Adds 50 each time
• send_email() - Sends email each time

Simple Analogy

Think of idempotency like a light switch:

• Toggle switch (non-idempotent): Press once = on, press again = off (different result)
• On/Off switch (idempotent): Press “on” = on, press “on” again = still on (same result)

We want operations to be like on/off switches - safe to press multiple times!

---

Making Operations Idempotent

Strategy 1: Idempotency Keys

Client provides unique key. Server checks if seen before.

How it works:

1. Client generates unique idempotency key
2. Server checks if key exists
3. If new → Execute operation, store result with key
4. If exists → Return cached result (don’t execute)

Idempotency Key Implementation

---

Exactly-Once Processing

Ensures message processed exactly once.

Components

1. Idempotent Operations - Safe to retry
2. Deduplication - Track processed messages
3. Distributed Coordination - Handle duplicates across servers

Deduplication Implementation

---

Distributed Idempotency

For multiple servers, use shared storage (Redis/Database):

---

Best Practices

1. Use Idempotency Keys

Always provide idempotency keys for mutating operations:

POST /orders
Idempotency-Key: abc123-xyz789
Content-Type: application/json

{ "user_id": 456, "amount": 99.99 }

2. Generate Keys Client-Side

Client generates unique key:

• UUID
• Timestamp + random
• Hash of request content

3. Store Results

Cache results with keys:

• Store successful results
• Don’t store failed results (allow retry)
• Set TTL (e.g., 24 hours)

4. Check Before Processing

Always check key before processing:

• If exists → Return cached result
• If new → Process and cache

5. Make Operations Idempotent

Design operations to be idempotent:

• Use “set” instead of “add”
• Check state before modifying
• Use conditional updates

---

Key Takeaways

Idempotency is Critical

Make operations safe to retry. Essential for distributed systems, retries, at-least-once delivery.

Idempotency Keys

Use unique keys to detect duplicates. Check before processing. Cache results. Return cached on duplicate.

Exactly-Once

Exactly-once requires idempotency + deduplication. Track processed messages. Use distributed storage for multiple servers.

Store Results

Cache successful results with keys. Don’t cache failures. Set TTL. Check before processing.

---

Next Steps

• Review Message Queues - idempotency critical for message processing
• Master Saga Pattern - sagas need idempotent steps
• Understand Event-Driven Architecture - events should be idempotent