Idempotency in Distributed Systems: Retry Safety, Deduplication & Patterns

What Is Idempotency?#

An operation is idempotent if performing it multiple times produces the same result as performing it once. In mathematics, abs(abs(x)) === abs(x). In distributed systems, an idempotent API guarantees that replaying the same request — whether due to a timeout, retry, or network partition — won't cause unintended side effects.

This property is foundational to retry safety and exactly-once delivery semantics.

Why Idempotency Matters#

Distributed systems fail in partial, unpredictable ways:

• A client sends a request, the server processes it, but the response is lost. The client retries.
• A load balancer times out and re-routes the same request to another node.
• A message broker delivers the same event more than once.

Without idempotency, each retry can create a duplicate charge, a duplicate order, or corrupted state. With it, retries are safe by definition.

HTTP Method Idempotency#

The HTTP spec defines idempotency per method:

PUT replaces a resource entirely — repeating it yields the same state. POST typically creates a new resource, so repeating it can create duplicates. This is why POST endpoints need explicit idempotency mechanisms.

Idempotency Keys#

An idempotency key is a unique client-generated token attached to a request. The server uses it to detect and deduplicate retries.

// Client sends a unique key with each logical operation
const response = await fetch("/api/payments", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Idempotency-Key": "a1b2c3d4-uuid-per-operation",
  },
  body: JSON.stringify({ amount: 5000, currency: "usd" }),
});

On the server side:

async function handlePayment(req: Request) {
  const idempotencyKey = req.headers["idempotency-key"];

  // Check if we already processed this key
  const existing = await db.idempotencyStore.findOne({ key: idempotencyKey });
  if (existing) {
    return existing.response; // Return cached response
  }

  // Process the payment
  const result = await processPayment(req.body);

  // Store the result keyed by idempotency key
  await db.idempotencyStore.insertOne({
    key: idempotencyKey,
    response: result,
    createdAt: new Date(),
  });

  return result;
}

Key design decisions:

• Keys should expire after a reasonable TTL (e.g., 24-48 hours).
• The store must use an atomic check-and-set to avoid race conditions between concurrent retries.
• Return the same status code and body for replayed requests.

Database Idempotency: Upserts & Conditional Writes#

At the database layer, two patterns enforce idempotency without application-level key tracking.

Upserts#

-- Insert or update — safe to retry
INSERT INTO inventory (sku, quantity)
VALUES ('WIDGET-42', 10)
ON CONFLICT (sku)
DO UPDATE SET quantity = 10;

The key insight: this sets the quantity to an absolute value, not a relative increment. SET quantity = quantity + 10 is not idempotent.

Conditional Writes (Optimistic Concurrency)#

UPDATE orders
SET status = 'shipped', version = version + 1
WHERE id = 'order-123' AND version = 3;
-- Affected rows = 0 means someone else already updated it

If the row was already modified, the WHERE clause won't match and zero rows change — a safe no-op on retry.

Payment Idempotency: Stripe's Approach#

Stripe's API is the gold standard for payment idempotency. Every mutating endpoint accepts an Idempotency-Key header:

const stripe = require("stripe")("sk_live_...");

const charge = await stripe.charges.create(
  {
    amount: 2000,
    currency: "usd",
    source: "tok_visa",
  },
  {
    idempotencyKey: "order-7890-attempt-1",
  }
);

Stripe stores the result for 24 hours. If you retry with the same key, you get back the original response — no double charge. If you send the same key with different parameters, Stripe returns an error, preventing misuse.

Distributed Transaction Patterns#

When an operation spans multiple services, single-resource idempotency isn't enough. Two patterns dominate.

Saga Pattern#

A saga breaks a distributed transaction into a sequence of local transactions, each with a compensating action for rollback:

1. Order Service     → Create order   (compensate: cancel order)
2. Payment Service   → Charge card    (compensate: refund)
3. Inventory Service → Reserve stock  (compensate: release stock)

Each step must be idempotent. If step 2 fails after the charge succeeds but before acknowledgment, the retry should detect the existing charge via its idempotency key and return success.

Two-Phase Commit (2PC)#

2PC uses a coordinator to ensure all-or-nothing across participants:

1. Prepare phase: coordinator asks each participant to vote commit/abort.
2. Commit phase: if all vote commit, coordinator tells everyone to commit.

2PC provides stronger consistency than sagas but has a blocking problem — if the coordinator crashes during phase 2, participants hold locks indefinitely. In practice, most teams prefer sagas for their resilience.

Deduplication Strategies#

Beyond idempotency keys, several strategies prevent duplicate processing:

1. Message Deduplication Table#

async function handleEvent(event: Event) {
  const inserted = await db.processedEvents.insertIfNotExists({
    eventId: event.id,
    processedAt: new Date(),
  });

  if (!inserted) {
    return; // Already processed — skip
  }

  await processEvent(event);
}

2. Natural Idempotency via Absolute State#

Design operations around setting state rather than modifying state:

// Not idempotent — each call increments
await db.accounts.updateOne(
  { id: accountId },
  { $inc: { balance: 100 } }
);

// Idempotent — each call sets the same final state
await db.accounts.updateOne(
  { id: accountId },
  { $set: { balance: finalBalance, lastTxId: txId } }
);

3. Consumer Offset Tracking#

In event streaming (Kafka, etc.), consumers track the last processed offset. On restart, they resume from the stored offset rather than reprocessing everything. Combined with idempotent writes, this achieves exactly-once delivery semantics.

Checklist for Idempotent APIs#

1. Generate idempotency keys client-side — UUIDs or deterministic hashes of the operation.
2. Store responses atomically — use a unique constraint on the key column.
3. Expire keys — TTLs prevent unbounded storage growth.
4. Make database writes idempotent — upserts, conditional updates, absolute state.
5. Design compensations — every saga step needs a rollback path.
6. Test with chaos — inject duplicate requests, network delays, and partial failures.

Conclusion#

Idempotency isn't a nice-to-have — it's a requirement for any system where retries happen (and they always do). Whether you're building an idempotent API with idempotency keys, designing distributed transactions with sagas, or implementing deduplication in an event pipeline, the principle stays the same: make every operation safe to repeat.

Build systems that expect failure, and failure stops being a problem.

Ready to go deeper on system design? Explore more posts at codelit.io.

---

149 articles on system design at codelit.io/blog.