Idempotent API Design#

An operation is idempotent if performing it multiple times produces the same result as performing it once. In distributed systems, network failures, timeouts, and retries are inevitable — idempotency is what keeps your system from double-charging customers or creating duplicate records.

Why Idempotency Matters#

Client → POST /payments → timeout (no response)
Client → POST /payments → retry
Server → processes payment AGAIN → customer charged twice

Without idempotency, every retry is a gamble. With it:

Client → POST /payments (key: abc-123) → timeout
Client → POST /payments (key: abc-123) → retry
Server → recognizes key abc-123 → returns original result

HTTP Methods and Idempotency#

Not all HTTP methods are created equal:

Key insight: POST is the most dangerous method because it is not idempotent by default. This is exactly where idempotency keys come in.

The Idempotency Key Pattern (Stripe Model)#

Stripe popularized this approach, and it has become an industry standard:

// Client sends a unique key with the request
const response = await fetch('/api/payments', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Idempotency-Key': 'user-123-order-456-attempt-1'
  },
  body: JSON.stringify({
    amount: 2999,
    currency: 'usd',
    customer: 'cus_abc123'
  })
});

Server-Side Implementation#

async function handlePayment(req, res) {
  const idempotencyKey = req.headers['idempotency-key'];

  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Idempotency-Key header required' });
  }

  // Check if we already processed this key
  const existing = await db.idempotencyKeys.findOne({
    key: idempotencyKey
  });

  if (existing) {
    // Return the cached response
    return res.status(existing.statusCode).json(existing.body);
  }

  // Process the payment
  try {
    const result = await processPayment(req.body);

    // Store the result keyed by idempotency key
    await db.idempotencyKeys.insertOne({
      key: idempotencyKey,
      statusCode: 200,
      body: result,
      createdAt: new Date(),
      expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000) // 24h TTL
    });

    return res.status(200).json(result);
  } catch (error) {
    // Only cache terminal errors, not transient ones
    if (isTerminalError(error)) {
      await db.idempotencyKeys.insertOne({
        key: idempotencyKey,
        statusCode: error.statusCode,
        body: { error: error.message },
        createdAt: new Date()
      });
    }
    throw error;
  }
}

Key Design Decisions#

• TTL: Keys should expire (Stripe uses 24 hours). Stale keys waste storage.
• Scope: Keys should be scoped per API endpoint, not globally.
• Generation: Clients generate keys — typically UUID v4 or a deterministic hash of the operation.
• Locking: Use database-level locks to prevent race conditions on concurrent retries.

Database Upserts for Idempotency#

When your operation maps to a database write, upserts give you idempotency for free:

-- PostgreSQL: Insert or update based on unique constraint
INSERT INTO orders (order_id, user_id, amount, status)
VALUES ('ord-456', 'usr-123', 29.99, 'pending')
ON CONFLICT (order_id)
DO UPDATE SET
  amount = EXCLUDED.amount,
  status = EXCLUDED.status,
  updated_at = NOW();

// MongoDB equivalent
await db.orders.updateOne(
  { orderId: 'ord-456' },
  {
    $set: {
      userId: 'usr-123',
      amount: 29.99,
      status: 'pending',
      updatedAt: new Date()
    }
  },
  { upsert: true }
);

When upserts work: The operation has a natural unique identifier (order ID, transaction ID).

When they don't: The operation creates something without a pre-determined ID.

Conditional Requests: ETag and If-Match#

HTTP has built-in mechanisms for safe concurrent updates:

// Step 1: Client fetches resource with its ETag
GET /api/documents/42
→ 200 OK
→ ETag: "version-7a3b"
→ { "title": "Draft", "content": "..." }

// Step 2: Client updates with If-Match
PUT /api/documents/42
If-Match: "version-7a3b"
{ "title": "Final", "content": "..." }

// If no one else modified it:
→ 200 OK, ETag: "version-8c4d"

// If someone else modified it first:
→ 412 Precondition Failed

Server Implementation#

app.put('/api/documents/:id', async (req, res) => {
  const ifMatch = req.headers['if-match'];
  const doc = await db.documents.findById(req.params.id);

  if (!doc) return res.status(404).end();

  const currentEtag = generateEtag(doc);

  if (ifMatch && ifMatch !== currentEtag) {
    return res.status(412).json({
      error: 'Resource modified by another request',
      currentEtag
    });
  }

  const updated = await db.documents.updateOne(
    { _id: req.params.id, version: doc.version }, // optimistic lock
    { $set: { ...req.body, version: doc.version + 1 } }
  );

  if (updated.modifiedCount === 0) {
    return res.status(412).json({ error: 'Concurrent modification' });
  }

  const newDoc = await db.documents.findById(req.params.id);
  res.set('ETag', generateEtag(newDoc));
  res.json(newDoc);
});

Client-Side Retry Logic#

Your API is only as safe as the client calling it:

async function retryableRequest(url, options, maxRetries = 3) {
  const idempotencyKey = options.idempotencyKey || crypto.randomUUID();

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      const response = await fetch(url, {
        ...options,
        headers: {
          ...options.headers,
          'Idempotency-Key': idempotencyKey
        }
      });

      // Don't retry client errors (4xx) — they won't succeed
      if (response.status >= 400 && response.status < 500) {
        return response;
      }

      // Retry server errors (5xx)
      if (response.status >= 500) {
        if (attempt < maxRetries) {
          await sleep(exponentialBackoff(attempt));
          continue;
        }
      }

      return response;
    } catch (networkError) {
      // Retry on network failures
      if (attempt < maxRetries) {
        await sleep(exponentialBackoff(attempt));
        continue;
      }
      throw networkError;
    }
  }
}

function exponentialBackoff(attempt) {
  const base = 1000;
  const jitter = Math.random() * 500;
  return base * Math.pow(2, attempt) + jitter;
}

Retry Rules#

1. Always use exponential backoff with jitter to avoid thundering herd.
2. Never retry 4xx errors — the request itself is wrong.
3. Always retry network errors and 5xx — these are transient.
4. Set a maximum retry count — infinite retries cause cascading failures.
5. Reuse the same idempotency key across all retry attempts.

Putting It All Together#

A production checklist for idempotent APIs:

• All POST endpoints accept Idempotency-Key header
• Idempotency keys stored with TTL (24 hours typical)
• Database operations use upserts where possible
• PUT/PATCH endpoints support If-Match / ETag
• Client SDKs implement retry with exponential backoff
• Race conditions handled with database-level locking
• Monitoring alerts on duplicate key hits (signals reliability issues)

Common Pitfalls#

Caching errors incorrectly: Only cache deterministic errors. A 503 today might succeed tomorrow — do not cache it under the idempotency key.

Key collisions: Using sequential IDs or predictable keys causes cross-user conflicts. Always use UUIDs or user-scoped keys.

Ignoring side effects: If your endpoint sends an email AND creates a record, the idempotency key must guard both operations, not just the database write.

Missing the window: If the key expires before the client retries, you lose idempotency. Set TTL based on your longest reasonable retry window.

---

Idempotent API design is the foundation of reliable distributed systems. Get this right, and retries become safe, webhooks become reliable, and your users never see duplicate charges.

Article #272 of the Codelit engineering series. Browse all articles at codelit.io