Webhook Architecture: Reliable Event Delivery#

Webhooks are HTTP callbacks — your server sends a POST request to the subscriber's URL when something happens. Simple in concept, tricky to make reliable.

How Webhooks Work#

Event occurs → Your Server → POST to subscriber URL → Subscriber processes
                          ← 200 OK (acknowledged)

The Reliability Problem#

Networks fail. Subscribers crash. Without retry logic, events get lost:

Event → POST → subscriber returns 500 → event lost forever ❌
Event → POST → network timeout → event lost forever ❌
Event → POST → subscriber is down → event lost forever ❌

Patterns for Reliability#

1. Retry with Exponential Backoff#

Attempt 1: immediate
Attempt 2: 1 minute
Attempt 3: 5 minutes
Attempt 4: 30 minutes
Attempt 5: 2 hours
Attempt 6: 8 hours (final)

After all retries fail → mark as failed → alert subscriber.

Stripe retries 3 times over 24 hours. GitHub retries for 3 days.

2. Signature Verification#

Prove the webhook came from you, not an attacker:

// Sender: sign the payload
const signature = hmac('sha256', secret, JSON.stringify(payload));
headers['X-Webhook-Signature'] = signature;

// Receiver: verify
const expected = hmac('sha256', secret, body);
if (signature !== expected) return 401; // reject

Always verify signatures. Without this, anyone can POST to your webhook URL.

3. Idempotency#

Retries mean duplicate deliveries. Receivers must handle them:

headers['X-Webhook-ID'] = 'evt_abc123';

// Receiver
const processed = await redis.setnx(`webhook:${eventId}`, '1');
if (!processed) return 200; // already handled, skip

4. Outbox Pattern#

Ensure events are published reliably:

-- In the same transaction as the business logic:
BEGIN;
  INSERT INTO orders (id, amount) VALUES ('ord_123', 100);
  INSERT INTO webhook_outbox (event_type, payload) VALUES ('order.created', '...');
COMMIT;

-- Background worker polls outbox → sends webhooks → marks as sent

Never publish a webhook directly in the request handler — if the request fails after sending, the webhook fires but the action didn't complete.

5. Fan-Out Queue#

Don't send webhooks synchronously. Queue them:

Event → Queue (SQS/Kafka) → Webhook Worker → POST to subscriber
                                            → retry on failure
                                            → dead letter on exhaust

Benefits: Non-blocking, parallel delivery, retry isolation.

Architecture#

Your Service → Event occurs → Webhook Service
                                  → Store event in DB
                                  → Queue delivery jobs
                                  → Worker: POST + retry
                                  → Log: delivery status
                                  → Dashboard: subscriber health

Subscriber → receives POST → verifies signature → processes → returns 200

Webhook Event Format#

Standard format (similar to Stripe/GitHub):

{
  "id": "evt_abc123",
  "type": "order.created",
  "created": "2026-03-28T14:30:00Z",
  "data": {
    "id": "ord_456",
    "amount": 9999,
    "currency": "usd",
    "customer": "cus_789"
  }
}

Include: Event ID, type, timestamp, full payload. Don't include: Secrets, tokens, or internal IDs the subscriber can't use.

Delivery Guarantees#

At-least-once is the standard. Require subscribers to be idempotent.

Monitoring#

Track per subscriber:

• Delivery rate — % of successful deliveries
• Latency — time from event to successful delivery
• Failure rate — % of failed attempts
• Consecutive failures — auto-disable after N failures

Subscriber health:
  acme.com/webhooks: 99.2% success, avg 230ms, 0 consecutive failures ✓
  corp.io/hooks:     85% success, avg 1.2s, 12 consecutive failures ⚠️ → auto-disabled

Best Practices#

1. Sign every webhook — HMAC-SHA256 with per-subscriber secret
2. Include event ID — subscribers need it for dedup
3. Retry with backoff — 5-6 attempts over 24 hours
4. Use a queue — never send synchronously from the request handler
5. Outbox pattern — ensure events match database state
6. Subscriber dashboard — show delivery logs, let them resend
7. Auto-disable after N consecutive failures — alert the subscriber
8. Timeout at 30 seconds — don't wait forever for slow subscribers

Design webhook architectures at codelit.io — generate interactive diagrams with infrastructure exports.

---

114 articles on system design at codelit.io/blog.