Message Queue Architecture: Kafka vs RabbitMQ vs SQS#

Message queues decouple services, absorb traffic spikes, and enable async processing. But choosing the wrong one creates operational nightmares.

This guide covers when to use each and the patterns that make queues reliable.

Why Message Queues?#

Without a queue:

API → calls Payment Service directly → Payment Service is slow/down → API times out → user sees error

With a queue:

API → publishes to queue → returns 202 Accepted immediately
Queue → Payment Service processes when ready → retries on failure

Benefits: Decoupling, resilience, traffic smoothing, async processing.

The Big Three#

Apache Kafka — Distributed Event Log#

Kafka is a distributed, append-only log. Messages are retained for days/weeks, and multiple consumers can read independently.

Producer → Topic (partitioned log)
              Partition 0: [msg1, msg2, msg5, msg8, ...]
              Partition 1: [msg3, msg4, msg6, msg9, ...]
              Partition 2: [msg7, msg10, ...]

Consumer Group A: reads all partitions (each consumer gets subset)
Consumer Group B: independently reads all partitions (replay!)

Key concepts:

• Topics — named streams of messages
• Partitions — parallelism unit (more partitions = more throughput)
• Consumer groups — each group reads all messages independently
• Offsets — consumers track their position (can replay from any point)
• Retention — messages kept for configurable time (not deleted on read)

Best for: Event streaming, event sourcing, real-time analytics, CDC, log aggregation

RabbitMQ — Traditional Message Broker#

RabbitMQ is a traditional message broker. Messages are delivered to consumers and deleted after acknowledgment.

Producer → Exchange → routes to → Queue A → Consumer A
                                → Queue B → Consumer B

Exchange types:
  Direct:  route by exact routing key
  Fanout:  broadcast to all bound queues
  Topic:   route by pattern matching (order.*, payment.#)
  Headers: route by message headers

Key concepts:

• Exchanges — route messages to queues based on rules
• Queues — store messages until consumed
• Bindings — rules connecting exchanges to queues
• Acknowledgments — consumer confirms processing
• Dead letter queue — failed messages go here

Best for: Task queues, RPC, complex routing, priority queues

AWS SQS — Managed Queue#

SQS is fully managed — no infrastructure to operate.

Producer → SQS Queue → Consumer polls for messages
                     → processes → deletes message
                     → visibility timeout prevents double processing

Two types:

• Standard — unlimited throughput, at-least-once delivery, best-effort ordering
• FIFO — exactly-once, strict ordering, 300 TPS (3000 with batching)

Best for: Simple async tasks, Lambda triggers, zero-ops requirement

Comparison#

Queue Patterns#

1. Work Queue (Task Distribution)#

Producer → Queue → Consumer 1 (processes task)
                 → Consumer 2 (processes task)
                 → Consumer 3 (processes task)

Each message goes to ONE consumer. Load balanced across workers.

Use case: Background jobs (email sending, image processing, PDF generation) Best tool: RabbitMQ or SQS

2. Pub/Sub (Fan-Out)#

Producer → Topic/Exchange → Consumer A (gets ALL messages)
                          → Consumer B (gets ALL messages)
                          → Consumer C (gets ALL messages)

Every message goes to ALL consumers.

Use case: Event notifications (order created → email + analytics + inventory) Best tool: Kafka (consumer groups) or RabbitMQ (fanout exchange)

3. Dead Letter Queue#

Main Queue → Consumer processes → fails 3 times → Dead Letter Queue
                                                      ↓
                                               Manual review / alert

Failed messages don't block the queue. They're moved aside for investigation.

Always implement DLQs. Without them, poison messages block your entire pipeline.

4. Priority Queue#

Queue with priorities:
  Priority 1 (high): [payment_failed, security_alert]
  Priority 2 (medium): [order_confirmation]
  Priority 3 (low): [marketing_email, analytics_event]

High-priority messages processed first.

Best tool: RabbitMQ (native priority queues)

5. Request-Reply#

Client → Request Queue → Server processes → Reply Queue → Client
          (correlation_id: "abc")                (correlation_id: "abc")

Synchronous RPC over async queues.

Best tool: RabbitMQ (built-in reply-to header)

6. Saga / Choreography#

Order Service → publishes "OrderCreated" → Kafka
Payment Service → consumes → publishes "PaymentSucceeded"
Inventory Service → consumes → publishes "InventoryReserved"
Shipping Service → consumes → publishes "ShipmentCreated"

Services coordinate via events. No central orchestrator.

Best tool: Kafka (event log for replay and debugging)

Delivery Guarantees#

At-least-once + idempotent consumers is the practical standard. True exactly-once is expensive and rarely needed.

Making Consumers Idempotent#

// Before processing, check if already handled
const existing = await db.findOne({ messageId: msg.id });
if (existing) return; // skip duplicate

await processMessage(msg);
await db.insert({ messageId: msg.id, processedAt: new Date() });

Architecture Examples#

E-Commerce Async Processing#

Web App → API → SQS (order queue) → Order Worker
                                         → Payment Service
                                         → Inventory Service
              → SQS (email queue) → Email Worker → SendGrid
              → SQS (analytics)  → Analytics Worker → ClickHouse

Real-Time Event Platform#

Services → Kafka → Stream Processor (Flink)
                      → Real-time dashboards
                      → Fraud detection
                      → Recommendation engine
                 → S3 (archival)
                 → Elasticsearch (search index)

Generate your queue architecture →

Quick Decision#

Summary#

1. Kafka for event streaming, analytics, and replay
2. RabbitMQ for task queues and complex routing
3. SQS when you want zero operational overhead
4. Always add dead letter queues — poison messages will happen
5. At-least-once + idempotency is the practical standard
6. Don't over-engineer — SQS handles most async needs

---

Design message queue architectures at codelit.io — generate interactive diagrams with Kafka, RabbitMQ, and infrastructure exports.