Async Processing Patterns: Queues, Workers & Background Jobs

Not everything needs to happen before you send the HTTP response. Moving work off the critical path is one of the highest-leverage performance techniques in backend engineering.

Why Go Async?#

Synchronous request handling means the user waits for everything — email sending, PDF generation, analytics, third-party API calls. Async processing lets you respond fast and handle the rest in the background.

Sync:   Request -> DB write -> Send email -> Generate PDF -> Response (2.3s)
Async:  Request -> DB write -> Enqueue jobs -> Response (120ms)
                                  |-> Worker sends email
                                  |-> Worker generates PDF

1. Fire-and-Forget#

Dispatch a task and never check the result.

# Using Celery
@celery.task
def track_analytics(event_data):
    analytics_service.track(event_data)

# In your view
track_analytics.delay({"event": "signup", "user_id": 42})
return Response({"status": "ok"})  # don't wait

When to use: Analytics, logging, non-critical notifications. Risk: If the task fails, nobody knows. Add dead-letter queues for observability.

2. Request-Reply (Async RPC)#

Send a message and wait for a response on a reply queue.

Producer -> Request Queue -> Consumer
                                |
Producer <- Reply Queue <-------+

# Correlation ID ties request to response
correlation_id = str(uuid.uuid4())
channel.basic_publish(
    exchange="",
    routing_key="rpc_queue",
    properties=pika.BasicProperties(
        reply_to=callback_queue,
        correlation_id=correlation_id,
    ),
    body=json.dumps({"action": "resize_image", "url": image_url}),
)

When to use: Distributed computation where the caller needs the result but can tolerate latency.

3. Pub/Sub (Publish-Subscribe)#

One event, many consumers. Each subscriber processes independently.

Order placed -> [order.created]
                   |-> Inventory service (reserve stock)
                   |-> Email service (send confirmation)
                   |-> Analytics service (track conversion)

# Publishing (Redis Streams)
redis.xadd("orders", {"event": "created", "order_id": "123", "total": "99.00"})

# Consuming (consumer group)
redis.xreadgroup("inventory-group", "worker-1", {"orders": ">"}, count=10)

Key distinction from work queues: In pub/sub, every subscriber gets every message. In work queues, each message goes to one worker.

4. Work Queue (Task Queue)#

Distribute tasks across a pool of workers. Each task is processed by exactly one worker.

# Bull (Node.js)
const queue = new Bull("email-queue", { redis: { host: "localhost" } });

queue.process(5, async (job) => {    // 5 concurrent workers
    await sendEmail(job.data.to, job.data.subject, job.data.body);
});

// Producer
await queue.add({ to: "user@example.com", subject: "Welcome!", body: "..." });

Concurrency control: Set worker count based on the task type. CPU-bound tasks: match core count. I/O-bound tasks: go higher.

5. Delayed Processing#

Schedule tasks to run at a specific time in the future.

# Celery — send reminder 24 hours after signup
send_reminder.apply_async(
    args=[user_id],
    countdown=86400,  # 24 hours in seconds
)

# Bull — delayed job
await queue.add(
    { userId: 42, type: "trial_expiry_warning" },
    { delay: 3 * 24 * 60 * 60 * 1000 }  // 3 days
);

When to use: Reminders, trial expirations, scheduled reports, retry with backoff.

6. Batch Processing#

Accumulate items and process them together for efficiency.

class BatchProcessor:
    def __init__(self, batch_size=100, flush_interval=5):
        self.buffer = []
        self.batch_size = batch_size
        self.flush_interval = flush_interval

    def add(self, item):
        self.buffer.append(item)
        if len(self.buffer) >= self.batch_size:
            self.flush()

    def flush(self):
        if not self.buffer:
            return
        # Bulk insert is 10-50x faster than individual inserts
        db.bulk_insert("events", self.buffer)
        self.buffer = []

When to use: Analytics ingestion, bulk API calls (Stripe batch charges), ETL pipelines.

7. Long-Running Tasks#

Tasks that take minutes or hours need special handling: progress tracking, checkpointing, and resumability.

@celery.task(bind=True)
def generate_report(self, report_id):
    total_steps = 1000
    for i in range(total_steps):
        process_chunk(report_id, i)
        # Update progress so the UI can show a progress bar
        self.update_state(
            state="PROGRESS",
            meta={"current": i + 1, "total": total_steps}
        )
    return {"status": "complete", "url": f"/reports/{report_id}.pdf"}

Checkpointing: Save progress so tasks can resume after crashes rather than restarting from zero.

8. Polling vs Webhooks#

Two approaches for the client to get async results.

Polling#

Client: POST /api/export         -> { "job_id": "abc123" }
Client: GET  /api/export/abc123  -> { "status": "processing" }
Client: GET  /api/export/abc123  -> { "status": "processing" }
Client: GET  /api/export/abc123  -> { "status": "done", "url": "..." }

Pros: Simple, stateless, works everywhere. Cons: Wasted requests, delayed awareness of completion.

Webhooks#

Client: POST /api/export
        { "callback_url": "https://myapp.com/hooks/export" }

# When done, the server calls:
POST https://myapp.com/hooks/export
{ "job_id": "abc123", "status": "done", "url": "..." }

Pros: Instant notification, no wasted requests. Cons: Client must expose an endpoint. Delivery can fail (need retries, HMAC verification).

Hybrid approach: Support both. Webhooks for real-time, polling as fallback.

9. Tool Comparison#

Temporal for Complex Workflows#

When tasks have dependencies, retries, and compensation logic, use a workflow engine:

# Temporal workflow
@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order_id):
        payment = await workflow.execute_activity(
            charge_payment, order_id, start_to_close_timeout=timedelta(seconds=30)
        )
        if payment.success:
            await workflow.execute_activity(reserve_inventory, order_id)
            await workflow.execute_activity(send_confirmation, order_id)
        else:
            await workflow.execute_activity(notify_payment_failure, order_id)

Reliability Patterns#

• Idempotency: Design tasks so running them twice produces the same result.
• Dead-letter queues: Capture failed messages for inspection.
• Exponential backoff: delay = min(base * 2^attempt, max_delay).
• Poison pill detection: Move messages to DLQ after N failures.
• Graceful shutdown: Finish in-progress tasks before stopping workers.

Key Takeaways#

• Default to async for anything the user does not need to see immediately.
• Work queues for distributing tasks; pub/sub for broadcasting events.
• Temporal or similar for multi-step workflows with compensation.
• Always plan for failure: idempotency, retries, dead-letter queues.
• Polling + webhooks together give the best client experience.

---

This is article #267 in the Codelit engineering series. Explore more backend architecture, distributed systems, and performance guides at codelit.io.