Job Queues & Background Workers

The Problem

Many operations are too slow or unreliable to execute during a user’s HTTP request — sending emails, processing images, generating reports, syncing data with third-party APIs, or running ML inference. Job queues decouple the request from the work: the API enqueues a job and responds immediately, while background workers process the job asynchronously. The challenge is ensuring jobs are processed reliably (at-least-once delivery), handling failures with intelligent retries, scaling workers independently, prioritizing urgent jobs, and monitoring the health of the entire pipeline.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

Retry Strategies and Backoff

Job failures are inevitable — network timeouts, rate limits, temporary outages. The retry strategy determines recovery speed and system impact.

Fixed interval: Retry every 30 seconds. Simple but can cause synchronized retries (thundering herd) if many jobs fail simultaneously.

Exponential backoff: 1s, 2s, 4s, 8s, 16s, … Spreads retries over time. Add jitter (random 0-50% of the delay) to prevent synchronized retries: delay = min(base * 2^attempt + random(0, base * 2^attempt * 0.5), max_delay)

Custom backoff: Different error types warrant different strategies:

• Rate limited (429): Use the Retry-After header from the response
• Network error: Exponential backoff (transient failure)
• Validation error (400): Don’t retry — move to DLQ immediately
• Internal error (500): Retry with exponential backoff

Max attempts: Typically 3-5 retries for API calls, 10+ for critical jobs. After max attempts, move to the dead-letter queue. Always cap the maximum delay (e.g., 1 hour) to prevent infinite backoff.

Idempotency: Since jobs may be processed more than once (at-least-once delivery), the job handler must be idempotent. Use unique job IDs and check for prior completion before executing side effects (sending emails, charging cards).

Scaling Workers

Worker scaling depends on the job type and processing characteristics.

CPU-bound jobs (image processing, PDF generation, ML inference): Scale by adding more worker processes/containers. Each worker runs one job at a time. Use container orchestration (Kubernetes HPA) to autoscale based on CPU utilization.

I/O-bound jobs (API calls, email sending, database writes): Increase concurrency per worker. A single Node.js worker with BullMQ can process 10-50 concurrent I/O-bound jobs. Scale based on queue depth rather than CPU.

Autoscaling formula: Target = ⌈queue_depth / (processing_rate × target_drain_time)⌉. If you have 10,000 pending jobs, each worker processes 100 jobs/minute, and you want to drain in 10 minutes: Target = ⌈10,000 / (100 × 10)⌉ = 10 workers.

Queue-based autoscaling: Monitor queue depth and processing latency. Scale up when queue depth exceeds threshold or processing latency exceeds SLA. Scale down when queues are empty. Use a cooldown period to avoid thrashing.

Graceful shutdown: When scaling down or deploying, workers must finish their current job before terminating. Implement SIGTERM handling: stop accepting new jobs, wait for in-progress jobs to complete (with a timeout), then exit. Kubernetes provides a terminationGracePeriodSeconds for this.

Job Scheduling and Cron

Beyond on-demand jobs, many systems need scheduled and recurring jobs.

Delayed jobs: Enqueue a job to be processed at a specific future time. Use case: “Send a follow-up email 24 hours after signup.” BullMQ and Sidekiq support this natively with a delay parameter.

Recurring jobs (cron): Jobs that run on a schedule — daily reports, hourly data sync, weekly cleanup. Implementation options:

• In-queue scheduling: BullMQ’s repeat option creates jobs on a cron schedule. The queue library manages the schedule.
• External scheduler: A cron service (node-cron, Kubernetes CronJobs) enqueues jobs at the scheduled time. Simpler but requires a separate scheduler process.

Distributed cron: In a multi-instance deployment, you must ensure the cron job is triggered exactly once, not once per instance. Solutions:

• Use a distributed lock (Redis SETNX with TTL) before enqueuing
• Use a single scheduler instance (leader election)
• Use BullMQ’s built-in repeat — it handles deduplication automatically

Missed schedules: If the scheduler is down when a job should run, should it catch up? BullMQ skips missed runs by default. For critical jobs (daily billing), implement catch-up logic that checks the last run time on startup and enqueues missed jobs.

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: Job Queues & Background Workers

You are an expert software architect and teacher. I want to understand job queues & background workers as a practical system design capability. (Async task processing with retries, dead-letter queues, and priority scheduling).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

Many operations are too slow or unreliable to execute during a user's HTTP request — sending emails, processing images, generating reports, syncing data with third-party APIs, or running ML inference. Job queues decouple the request from the work: the API enqueues a job and responds immediately, while background workers process the job asynchronously. The challenge is ensuring jobs are processed reliably (at-least-once delivery), handling failures with intelligent retries, scaling workers independently, prioritizing urgent jobs, and monitoring the health of the entire pipeline.

## Architectural Approaches

<ApproachList>
<Approach
  name="Redis-Based Queue (BullMQ, Sidekiq)"
  pros={[
    'Very fast enqueue/dequeue — Redis is in-memory',
    'Rich ecosystem — BullMQ, Sidekiq, Celery have mature features',
    'Built-in delayed jobs, rate limiting, and priority queues',
    'Easy to set up — Redis is already in most stacks',
  ]}
  cons={[
    'Redis memory is expensive — not ideal for millions of queued jobs',
    'Durability risk — Redis can lose data on crash (mitigated with AOF)',
    'Scaling beyond single Redis instance requires Redis Cluster',
    'Not suitable for very long-running jobs (hours+)',
  ]}
>
Use Redis as the job queue backend. Libraries like **BullMQ** (Node.js), **Sidekiq** (Ruby), or **Celery** (Python with Redis broker) provide job scheduling, retries, priority queues, ...

[Full prompt copied to clipboard]