If you’ve ever hit “submit” on a web form and watched the browser spin for ten seconds while the server sent an email, resized an image, and updated three external services before finally returning a response – you’ve felt the problem that background jobs solve. The fix is to take anything that doesn’t need to happen before the response and do it asynchronously, after the response has already been sent.
This sounds simple, and the basic implementation is. The subtleties come in when you start thinking about reliability, failure handling, ordering guarantees, and worker concurrency. Let’s walk through the whole picture.
The core pattern
A job queue has three parts: a producer that enqueues work, a storage layer that holds pending jobs, and one or more workers that pull jobs off the queue and execute them.
# Producer: enqueue from the web request handler
def handle_user_signup(user):
db.create_user(user)
queue.enqueue('send_welcome_email', user_id=user.id)
queue.enqueue('provision_default_workspace', user_id=user.id)
return redirect('/dashboard') # returns immediately
# Worker: runs in a separate process
def send_welcome_email(user_id):
user = db.get_user(user_id)
email_client.send(
to=user.email,
template='welcome',
context={'name': user.first_name}
)
The web process enqueues two jobs and returns the redirect in milliseconds. The worker processes pick up the jobs and run them in the background. The user sees a fast response; the email and workspace setup happen seconds later.
Choosing a queue backend
Redis is the most common queue backend for small to medium applications. It’s fast, widely available as a managed service, and supported by mature libraries in every language. BullMQ (Node.js), Sidekiq (Ruby), Celery with the Redis broker (Python), and Faktory all use Redis.
Database-backed queues are worth considering if you want to avoid adding Redis to your stack. pg-boss (Node.js/Postgres) and Django’s django-db-queue store jobs in your existing database. The throughput ceiling is lower than Redis, but for most applications that’s fine – if you’re processing hundreds of jobs per minute rather than thousands per second, a database queue is simpler operationally.
For high-throughput or multi-team scenarios, dedicated message brokers like RabbitMQ or managed services like AWS SQS provide better durability guarantees, at the cost of more setup. SQS in particular is hard to beat for reliability on AWS – jobs are stored durably, delivery is at-least-once, and you pay per message rather than running infrastructure.
Retries and failure handling
Jobs fail. The network is unreliable, external APIs return errors, bugs in your job code surface. A queue system needs a retry strategy.
The standard approach is exponential backoff with a jitter. After the first failure, retry in 30 seconds. After the second, in two minutes. After the third, in ten minutes. Add a small random offset (jitter) to prevent all retrying jobs from hitting the same resource simultaneously after a recovery.
# BullMQ example: job options with retry configuration
const queue = new Queue('email', { connection: redisConfig })
await queue.add('send_welcome', { userId: user.id }, {
attempts: 5,
backoff: {
type: 'exponential',
delay: 30000, // 30 seconds initial delay
},
removeOnComplete: { count: 1000 },
removeOnFail: { count: 5000 },
})Jobs that exhaust all retries should go to a dead letter queue (DLQ) – a separate queue that holds failed jobs for inspection. Don’t silently discard them. A DLQ is how you find out that a category of jobs has been failing for two days, diagnose why, fix the bug, and replay the affected jobs.
Idempotency
At-least-once delivery means a job might run more than once – during a worker crash mid-execution, a network partition, or a retry after a transient failure. Your job code needs to be safe to re-run.
For jobs with side effects (sending emails, charging a card, creating records), idempotency keys are the standard solution. Before performing the action, check whether it’s already been performed for this job’s ID. If it has, skip it.
def provision_workspace(job_id, user_id):
if WorkspaceProvisionRecord.exists(job_id=job_id):
logger.info(f"Job {job_id} already processed, skipping")
return
workspace = create_workspace(user_id)
WorkspaceProvisionRecord.create(job_id=job_id, workspace_id=workspace.id)This is especially important for anything that touches an external API. Most third-party APIs and email providers accept an idempotency key header precisely because they expect you to retry on failure.
Scheduled and cron jobs
Background job systems also handle work that needs to run on a schedule: sending weekly digests, running database cleanup, polling an external API for status updates. This is the job queue’s answer to cron.
The advantage over cron is visibility – you can see scheduled jobs in the same dashboard as your on-demand jobs, track execution history, and retry on failure. Cron silently drops failed jobs unless you’ve wired up alerting explicitly.
# Celery beat example: scheduled tasks
CELERYBEAT_SCHEDULE = {
'send-weekly-digest': {
'task': 'myapp.tasks.send_weekly_digest',
'schedule': crontab(day_of_week='monday', hour=9, minute=0),
},
'cleanup-expired-sessions': {
'task': 'myapp.tasks.cleanup_expired_sessions',
'schedule': timedelta(hours=1),
},
}Observability for background jobs
Background job failures are invisible unless you make them visible. At minimum, instrument your workers with: job execution duration, failure rate per job type, queue depth (how many jobs are waiting), and worker concurrency utilization.
Most job libraries expose these metrics. BullMQ has a dashboard (Bull Board). Sidekiq has a built-in web UI. Celery integrates with Flower. If you’re using a custom setup, emit structured log events and aggregate them in your logging infrastructure.
The BullMQ documentation is the best reference for the Node.js/Redis side of this: docs.bullmq.io. For the broader patterns, AWS’s documentation on SQS gives a good treatment of the reliability properties you should expect from any queue system: docs.aws.amazon.com/AWSSimpleQueueService.