How Background Workers Process Jobs: Queue, Run, Retry

by Souy Soeng /October 17, 2026

Diagram of a sign-up flowing from a web app into a Redis job queue, then to a worker that sends a welcome email

Background workers are separate processes that take slow work, like sending an email, out of a user's request and run it a moment later. Your app writes a small job into a queue and replies straight away. A worker picks the job up, runs it, and tries again if something goes wrong.

This guide follows one example from start to finish: a user signs up and should get a welcome email. You will define the job, enqueue it, start a worker, add retries with backoff, and protect the system against repeats and permanent failures. The code uses JavaScript with BullMQ and Redis. Python and Ruby tools like Celery and Sidekiq follow the same pattern.

In short

A job is one small task with simple inputs, such as a user id and an email address.
The web app adds the job to a queue and responds immediately instead of doing the slow work itself.
A worker runs as its own process and loops: wait, pull, run, record the result.
Retries with exponential backoff turn temporary failures into non-events, and a cap on attempts stops endless loops.
Idempotent jobs and a dead-letter queue make retries safe and keep broken jobs visible.

Decide which work belongs in a background job

Before writing code, picture the whole system. A user signs up and the request reaches your web app. Instead of sending the welcome email itself, the app drops a job into a queue in Redis. A worker, running as a separate process, pulls the job out and talks to the email service. That gives you two paths: a fast one that answers the user, and a slower one that runs quietly in the background.

Not everything should move to that slower path. The rule is simple. If the user needs the result to see their next page, keep the work in the request. If they can get it a moment later, make it a job. Sending a welcome email, resizing an uploaded photo, or building a PDF report are good background jobs. Checking a login password or loading the page someone asked for are not.

To follow along, you need basic programming knowledge, a running queue such as Redis or a database used as a queue, and a queue library for your language. Redis is the most common choice, and the examples here use BullMQ, a Node library built on it.

Ask: can the user live without this result for a few seconds? If yes, queue it.
Keep anything the reply depends on inside the request.
Confirm you have a running queue (Redis or a database) and a queue library installed.

Define the job as one small task

A job is just a function with clear inputs. Think of it as a sticky note for your future self: it says exactly what to do and carries only the information needed to do it. In this example, a file called sendWelcome takes one data object, pulls out the user id and email address, and sends one welcome email. That is the entire job.

Pass small, simple values like ids and addresses rather than whole objects. A full user record stuffed into the job might be out of date by the time the worker runs it. Keep each job to one unit of work as well. If you send a newsletter to a thousand people, create a thousand jobs. Then one bad address fails only one job, not the whole batch.

The job takes a single data object with only the values it needs.
Inputs are ids and addresses, not full records.
Each job does one thing, such as one email to one user.

# jobs/sendWelcome.js: one job, one task
async function sendWelcome(data) {
  const { userId, email } = data;
  await mailer.send(email, 'Welcome!');
}

Enqueue the job and reply to the user fast

Enqueueing is the moment your app stops doing the slow work and simply writes down what needs doing. It is a tiny write to Redis, usually much quicker than talking to an email provider. Import Queue from BullMQ, create a queue named emails, and inside your sign-up route call emails.add with the job name, welcome, and its small data object. Then send the reply straight away.

Inside, the app does two quick things: save the user, then add the job. The job lands in Redis as a small record, something like job welcome, user id 42, status waiting. A job is just data. Even if no worker is running yet, it is not lost. It sits safely in Redis until a worker comes online.

Two details matter here. The producer and the worker must use the same queue name, and a typo is the classic reason jobs pile up and never run. The route should also await only the add call, never the email. If you call the mailer in the route as well, you lose the whole benefit of the queue.

The queue name in the app matches the name the worker listens to.
The route awaits the add call only, not the email send.
The user sees a response before the email goes out.

import { Queue } from 'bullmq';
const emails = new Queue('emails');
# inside your signup route
await emails.add('welcome', { userId, email });
res.send('Account created!');

Start a worker process and understand its loop

A job waiting in the queue needs something to run it. Create a file called worker.js, import Worker from BullMQ, and create a worker on the same queue name, emails. Give it a function that receives each job and calls sendWelcome with the job's data. Run it with node worker.js. The worker lives in its own process: your web server does not load it, and it does not load your web server.

Once running, the worker repeats a simple loop. It waits for a job, pulls one and locks it so no other worker grabs it, runs the handler, and records the result as done or failed. Then it starts again. This loop looks the same in almost every job system, which is why Celery, Sidekiq and BullMQ feel familiar once you know it.

Because each pulled job is locked, several workers can share one queue safely, so you can scale without changing code. A concurrency setting also lets one worker handle several jobs at once, which suits emails that mostly wait on the network. Finally, if sendWelcome throws an error, the library marks the job as failed. That signal is what retries depend on.

The worker listens to the same queue name as the app.
A process manager or its own container restarts the worker if it crashes.
The handler throws an error on failure instead of hiding it.
Add workers or raise concurrency when the queue grows.

# worker.js
import { Worker } from 'bullmq';
new Worker('emails', async (job) => {
  await sendWelcome(job.data);
});
# run it: node worker.js

Configure retries with exponential backoff

In the real world, some jobs will fail. Most failures are temporary: a timeout, a busy email provider, a network blip. Trying again a little later usually works. To set this up, go back to your add call and pass a third argument with options. Attempts set to five gives the job up to five tries in total. A backoff of type exponential with a delay of 1,000 milliseconds means the first wait is one second, and each wait after that doubles.

Here is one welcome email for user 42 while the provider is having a rough minute. Attempt one times out, so the worker waits one second. Attempt two times out, so it waits two seconds. Attempt three gets a busy response, so it waits four seconds. Attempt four sends the email. The user never knew anything went wrong, and the fifth attempt was spare headroom.

Waiting longer each time matters because hammering an overloaded service every second only makes things worse. Spacing retries gives it room to recover. Setting a limit matters too: a job with a broken email address will never succeed, so retrying it forever just wastes work.

Attempts are capped at a sensible number, such as five.
Backoff is exponential, so waits grow: 1, 2, 4 seconds and so on.
Permanent failures stop retrying once attempts run out.

# retry up to 5 times, waiting longer
await emails.add('welcome', data, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
});

Confirm your jobs complete

With every step in place, check that it really works. Open two terminals. Start your web app in one and run node worker.js in the other. Sign up a few test users from the browser. The web app adds welcome jobs to the queue, the worker processes them, and each email goes out shortly after its sign-up.

Make the results visible by listening to the worker's completed and failed events and printing the job id. Every job then leaves a clear trail in the terminal. Next, break it on purpose: make sendWelcome throw an error for one test user. You should see that job fail, wait, and try again with growing gaps. If you do, your retries work.

Web app and worker run as two separate processes.
Completed and failed events are logged with job ids.
A deliberately failing job retries with longer waits each time.

Fix the most common background worker problems

A few problems show up again and again. None of them mean your design is wrong; they are small configuration or habit issues with well-known fixes. Duplicates happen when a retry sends the email twice, and the fix is to make the job idempotent. Stuck jobs sit waiting forever, usually because the worker is not running or listens to a different queue name.

Endless failures happen when a bad job keeps retrying, so cap attempts and use a dead-letter queue. A lack of connection usually means the Redis host or port is wrong. Before changing anything, read the error. Most queue libraries store the failure reason on the failed job itself, and it usually points straight at the fix.

Duplicate emails: make the job idempotent.
Stuck jobs: check the worker runs and uses the same queue name.
Endless failures: cap attempts and add a dead-letter queue.
No connection: check the Redis host and port.

Protect jobs with idempotency and a dead-letter queue

Retries are only safe if running a job twice causes no harm. An idempotent job checks before acting. For example, it records welcome sent for user 42 and skips the email if that record already exists. A retry after a half-finished run then cannot send a second email. Running the job twice has the same effect as running it once.

Jobs that keep failing need somewhere to go other than round and round forever. When a job uses up all its attempts, it moves to a dead-letter queue of failed jobs. There you can inspect it, fix the cause, and replay it. Check this queue regularly or set up an alert, so broken jobs get fixed instead of being silently forgotten.

The job checks whether its work is already done before acting.
Jobs that exhaust their attempts move to a dead-letter queue.
Someone reviews or is alerted about the dead-letter queue.

Key takeaways

Define small jobs with clear, simple inputs.
Enqueue the job and respond fast, so the user never waits.
Run workers as separate processes that loop over the queue.
Retry temporary failures with exponential backoff and a cap on attempts.
Make jobs idempotent so retries are safe.
Park repeated failures in a dead-letter queue for review.

Frequently asked questions

What is a background worker?

A background worker is a separate process that runs slow or optional tasks outside the user's request. The app adds a job to a queue, and the worker picks it up and runs it a moment later.

Do I need Redis to use a job queue?

Redis is the most common choice, and libraries like BullMQ are built on it. You can also use a database as a queue if that is what you already run.

What does exponential backoff mean for retries?

It means the wait between attempts grows each time, usually by doubling. With a one-second delay, the waits are one, two, then four seconds, which gives a struggling service time to recover.

Why are my jobs stuck in the queue and never running?

The usual causes are that no worker is running or that the worker listens to a different queue name than the app uses. Check both, and check the Redis host and port if the worker cannot connect.

How do I stop a retry from sending the same email twice?

Make the job idempotent. Record that the email was sent for that user and have the job skip sending if the record already exists.

What is a dead-letter queue?

It is a place where jobs go after they have used all their retry attempts. Instead of retrying forever, they wait there so a person can inspect the error, fix the cause, and replay them.

Watch the full video on YouTube →

Tag

Ad