Background Jobs in Node.js with BullMQ and Redis

The moment an API endpoint sends an email, generates a PDF or calls a slow third party inline, its response time is hostage to someone else's uptime. The fix is old and proven: acknowledge the request, enqueue the work, and let a separate worker process it with retries. In the Node ecosystem, BullMQ on Redis is our default for this job, and it has been reliable across every client project we have shipped it on.

Queues and workers in a few lines

import { Queue, Worker } from 'bullmq';

const connection = { host: 'redis', port: 6379 };
export const emailQueue = new Queue('email', { connection });

// in the API: fast, fire-and-forget
await emailQueue.add('welcome', { userId }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 3000 },
  removeOnComplete: 1000,
});

// in a separate worker process
new Worker('email', async (job) => {
  const user = await getUser(job.data.userId);
  await sendWelcomeEmail(user);
}, { connection, concurrency: 10 });

Run workers as their own deployable, not inside the API process. They scale on queue depth rather than request rate, they can be restarted without dropping HTTP traffic, and a poison job that leaks memory takes down a worker rather than your API.

Give workers the same shutdown care as the API: on SIGTERM, call worker.close(), which stops claiming new jobs and waits for active ones to finish inside your grace period. Jobs that genuinely run long should be split smaller or checkpoint their progress, because a deploy landing mid-job is a certainty over the lifetime of a system. For CPU-heavy handlers, BullMQ's sandboxed processors run each job in a separate process, keeping the worker's event loop responsive and letting a crash take out one job rather than the fleet.

Retries force you into idempotency

The retry settings above mean a job may run more than once — after a crash mid-job, after a timeout, after Redis reconnects. That is not a flaw; at-least-once delivery is the honest contract of every practical queue. Your job handlers must therefore be idempotent: charging a card twice is a disaster, while upserting a row twice is a no-op. Techniques that get you there include natural idempotency keys (job ID as the payment reference), conditional writes, and checking completion state before acting.

Design every job as if it will run twice, because one day it will.

The features you will actually use

• Delayed jobs: send the follow-up email in three days without a cron scanning the database.
• Repeatable jobs: cron-style schedules managed in code alongside the handler they trigger.
• Flows: parent jobs that wait for children, for fan-out work like processing every page of a report.
• Rate limiting per queue: stay inside a third-party API's quota by throttling the worker, not the producer.
• Priorities: let the password-reset email jump the newsletter batch.

Operating it in production

Two habits keep queue systems boring. First, watch queue depth and job age, and alert on age, not depth — a deep queue draining fast is fine, while a shallow queue whose oldest job is an hour late is an incident. Second, decide what happens to jobs that exhaust their retries: keep them in the failed set, alert on its growth, and build a small admin action to requeue after fixing the cause. Dead jobs deleted silently are bugs you have chosen not to hear about.

Set removeOnComplete and removeOnFail thresholds from day one as in the example — an unbounded completed set will quietly eat your Redis memory over months.

Queues are one of those investments that make every subsequent feature cheaper: once the rails exist, moving work off the request path becomes the easy default rather than a project.