Week 12: Background Jobs & Queues

1. Why Background Jobs

Consider a signup endpoint that creates a user, then sends a welcome email through a third-party provider like SendGrid or Postmark. If sending the email happens inline in the handler, the client waits on that network call before getting a response — and if the email provider is slow or briefly down, your signup endpoint is now slow or down too, for a feature that has nothing to do with whether the account was actually created.

A job queue decouples the two: the request handler does the part that has to happen synchronously (create the user, return 201), then enqueues a job describing the email to send and returns immediately. A separate worker process — running independently of your Express app, often as its own deployed process — picks jobs off the queue and processes them, retrying on failure without ever blocking a client's request.

inline (blocking) vs. queued (non-blocking)

// Inline: request waits on the email provider's full round trip
POST /signup -> create user -> await sendEmail() -> 201 (slow, coupled to email uptime)

// Queued: request only waits on an in-memory Redis write
POST /signup -> create user -> await queue.add("welcome-email", {...}) -> 201 (fast)
                                        |
                                        v
                              [worker process, separate from the API]
                              picks up the job, sends the email, retries on failure

Not every side effect needs a queue

A cheap, fast, in-process operation (writing an audit log row in the same transaction as the main write) doesn't need this machinery. Reach for a queue when the side effect is slow, calls an external service that can fail or rate-limit you, or is something the client genuinely doesn't need to wait for to get a correct response.

2. Setting Up BullMQ Queues & Workers

BullMQ is built on Redis and reuses the same ioredis connection style from Week 11. A queue is what your API pushes jobs onto; a worker is a separate process that consumes them. Keeping them in separate files (and, in production, separate deployed processes) means a crash in a worker never takes your API down, and you can scale each independently.

terminal

npm install bullmq ioredis

src/lib/queue-connection.ts

import { Redis } from "ioredis";

// BullMQ requires this option on any connection it manages.
export function createQueueConnection(): Redis {
  return new Redis(process.env.REDIS_URL ?? "redis://localhost:6379", {
    maxRetriesPerRequest: null,
  });
}

src/queues/email-queue.ts

import { Queue } from "bullmq";
import { createQueueConnection } from "../lib/queue-connection.js";

export type WelcomeEmailJob = {
  userId: string;
  email: string;
};

export const emailQueue = new Queue<WelcomeEmailJob>("emails", {
  connection: createQueueConnection(),
});

Enqueue a job from the route that used to send the email inline — the handler no longer imports an email provider at all:

src/routes/auth.ts

import { Router } from "express";
import { prisma } from "../lib/prisma.js";
import { emailQueue } from "../queues/email-queue.js";

export const authRouter = Router();

authRouter.post("/signup", async (req, res, next) => {
  try {
    const user = await prisma.user.create({ data: req.body });

    await emailQueue.add("welcome-email", {
      userId: user.id,
      email: user.email,
    });

    res.status(201).json({ id: user.id, email: user.email });
  } catch (err) {
    next(err);
  }
});

The worker lives in its own file, started as its own process (node dist/workers/email-worker.js), separate from the Express app's entry point:

src/workers/email-worker.ts

import { Worker } from "bullmq";
import { createQueueConnection } from "../lib/queue-connection.js";
import type { WelcomeEmailJob } from "../queues/email-queue.js";
import { sendEmail } from "../lib/email-provider.js";

const worker = new Worker<WelcomeEmailJob>(
  "emails",
  async (job) => {
    console.log(`Processing job ${job.id}: welcome email for ${job.data.email}`);
    await sendEmail({
      to: job.data.email,
      subject: "Welcome!",
      body: `Thanks for signing up, user ${job.data.userId}.`,
    });
  },
  { connection: createQueueConnection(), concurrency: 5 }
);

worker.on("completed", (job) => console.log(`Job ${job.id} completed`));
worker.on("failed", (job, err) =>
  console.error(`Job ${job?.id} failed:`, err.message)
);

Keep job payloads small and re-fetchable

Pass IDs (userId) rather than large objects into a job, and have the worker re-fetch fresh data from the database when it runs. Jobs can sit in the queue for a while before processing — a payload captured at enqueue time can be stale by the time it actually runs.

3. Scheduled & Repeatable Jobs

BullMQ can also run a job on a schedule rather than immediately — useful for things like a nightly digest email or a daily report generation job. Pass a repeat option with a standard cron pattern when adding the job:

src/queues/report-queue.ts

import { Queue } from "bullmq";
import { createQueueConnection } from "../lib/queue-connection.js";

export const reportQueue = new Queue("reports", {
  connection: createQueueConnection(),
});

// Registered once, e.g. at deploy time or app startup -- BullMQ tracks
// the schedule in Redis and won't create duplicate repeat jobs on restart
// as long as the job name + repeat options stay the same.
await reportQueue.add(
  "daily-summary",
  {},
  {
    repeat: { pattern: "0 8 * * *" }, // every day at 08:00, server timezone
    jobId: "daily-summary-job",       // stable ID prevents duplicate schedules
  }
);

The worker for a repeatable job looks identical to any other worker — BullMQ handles re-enqueuing the next occurrence automatically once the current run completes:

src/workers/report-worker.ts

import { Worker } from "bullmq";
import { createQueueConnection } from "../lib/queue-connection.js";
import { generateDailySummary } from "../lib/reports.js";

const worker = new Worker(
  "reports",
  async (job) => {
    console.log(`Running scheduled job: ${job.name}`);
    await generateDailySummary();
  },
  { connection: createQueueConnection() }
);

A one-off job scheduled for the future (rather than a recurring one) uses delay instead of repeat — queue.add("reminder", data, { delay: 60 * 60 * 1000 }) runs exactly once, one hour from now. Both mechanisms rely on the same underlying delayed-job machinery BullMQ manages inside Redis; you don't need a separate scheduler service.

Register repeat schedules idempotently

Because a stable jobId prevents duplicate schedules, it's safe to call the queue.add(...) registration code on every app startup rather than as a manual one-time step — restarting your API in production won't accidentally create a second daily job running alongside the first.

4. Retries, Backoff & Failure Handling

External calls fail — a rate limit, a momentary outage, a network blip. BullMQ retries a failed job automatically when you configure attempts and a backoff strategy on the job (or as a default on the queue):

src/queues/email-queue.ts (enqueue with retry config)

await emailQueue.add(
  "welcome-email",
  { userId: user.id, email: user.email },
  {
    attempts: 5,
    backoff: {
      type: "exponential",
      delay: 5000, // 5s, 10s, 20s, 40s, 80s between attempts
    },
    removeOnComplete: { age: 3600 },   // clean up completed jobs after 1 hour
    removeOnFail: { age: 24 * 3600 },  // keep failed jobs around a day for inspection
  }
);

Exponential backoff spaces out retries with rapidly increasing delays instead of retrying immediately in a tight loop — retrying instantly against a provider that's rate-limiting you or briefly overloaded just adds more load to an already-struggling system. If a job exhausts all its attempts, it lands in the failed set rather than disappearing silently:

src/workers/email-worker.ts (failure visibility)

worker.on("failed", (job, err) => {
  if (job && job.attemptsMade >= (job.opts.attempts ?? 1)) {
    // Final attempt exhausted -- this needs a human or an alert, not a silent drop.
    console.error(`Job ${job.id} permanently failed after ${job.attemptsMade} attempts:`, err.message);
  } else {
    console.warn(`Job ${job?.id} failed, will retry:`, err.message);
  }
});

You can also inspect and act on permanently failed jobs directly from the queue, which is useful for an admin endpoint or a scheduled sweep:

inspecting failed jobs

const failedJobs = await emailQueue.getFailed(0, 20);
for (const job of failedJobs) {
  console.log(job.id, job.data, job.failedReason);
}

// Manually retry a specific failed job, e.g. after fixing a bug that caused it:
await failedJobs[0]?.retry();

If a worker process crashes mid-job — the process is killed, the machine restarts — BullMQ's stalled job detection notices the job's lock hasn't been renewed within the expected window and returns it to the queue to be picked up again, so a crash doesn't silently lose the job.

Make job processors idempotent

Because stalled jobs and retries can both cause the same job to run more than once, write processors so running them twice is safe — check "has this email already been sent for this user+type" before sending, rather than assuming exactly-once execution. Queues guarantee at-least-once delivery, not exactly-once.

5. Hands-on Exercise

Hands-on

Move welcome emails off the request path and add a daily digest

Rebuild your signup flow around a queue instead of an inline call, then add a second, scheduled job.

Requirements:

Create an emails queue and a worker in a separate file that simulates sending an email with a 1–2 second setTimeout-based delay and logs the result.
Wire your signup route to enqueue a welcome-email job instead of awaiting the send inline, and confirm with logging that the HTTP response returns before the worker finishes processing.
Configure the job with attempts: 4 and exponential backoff, then deliberately make the worker throw for a specific test email address to observe the retry timing in your logs.
Add a second queue with a repeatable job (any cron pattern of your choice, e.g. every 2 minutes for testing) that logs a fake "daily digest" summary.
Write a small script or route that calls queue.getFailed() and prints out any permanently failed jobs with their failedReason.

Hint

Use a short backoff delay (a few hundred milliseconds instead of 5 seconds) while testing the retry behavior, so you don't spend minutes watching your terminal — bump it back up to something realistic once you've confirmed the retry logic works.

6. Knowledge Check

Four quick questions. Expand each to check your answer.

Why does returning the HTTP response right after `queue.add(...)`, without waiting for the job to run, actually make the API more reliable, not less?

queue.add only has to write a small payload to Redis, which is fast and doesn't depend on any third-party service being up. The request's success no longer depends on the email provider's uptime or latency, so a slow or briefly failing external service can't turn into a slow or failing signup endpoint -- the two concerns are decoupled, and the queue's own retry logic absorbs the external failure instead of the client seeing it.

Why run the worker as a separate process from the Express API rather than defining the job processor inline in the same app?

A separate process means a crash or a slow, CPU-heavy job in the worker can't take down or stall the API process handling live HTTP requests, and vice versa -- the two failure domains are isolated. It also lets you scale them independently: if job volume spikes, you run more worker instances without touching the API, and if request volume spikes, you scale the API without over-provisioning workers.

Why is exponential backoff a better default than a fixed retry delay for a job that calls an external API?

A fixed short delay means every failed job retries at roughly the same rate almost immediately, which can pile more load onto a provider that's already struggling or actively rate-limiting you -- making the outage worse instead of giving it time to recover. Exponential backoff spaces retries out increasingly, giving a transient problem time to clear on its own while still eventually succeeding once the underlying issue resolves.

Why must a job processor be safe to run more than once, even though you configured `attempts` and it "should" only run once on success?

Queues like BullMQ guarantee at-least-once delivery, not exactly-once: if a worker process crashes after doing the real work but before BullMQ records the job as completed, stalled-job detection will hand that same job to another worker to run again. Writing processors idempotently -- checking "has this already happened" before acting, or making the action itself safe to repeat -- means that unavoidable duplicate execution doesn't turn into duplicate emails or duplicate charges.