Week 18: Distributed Transactions & the Saga Pattern

1. Why Two-Phase Commit Doesn't Fit Microservices

A single database transaction relies on ACID guarantees the database engine enforces directly. The classic distributed equivalent, two-phase commit (2PC), tries to extend that same guarantee across multiple databases: a coordinator asks every participant to prepare to commit, waits for all of them to confirm they can, then tells everyone to actually commit. It works — but it has properties that make it a poor fit for the independent, separately-owned services Week 17's gRPC work makes it easy to build.

• Blocking — every participant holds its local locks from "prepare" until the coordinator's final decision arrives; if the coordinator or network is slow, every participant's resources stay locked, including rows unrelated services need.
• A single coordinator is a new point of failure — if it crashes between "prepare" and "commit," participants can be left holding a decision they can't resolve on their own.
• Tight coupling — every participating service has to support the same distributed transaction protocol and be simultaneously available for the transaction to complete at all, which works against the entire point of services that can deploy, fail, and scale independently.

The Saga pattern takes a different tradeoff entirely: instead of one atomic operation, a saga is a sequence of local transactions, each in its own service and its own database, where each step publishes an event that triggers the next. There's no cross-service lock held anywhere — the tradeoff is giving up strict consistency in favor of eventual consistency, and taking on the responsibility of undoing already-completed steps if a later one fails.

2. A Choreographed Saga with BullMQ

Consider placing an order that needs to reserve inventory and charge a payment, spanning three services. In a choreographed saga, each service reacts to events from the previous step and publishes its own — there's no central coordinator, just BullMQ jobs triggering each other through the Redis-backed queue from Week 12.

import { Worker } from "bullmq";

new Worker("order-events", async (job) => {
  if (job.name === "order.create") {
    const order = await saveOrder(job.data, "pending");
    await inventoryQueue.add("inventory.reserve", {
      orderId: order.id,
      items: job.data.items,
    });
  }
}, { connection: redisConnection });

new Worker("order-events", async (job) => {
  if (job.name === "order.cancel") {
    // the compensating action for a failed reservation
    await updateOrderStatus(job.data.orderId, "cancelled");
  }
}, { connection: redisConnection });

new Worker("inventory-events", async (job) => {
  if (job.name === "inventory.reserve") {
    const { orderId, items } = job.data;

    if (await tryReserve(items)) {
      await paymentQueue.add("payment.confirm", { orderId });   // next step
    } else {
      await orderQueue.add("order.cancel", { orderId });         // compensate
    }
  }
}, { connection: redisConnection });

Each worker only knows about the next job to enqueue, not the whole saga — the flow emerges from jobs triggering each other via queue.add(), the same queue mechanism from Week 12, rather than any single piece of code owning the full sequence. A failure at the inventory step enqueues order.cancel instead of the payment step, which is the compensating action that undoes the order-service's already-committed local transaction.

3. The Transactional Outbox Pattern

Section 2's chain depends on a subtle but critical guarantee: when saveOrder commits order to the database, the inventoryQueue.add() call right after it must also actually reach Redis — reliably, even if the process crashes between the two. Calling queue.add() directly after a Prisma write has a real failure window: if the process crashes right after the database commit but before the BullMQ call reaches Redis, the order is durable but the event is lost forever, and the saga stalls in a "pending" state nothing will ever advance.

The transactional outbox pattern closes that gap by writing the event to an outbox table in the same Prisma transaction as the actual business change — both succeed or both roll back together, guaranteed by the single database's own ACID properties, the same guarantee this whole lesson opened by saying doesn't extend across services.

async function createOrder(data: CreateOrderInput) {
  return prisma.$transaction(async (tx) => {
    const order = await tx.order.create({ data: { ...data, status: "pending" } });

    await tx.outboxEvent.create({
      data: {
        aggregateType: "Order",
        aggregateId: order.id,
        eventType: "order.create",
        payload: JSON.stringify({ orderId: order.id, items: data.items }),
      },
    });

    return order;   // commits BOTH rows together, or neither
  });
}

A separate poller — itself a scheduled BullMQ job — then reads unpublished rows from the outbox table and enqueues them, marking each one published only after a confirmed enqueue:

new Worker("outbox-poller", async () => {
  const pending = await prisma.outboxEvent.findMany({ where: { published: false } });

  for (const event of pending) {
    await orderQueue.add(event.eventType, JSON.parse(event.payload));
    await prisma.outboxEvent.update({
      where: { id: event.id },
      data: { published: true },
    });
  }
}, { connection: redisConnection });

await outboxPollerQueue.add("poll", {}, { repeat: { every: 500 } });

This design accepts at-least-once delivery rather than exactly-once — a crash between enqueueing and marking published can cause a duplicate — which is why every worker in a saga needs to be idempotent: processing the same event twice must produce the same end state as processing it once, typically by checking whether that event's ID has already been handled before acting on it.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.