Week 18: Distributed Transactions & the Saga Pattern

1. Why Two-Phase Commit Doesn't Fit Microservices

A single database transaction relies on ACID guarantees the database engine enforces directly. The classic distributed equivalent, two-phase commit (2PC), tries to extend that same guarantee across multiple databases: a coordinator asks every participant to prepare to commit, waits for all of them to confirm they can, then tells everyone to actually commit. It works — but it has properties that make it a poor fit for the independent, separately-owned services this course has built since Week 9.

• Blocking — every participant holds its local locks from "prepare" until the coordinator's final decision arrives; if the coordinator or network is slow, every participant's resources stay locked, including rows unrelated services need.
• A single coordinator is a new point of failure — if it crashes between "prepare" and "commit," participants can be left holding a decision they can't resolve on their own.
• Tight coupling — every participating service has to support the same distributed transaction protocol and be simultaneously available for the transaction to complete at all, which works against the entire point of services that can deploy, fail, and scale independently.

The Saga pattern takes a different tradeoff entirely: instead of one atomic operation, a saga is a sequence of local transactions, each in its own service and its own database, where each step publishes an event that triggers the next. There's no cross-service lock held anywhere — the tradeoff is giving up strict consistency in favor of eventual consistency, and taking on the responsibility of undoing already-completed steps if a later one fails.

2. A Choreographed Saga with Celery

Consider placing an order that needs to reserve inventory and charge a payment, spanning three services. In a choreographed saga, each service reacts to events from the previous step and publishes its own — there's no central coordinator, just Celery tasks triggering each other through the broker from Weeks 9–10.

from celery import shared_task

@shared_task
def create_order(order_data: dict) -> None:
    order = save_order(order_data, status="pending")
    reserve_inventory.delay(order.id, order_data["items"])   # trigger next step

@shared_task
def cancel_order(order_id: int) -> None:
    # the compensating action for a failed reservation
    update_order_status(order_id, status="cancelled")

from celery import shared_task

@shared_task
def reserve_inventory(order_id: int, items: list[dict]) -> None:
    if try_reserve(items):
        confirm_payment.delay(order_id)          # trigger the next step
    else:
        cancel_order.delay(order_id)              # trigger the compensating action

Each task only knows about the next step to trigger, not the whole saga — the flow emerges from tasks calling each other via .delay(), the same async task dispatch from Weeks 9–10, rather than any single piece of code owning the full sequence. A failure at the inventory step calls cancel_order.delay() instead of the payment step, which is the compensating action that undoes the order-service's already-committed local transaction.

3. The Transactional Outbox Pattern

Section 2's chain depends on a subtle but critical guarantee: when create_order commits order to the database, the reserve_inventory.delay() call right after it must also actually reach the broker — reliably, even if the process crashes between the two. Calling .delay() directly after save_order() has a real failure window: if the process crashes right after the database commit but before the Celery call reaches RabbitMQ, the order is durable but the event is lost forever, and the saga stalls in a "pending" state nothing will ever advance.

The transactional outbox pattern closes that gap by writing the event to an outbox table in the same local database transaction as the actual business change — both succeed or both roll back together, guaranteed by the single database's own ACID properties, the same guarantee this whole lesson opened by saying doesn't extend across services.

async def create_order(db: AsyncSession, order_data: dict) -> Order:
    order = Order(**order_data, status="pending")
    db.add(order)
    await db.flush()   # order.id is now available, nothing committed yet

    outbox_event = OutboxEvent(
        aggregate_type="Order",
        aggregate_id=str(order.id),
        event_type="OrderCreated",
        payload=json.dumps({"order_id": order.id, "items": order_data["items"]}),
    )
    db.add(outbox_event)

    await db.commit()   # order AND event committed together, or neither is
    return order

A separate poller — itself a scheduled Celery task, or a change-data-capture tool like Debezium reading the database's write-ahead log — then reads unpublished rows from the outbox table and dispatches them, marking each one published only after a confirmed send:

@shared_task
def publish_pending_events() -> None:
    with SessionLocal() as db:
        pending = db.execute(
            select(OutboxEvent).where(OutboxEvent.published.is_(False))
        ).scalars().all()

        for event in pending:
            if event.event_type == "OrderCreated":
                payload = json.loads(event.payload)
                reserve_inventory.delay(payload["order_id"], payload["items"])
            event.published = True

        db.commit()

This design accepts at-least-once delivery rather than exactly-once — a crash between dispatching and marking published can cause a duplicate — which is why every Celery task in a saga needs to be idempotent: processing the same event twice must produce the same end state as processing it once, typically by checking whether that event's ID has already been handled before acting on it.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.