Week 22: Geo-Distributed Systems & Multi-Region Architecture

1. Why Go Multi-Region at All

Two distinct motivations, and it matters which one is actually driving a design, because they lead to different architectures:

• Latency. A user in Singapore talking to a single data center in Virginia pays real speed-of-light round-trip latency — often 200ms+ — no amount of caching or optimization within that one data center fixes this. Serving them from a nearby region instead is the only real fix.
• Availability. A single region can fail entirely — a power outage, a natural disaster, a cloud provider's regional incident. A system running in only one region has that region's availability as a hard ceiling on its own, no matter how well-architected everything inside that region is.

A system optimizing purely for latency might run active-active with users routed to their nearest region (Section 2); a system optimizing purely for availability, with latency less of a concern, might run active-passive with a standby region that only activates on failure. Real large-scale systems usually want both, which is exactly why the tradeoffs in Section 3 become unavoidable rather than optional.

2. Active-Active vs. Active-Passive

[Region A: US-East]  -- ACTIVE, serves 100% of traffic
        |
        | continuous replication
        v
[Region B: US-West]  -- PASSIVE (standby), receives replicated
                         data but serves no traffic

If Region A fails: promote Region B to active, redirect traffic
Simpler to reason about: only one region ever accepts writes,
so there's never a conflict between two regions' writes

[Region A: US-East] <--bidirectional replication--> [Region B: EU-West]
      ^ ACTIVE                                          ^ ACTIVE
      | serves nearby                                   | serves nearby
      | traffic                                         | traffic

Both regions accept writes AND serve reads, each for their
nearest users -- lower latency for everyone, but now two
regions can both write to the "same" data at nearly the
same time, and their writes need to be reconciled (Section 3)

Active-passive is the simpler, safer default: exactly one region accepts writes at any time, so there's no possibility of two regions disagreeing about what the current state of some piece of data is. Active-active buys real latency wins for a geographically spread user base, but it reintroduces a version of Week 9-10's CAP tradeoff at the region level — with two regions both able to write, what happens when they write conflicting updates to the same record before either replication catches up?

3. Deep Dive: Cross-Region Replication Lag

Replicating data between regions hundreds or thousands of miles apart takes real, physical time — tens to hundreds of milliseconds, bounded by the speed of light over that distance, not by how well-engineered the replication pipeline is. This unavoidable delay is replication lag, and it has direct consequences.

t=0ms:    User in US updates their profile bio via Region A
t=5ms:    Same user, on a different device, updates their bio
          via Region B (their phone happened to route there)
t=80ms:   Region A's write finally replicates to Region B
t=85ms:   Region B's write finally replicates to Region A

Both regions now have BOTH writes, arriving out of the order
they actually happened in from the user's perspective --
which one should "win" as the final bio?

Resolving this requires an explicit strategy, decided in advance, not discovered during an incident: last-write-wins (attach a timestamp to every write, the latest timestamp wins — simple, but can silently discard a real update if clocks aren't perfectly synchronized), vector clocks (track causal relationships between writes more precisely than a raw timestamp can, at the cost of real complexity), or application-level merge logic (for data where a sensible merge exists — e.g. a shopping cart's added items can often be unioned rather than one write simply overwriting the other). There's no conflict-free option once two regions can both accept writes to the same record; the design decision is which imperfect strategy fits the data best.

4. Designing a Failover Plan

"What happens when a region goes down" needs a concrete, tested plan — a plan that exists only on a slide is not a plan. Three questions a real failover design has to answer:

• How is failure detected? Health checks against the region need to be reliable enough not to false-positive on a brief network blip (echoing Week 20's presence-detection tuning), but fast enough not to leave users stranded for minutes on a real outage.
• What data might be lost? Any write that hadn't yet replicated out of a failing region at the moment it failed is at risk — active-passive with asynchronous replication can lose the last few hundred milliseconds of writes to the dead region; synchronous replication avoids this but adds latency to every write, another instance of Week 9-10's consistency-vs-latency tradeoff.
• How does traffic get redirected? DNS-based failover (updating which region a domain resolves to) is simple but slow to propagate (DNS caching can delay it by minutes); a global load balancer with active health checks can redirect within seconds, at the cost of more infrastructure to run.

5. Hands-on Exercise

6. Knowledge Check

Four quick questions. Expand each to check your answer.