1. API Gateway vs. Service Mesh
These two components solve related but distinct problems, and interviews commonly probe whether the distinction is actually understood or just name-dropped.
[Client] --north-south--> [API Gateway] --> [Service A]
|
east-west
|
v
[Service B] --> [Service C]
An API gateway (Week 11 introduced this) sits at the system's edge and handles everything about traffic entering from outside: authentication, rate limiting, request routing to the right backend service, and often response aggregation from multiple services into one client-facing response. It's a single, deliberately-placed component.
A service mesh handles traffic between services once they're already inside the system — retries, timeouts, load balancing across instances of a service, mutual TLS between services, and observability (Week 17) into every inter-service call. Rather than living in one place, a mesh's logic runs alongside every single service instance, which is what Section 2 covers.
An API gateway earns its place almost immediately — any system with an external client needs a front door. A service mesh's value scales with the number of services and the volume of inter-service traffic; a system with three or four services calling each other occasionally can usually get the resilience patterns from Section 4 with a lightweight client library, and doesn't yet need the operational overhead of running a mesh.
2. The Sidecar Pattern
A service mesh's defining implementation detail is the sidecar proxy: a small proxy process deployed alongside every service instance, intercepting all its network traffic in and out.
[Pod: Order Service]
+-- [Order Service container] -- your application code,
| unaware a sidecar exists
+-- [Sidecar proxy] -- intercepts every request in/out,
handles retries, timeouts, mTLS,
and reports metrics (Week 17)
Order Service calls Inventory Service by its normal address;
the sidecar transparently intercepts that call, applies the
mesh's policies, and forwards it to Inventory's own sidecar.
The entire point of this design is that Order Service's code never
changes — it makes a normal network call, unaware that a proxy is intercepting it.
Retry logic, timeout policy, and circuit breaking (Section 4) all live in the
sidecar's configuration, centrally managed, instead of being reimplemented (and
inevitably done inconsistently) inside every service's own codebase in whatever
language that team happens to use.
Every request now makes an extra network hop (app → sidecar → network → destination's sidecar → destination app), adding latency, and every service instance now runs an extra process consuming CPU and memory. This is exactly the kind of tradeoff Section 1's "does this system actually need a mesh yet" question is protecting against paying unnecessarily.
3. Service Discovery
In a system with dozens of services, each running multiple instances that scale up and down and get rescheduled to different machines, "what's the current network address of the Inventory Service" isn't a static fact — it's a question that has to be answered correctly, live, on every call.
[Inventory Service instance 1] --registers itself--> [Service Registry]
[Inventory Service instance 2] --registers itself-->
[Inventory Service instance 3] --registers itself-->
[Order Service] --"where is Inventory Service?"--> [Service Registry]
<--"try instance 2, at 10.0.4.17:8080"--
If instance 2 stops responding to health checks, the registry
stops returning it -- callers are automatically routed only to
healthy instances.
Each service instance registers itself with a central registry on startup and is removed (or marked unhealthy) if it stops responding to health checks — this is the same coordination problem Week 23 covers in more depth (a registry needs to agree on a consistent view of "who's alive" across a distributed set of watchers). In a service mesh, the sidecars typically handle discovery transparently: the application code just calls a logical service name, and its sidecar resolves that to a healthy instance's actual address.
4. Deep Dive: Circuit Breakers & Retries
Without protection, a single slow or failing service can bring down every service that calls it, and everything that calls those — a cascading failure. Two patterns, usually implemented in the sidecar (Section 2), prevent this.
CLOSED (normal):
requests flow through normally
--> if failure rate exceeds a threshold (e.g. 50% of the
last 20 requests failed) --> trip to OPEN
OPEN (tripped):
requests fail IMMEDIATELY, without even attempting the call
--> after a cooldown window (e.g. 30 seconds) --> HALF-OPEN
HALF-OPEN (testing recovery):
a small number of requests are allowed through as a probe
--> if they succeed --> close the circuit, back to CLOSED
--> if they fail --> re-open, back to OPEN
The counterintuitive part, worth stating explicitly in an interview: once the circuit is OPEN, calls fail fast, on purpose, without even attempting the network call. This looks like giving up, but it's protective — it stops the caller from piling up threads/connections waiting on a service that's already known to be struggling, which is exactly the mechanism that turns one failing service into a cascading outage.
Retries need equal care: naively retrying every failed call multiplies load onto a service that's already failing, making the problem worse. The standard fix is exponential backoff with jitter — each retry waits longer than the last (backoff), with some randomness added (jitter) so that many clients retrying the same failed call don't all retry at exactly the same instant and cause a synchronized second wave of load.
attempt 1: fails --> wait ~1s (+/- random jitter)
attempt 2: fails --> wait ~2s (+/- random jitter)
attempt 3: fails --> wait ~4s (+/- random jitter)
attempt 4: fails --> give up, surface the error
(combine with the circuit breaker above --
stop retrying at all once OPEN)
Retrying makes sense for a transient failure (a timeout, a momentary 503) but not for a request that failed because it was invalid (a 400, a validation error) — retrying an inherently-broken request just wastes calls without any chance of success. A resilient client distinguishes retryable failures from non-retryable ones, rather than retrying everything uniformly.
5. Hands-on Exercise
Diagnose and fix a cascading failure
In Week 15's decomposed ride-booking system, the Pricing Service starts responding slowly (2-second latency instead of its usual 50ms) due to an unrelated database issue. Twenty minutes later, the Order Service, the Booking API, and the mobile app's booking screen are all reporting errors, even though only Pricing is actually broken.
Requirements:
- Explain the mechanism by which Pricing's slowdown became a full outage for services that don't even own pricing logic — trace the cascade step by step.
- Propose a circuit breaker configuration (failure threshold, cooldown window) for calls from Order Service to Pricing Service, and explain what changes in Order Service's behavior once the breaker trips.
- Decide what Order Service should return to its own callers while the circuit is OPEN — a hard error, or a fallback (e.g. a cached or estimated price) — and justify the choice.
- Identify whether retries made this incident better or worse before the circuit breaker was added, and explain why.
- Name one thing Week 17's observability tooling would have shown that would have made this incident faster to diagnose.
For requirement 3: a fallback is only safe when a slightly-stale or approximate answer is acceptable to the product — a cached price shown with a "price may have changed" note is a reasonable fallback for browsing, but probably not for the final confirmation step right before charging a customer's card.
6. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
What's the core difference between what an API gateway handles and what a service mesh handles?
What's the core difference between what an API gateway handles and what a service mesh handles?
An API gateway handles north-south traffic — requests entering the system from outside clients, at a single, deliberately-placed edge component. A service mesh handles east-west traffic — calls between services already inside the system — with its logic distributed alongside every service instance rather than centralized in one place.
Q2
Why doesn't adopting a service mesh require changing each service's application code?
Why doesn't adopting a service mesh require changing each service's application code?
Each service is deployed alongside a sidecar proxy that transparently intercepts all its network traffic. Retry logic, timeouts, circuit breaking and mTLS all live in the sidecar's configuration, not in the service's own code — the service just makes an ordinary network call and is unaware a proxy is involved.
Q3
Why does an OPEN circuit breaker fail requests immediately, without even attempting the network call — isn't that worse than trying?
Why does an OPEN circuit breaker fail requests immediately, without even attempting the network call — isn't that worse than trying?
Failing fast protects the caller from piling up threads or connections waiting on a service already known to be struggling, and it protects the failing service from receiving additional load while it's trying to recover. Attempting every call anyway would keep adding pressure to an already-overloaded downstream service, which is exactly the mechanism that turns one failing service into a cascading outage.
Q4
Why does naive, immediate retrying of every failed request risk making an outage worse rather than better?
Why does naive, immediate retrying of every failed request risk making an outage worse rather than better?
If a service is failing because it's overloaded, every client immediately retrying every failed call multiplies the load hitting that already-struggling service, making recovery harder rather than easier. Exponential backoff with jitter spaces retries out and randomizes their timing so clients don't all hammer the service again in a synchronized second wave.