Week 17: Observability — Logging, Metrics & Distributed Tracing

1. The Three Pillars

"Observability" is often used loosely to mean "has dashboards" — the more precise and interview-relevant framing is three distinct kinds of telemetry, each answering a different question and each with a different cost profile:

Logs answer "what happened, in detail, for this one event?" High detail, high volume, expensive to store and search at scale.
Metrics answer "how much, how often, over time?" Pre-aggregated numbers (request count, error rate, latency percentiles) — cheap to store and query even at massive scale, because detail was already thrown away in exchange for that cheapness.
Traces answer "what path did this one specific request take, and where did it spend its time?" — the only one of the three that's inherently structured around a single request crossing multiple services, which is exactly what a monolith never needed and a microservices architecture (Weeks 15–16) can't function without.

None of the three replaces the others. Metrics tell you that p99 latency spiked at 2:14pm; logs and traces tell you why — which specific requests were slow, and where. A system with dashboards but no tracing can tell you something is wrong and roughly when, but not which of fifteen services in the request path was actually responsible.

2. Structured Logging

A log line like "User 4821 checkout failed" is readable by a human scrolling through one file, but nearly useless at the scale this course targets — you can't reliably query, filter or aggregate free-text strings across millions of log lines from dozens of services. Structured logging — emitting logs as key-value data instead of prose — fixes this.

unstructured vs. structured

// unstructured -- readable, but not queryable at scale
"User 4821 checkout failed: insufficient inventory for SKU-991"

// structured -- machine-parseable, filterable, aggregatable
{
  "timestamp": "2026-08-10T14:02:33Z",
  "level": "error",
  "event": "checkout_failed",
  "user_id": 4821,
  "sku": "SKU-991",
  "reason": "insufficient_inventory",
  "trace_id": "8f3e2b91..."          // ties this log line to Section 4's trace
}

The payoff: "show me every checkout_failed event with reason: insufficient_inventory in the last hour, grouped by sku" becomes a straightforward query against structured fields, instead of a fragile regex against free text that breaks the next time someone tweaks the log message's wording. The trace_id field is the bridge to Section 4 — it's what lets a specific log line be connected back to the exact request it came from, across every service that request touched.

3. RED & USE: Two Metrics Frameworks

Rather than deciding from scratch what to measure for every service, two well-established frameworks give a solid default, depending on what's being measured.

RED -- for request-driven services

Rate:     requests per second
Errors:   failed requests per second (or as a % of Rate)
Duration: how long requests take (usually p50/p95/p99, not
          just an average -- an average hides a slow tail
          that a small but real number of users experience)

USE -- for resources (CPU, memory, disk, queues)

Utilization: % of time the resource is busy
Saturation:  how much extra work is queued, waiting for
             the resource (e.g. queue depth, Week 8)
Errors:      count of error events for this resource
             (e.g. disk I/O errors)

RED is what you'd put on a dashboard for the Booking Service from Week 16's incident: its request rate, error rate, and latency percentiles immediately show that something's wrong. USE is what you'd check next, one layer down: is the database CPU saturated, is the connection pool exhausted, is a queue backing up — the resource-level signal that explains why. A percentile detail worth stating explicitly: p99 latency, not average latency, is what actually reflects the worst experience a real user has — an average can look healthy while 1% of users wait ten seconds.

4. Deep Dive: Distributed Tracing

A single user action — "place an order" — might touch six services in Week 15's decomposed system. Distributed tracing stitches together everything that happened across all six into one coherent picture of that one request.

a trace — one request, six spans

trace_id: 8f3e2b91...

[API Gateway]      0ms -------- 420ms  (total request time)
  [Order Service]     10ms --- 400ms
    [Inventory Svc]     15ms - 60ms
    [Payment Svc]       65ms - 380ms   <-- the slow one
      [Fraud Check]       70ms - 375ms <-- THIS is where the
                                            time actually went
    [Notification Svc]  382ms - 395ms

Each service's contribution to the request is a span, with a start time and duration; the full set of spans for one request, linked by a shared trace_id, is the trace. This is what actually answers Week 16's incident-diagnosis question precisely — not "Payment Service is slow" but "Payment Service is slow specifically because the Fraud Check sub-call inside it is slow," which is a completely different, much faster fix to identify.

The mechanism that makes this possible: a trace_id is generated once, at the very first service the request touches (usually the API gateway), and propagated forward — every service that makes a downstream call passes the same trace_id along in the request headers, and includes it in every log line it emits (Section 2's example). Without deliberately propagating that ID through every hop, tracing silently breaks the moment any one service forgets to forward it.

Tracing everything is usually too expensive

Recording a full trace for every single request at high scale is often prohibitively expensive to store. Production systems typically use sampling — tracing, say, 1% of normal requests in full detail, while always tracing 100% of requests that errored or were unusually slow, so the expensive detail is captured exactly where it's most useful and skipped where it mostly wouldn't be.

5. Hands-on Exercise

Hands-on

Instrument Week 16's cascading-failure incident

Revisit Week 16's incident: Pricing Service slows down, and Order Service, the Booking API and the mobile app all start reporting errors twenty minutes later.

Requirements:

Define the RED metrics you'd put on a dashboard for Pricing Service specifically, and state which one would have shown a signal first, before any downstream service was affected.
Define one USE metric that could explain why Pricing Service slowed down at the resource level, given the incident was caused by "an unrelated database issue."
Sketch what a trace for one "get price" request would look like during the incident, as a list of spans with rough durations, showing where the time was actually being spent.
Write the structured log fields (Section 2) you'd want on Pricing Service's error logs during this incident, including the field that ties a log line back to its trace.
Given traces are sampled (Section 4's tip), explain why "always trace requests that errored or were slow" specifically would guarantee this incident's slow requests were captured, even at a 1% baseline sampling rate.

Hint

For requirement 1: Duration (specifically p99) is usually the earliest signal in a slowdown-caused incident — error rate often stays low for a while, since slow-but-eventually-successful requests aren't errors, they're just late. A dashboard watching only Errors, not Duration, would have missed the early warning entirely.

6. Knowledge Check

Four quick questions. Expand each to check your answer.

Why do metrics scale to a much higher volume of traffic than logs, at a much lower storage cost?

Metrics are pre-aggregated numbers (counts, rates, percentiles) that deliberately discard per-event detail in exchange for being cheap to store and query even at massive scale. Logs retain full per-event detail, which is exactly what makes them expensive to store and search at high volume — the two make opposite tradeoffs between detail and cost.

Why is p99 latency a more useful metric to watch than average latency?

An average can look healthy even while a real, non-trivial slice of users (the slow tail) has a genuinely bad experience — a small number of very slow requests can be averaged away by a large number of fast ones. A high percentile like p99 directly reflects what the worst-affected users are actually experiencing, which an average hides.

What specific mechanism allows a trace to reconstruct one request's full path across six different services?

A single trace_id is generated at the first service the request touches and propagated forward through every downstream call's request headers; every service that handles the request records its own span (start time and duration) tagged with that same trace_id, and all spans sharing a trace_id are assembled into one trace. If any service in the chain fails to forward the trace_id, the chain of spans breaks at that point.

Why do production systems typically sample traces (e.g. 1% of normal traffic) instead of tracing every single request in full detail?

Recording a full trace for every request at high scale is often prohibitively expensive to store, since traces carry the same kind of high-detail data as logs. Sampling — tracing only a small percentage of normal traffic, while always tracing requests that errored or were unusually slow — captures that expensive detail where it's most valuable for debugging and skips it where it usually wouldn't be needed.