Week 13: Observability: Monitoring, Logging & Alerting

1. Prometheus Metrics & Instrumentation

Prometheus is a time-series metrics database that works by pulling — it scrapes an HTTP endpoint (conventionally /metrics) on your app at a regular interval, rather than your app pushing data out. Your job as the app owner is exposing that endpoint; a Prometheus client library does the bookkeeping.

from prometheus_client import Counter, Histogram, generate_latest
from flask import Flask, Response, request
import time

app = Flask(__name__)

REQUEST_COUNT = Counter(
    "http_requests_total", "Total HTTP requests",
    ["method", "path", "status"]
)
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency in seconds",
    ["path"]
)

@app.before_request
def start_timer():
    request.start_time = time.time()

@app.after_request
def record_metrics(response):
    latency = time.time() - request.start_time
    REQUEST_LATENCY.labels(path=request.path).observe(latency)
    REQUEST_COUNT.labels(
        method=request.method, path=request.path, status=response.status_code
    ).inc()
    return response

@app.route("/metrics")
def metrics():
    return Response(generate_latest(), mimetype="text/plain")

Prometheus has four core metric types: Counter (only goes up — request totals, error totals), Gauge (goes up or down — queue depth, active connections), Histogram (buckets observations — request latency, response size), and Summary (similar to Histogram but computes quantiles client-side). Almost every app metric you'll write fits a Counter or a Histogram.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "web-app"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: web
      - source_labels: [__address__]
        action: replace
        regex: (.+):\d+
        replacement: ${1}:8080
        target_label: __address__

Once data is flowing, PromQL is how you query it. The single most useful pattern you'll reuse constantly is rate() over a Counter, which converts a monotonically increasing total into a per-second rate over a time window:

# requests per second, per path, averaged over the last 5 minutes
rate(http_requests_total[5m])

# error rate as a percentage of total traffic
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m])) * 100

# 95th percentile request latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

2. Grafana Dashboards

Prometheus stores and queries data; Grafana visualizes it. You point Grafana at Prometheus as a data source, then build panels — each one backed by a PromQL query — and arrange them into a dashboard.

docker compose up -d prometheus grafana
# Prometheus: http://localhost:9090
# Grafana:    http://localhost:3000  (default admin/admin, change on first login)

In Grafana, add Prometheus as a data source (http://prometheus:9090 if they're on the same Compose network), then build a panel with a query like the latency one above. A first real dashboard for any HTTP service should cover what's often called the RED method: Rate (requests/sec), Errors (error rate), Duration (latency percentiles) — three panels that answer "is this service healthy right now" faster than almost anything else you could build.

{
  "title": "p95 Latency",
  "type": "timeseries",
  "targets": [
    {
      "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
      "legendFormat": "{{path}}"
    }
  ],
  "fieldConfig": {
    "defaults": { "unit": "s" }
  }
}

Dashboards built by clicking through the UI are hard to reproduce and easy to lose. Export a dashboard as JSON (Dashboard settings → JSON Model) and commit it to your repository — the same "config as a reviewable file" principle you've applied to Terraform and Kubernetes manifests all course long applies to dashboards too.

3. Centralized Structured Logging

docker logs and kubectl logs work fine for one container on one machine you already know is misbehaving. They fall apart the moment you have multiple replicas across multiple nodes: the request you're chasing could have hit any of them, and a Pod's logs disappear the moment it's rescheduled. Centralized logging solves this by shipping every container's logs to one searchable store.

docker compose up -d loki promtail grafana
# Promtail tails container log files and ships them to Loki
# Grafana queries Loki the same way it queries Prometheus, via LogQL

Shipping logs anywhere is only half the fix, though. Plain-text lines like User 4821 logged in at 14:32 are easy for a human to read once, but impossible to reliably filter or aggregate across millions of lines. Structured logging — emitting each log line as JSON with consistent field names — makes logs queryable the same way a database table is:

{"timestamp":"2026-08-03T14:32:07Z","level":"info","event":"user_login","user_id":4821,"latency_ms":42,"trace_id":"a1b2c3d4"}
{"timestamp":"2026-08-03T14:32:11Z","level":"error","event":"payment_failed","order_id":9931,"reason":"card_declined","trace_id":"e5f6a7b8"}

{app="web-app"} | json | level="error"
{app="web-app"} | json | event="payment_failed" | rate({app="web-app"}[5m])

A trace_id field, generated once per request and passed through every log line and downstream service call for that request, is what lets you pull the full story of a single failed request out of millions of interleaved lines from every replica — it's the single highest-leverage field to add to a logging setup.

4. Alerting Rules & On-Call Basics

Dashboards are for humans looking at a screen; alerting rules are for the system to tell you something's wrong before a human notices. Prometheus evaluates alerting rules continuously and hands anything that fires to Alertmanager, which handles deduplication, grouping, and routing to a paging tool (PagerDuty, Opsgenie, or a simple Slack webhook for a small team).

groups:
  - name: web-app.rules
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
          sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "web-app error rate above 5% for 5 minutes"
          runbook: "https://wiki.internal/runbooks/web-app-errors"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 latency above 1s for 10 minutes"

The for: 5m clause matters as much as the threshold itself — it requires the condition to stay true for the full duration before firing, which absorbs brief, self-correcting blips instead of paging on every one of them. A rule with no for clause fires on the very first scrape that crosses the threshold, and is a common cause of noisy alerting.

The deeper principle behind good alerting is alert on symptoms users feel, not on causes. "Error rate above 5%" and "p95 latency above 1s" are symptoms — they mean something is actually broken for a real user right now. "CPU usage above 80%" is a cause, and causes are frequently fine: an autoscaler might already be handling it, or a batch job might legitimately spike CPU without anyone being affected. Paging on every possible cause instead of on user-facing symptoms is exactly how teams end up with alert fatigue — so many low-value pages that on-call engineers start ignoring or muting alerts, including the real ones.

5. Hands-on Exercise

6. Knowledge Check

Four quick questions. Expand each to check your answer.