Week 16: Health Checks, Logging & Observability

1. Liveness, Readiness & Startup Probes

Three probe types, each answering a different question about a container, and triggering a different response when they fail:

what each probe protects against

livenessProbe    # "Is this container still working, or stuck/deadlocked?"
                  # FAILS -> Kubernetes RESTARTS the container.

readinessProbe   # "Is this container ready to receive traffic right now?"
                  # FAILS -> Pod is removed from Service Endpoints (Week 9) —
                  # no traffic routed to it, but it's NOT restarted.

startupProbe     # "Has this SLOW-STARTING app finished initializing yet?"
                  # Runs FIRST — liveness/readiness probes are held off until
                  # this succeeds, so a slow boot isn't mistaken for a hang.

deployment.yaml — all three probes, in context

spec:
  containers:
    - name: my-app
      image: myrepo/my-app:1.5.0
      startupProbe:
        httpGet:
          path: /healthz
          port: 3000
        failureThreshold: 30
        periodSeconds: 2      # allows up to 60s to start before liveness kicks in

      livenessProbe:
        httpGet:
          path: /healthz
          port: 3000
        periodSeconds: 10
        failureThreshold: 3    # 3 consecutive failures -> restart

      readinessProbe:
        httpGet:
          path: /ready
          port: 3000
        periodSeconds: 5
        failureThreshold: 2

Note /healthz and /ready are deliberately different endpoints — liveness should check "is the process fundamentally OK" (cheap, fast), while readiness can check "are my actual dependencies (database, cache) currently reachable," which is a meaningfully different and often more expensive question.

A common, dangerous mistake: liveness checking a downstream dependency

If /healthz checks your database connection and the database goes down, EVERY Pod's liveness probe fails simultaneously — Kubernetes restarts your entire fleet, which does nothing to fix a database outage and adds a self-inflicted cascading failure on top of it. Keep liveness narrowly about the process itself; put dependency checks in readiness instead.

2. Logs, kubectl-style

Before reaching for a full logging stack, know the built-in tools — they cover a surprising amount of real debugging:

kubectl logs, thoroughly

kubectl logs my-app-xyz                    # current container's logs
kubectl logs my-app-xyz -f                  # follow, live-tailing
kubectl logs my-app-xyz --previous          # logs from the PREVIOUS instance —
                                             # essential after a crash/restart, since
                                             # the current instance's logs won't show why it died
kubectl logs -l app=my-app --all-containers  # every Pod matching a label, at once
kubectl logs my-app-xyz -c sidecar-name      # a specific container, in a multi-container Pod

--previous is the one people forget under pressure — after a CrashLoopBackOff, the currently-running container's logs are from the fresh, just-started instance, not the one that actually crashed. Its logs are what tell you why.

kubectl logs doesn't scale past a handful of Pods

Once you have dozens of Pods across many nodes, you need logs aggregated centrally — a common stack is Fluent Bit (collecting) shipping to Loki or Elasticsearch (storing/searching). That's beyond this week's scope, but it's the natural next step once kubectl logs -l stops being enough.

3. Prometheus & Grafana via Helm

Prometheus scrapes and stores time-series metrics (CPU, memory, request counts, custom app metrics). Grafana visualizes them as dashboards. Together they're the most common open-source monitoring stack for Kubernetes — and, as flagged in Week 13, installed almost universally via Helm.

installing the stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace
# kube-prometheus-stack conveniently bundles Prometheus, Grafana AND Alertmanager
# together, pre-wired to scrape cluster and node metrics automatically

kubectl get pods -n monitoring     # Prometheus, Grafana and Alertmanager Pods, all Running

reaching Grafana's UI (local cluster, quick access)

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
# open localhost:3000 — default login is usually admin / prom-operator
# (check the chart's values.yaml or its NOTES.txt output for the exact default)

kube-prometheus-stack ships pre-built dashboards for cluster-level metrics (node CPU/memory, Pod counts, network) out of the box — you'll have something genuinely useful to look at before writing a single custom query.

kubectl top and Prometheus are related, not identical

kubectl top (Week 14) queries metrics-server for current, real-time snapshot numbers. Prometheus stores metrics over time, letting you see trends, historical spikes, and build alerts on sustained conditions — the two tools genuinely complement each other rather than overlapping.

4. A Dashboard & an Alert

A minimal but real Grafana panel and a Prometheus alert rule for a workload you actually care about:

a useful PromQL query for a Grafana panel

# CPU usage of your app's Pods, over time:
sum(rate(container_cpu_usage_seconds_total{namespace="default", pod=~"my-app-.*"}[5m])) by (pod)

# Restart count per Pod — a great early-warning signal:
kube_pod_container_status_restarts_total{namespace="default", pod=~"my-app-.*"}

alertrule.yaml — alert on a Pod restarting repeatedly

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
    - name: my-app
      rules:
        - alert: MyAppCrashLooping
          expr: increase(kube_pod_container_status_restarts_total{pod=~"my-app-.*"}[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "my-app has restarted more than 3 times in 15 minutes"

This PrometheusRule is itself just another Kubernetes custom resource — apply it the same way as anything else, and Prometheus (via the operator installed by kube-prometheus-stack) picks it up automatically.

Looking ahead to the capstone

Week 18 expects a monitoring dashboard as part of the capstone deliverable — treat this week's exercise as a dry run for exactly that, on a smaller, lower-stakes app first.

5. Hands-on Exercise

Hands-on

Add real probes to your app, then watch it in Grafana

Give your app genuine health-check endpoints and probes, deploy the monitoring stack, and build one dashboard panel that reflects something real about your app's behavior.

Part 1 — Probes that actually mean something:

Add a simple /healthz endpoint to your app (returns 200 if the process is alive) and, if it talks to a database or cache, a separate /ready endpoint that also checks that dependency.
Add liveness and readiness probes to your Deployment pointing at these two distinct endpoints.
Deliberately make /ready fail (e.g. temporarily point it at a database connection string that doesn't exist) and confirm via kubectl get endpoints that the Pod drops out of the Service's Endpoints — while kubectl get pods still shows it Running, not restarted.
Fix it and confirm it rejoins Endpoints automatically once /ready passes again.

Hint

This exercise is the direct, hands-on proof of the difference between liveness and readiness from Section 1 — if the Pod restarted instead of just dropping from Endpoints, double check which probe you actually broke.

Part 2 — Deploy the monitoring stack and build one panel:

Install kube-prometheus-stack via Helm, and port-forward to reach Grafana's UI.
Explore the pre-built cluster dashboards that ship with the chart — find your namespace's Pod CPU/memory in at least one of them.
Create a new dashboard with one custom panel showing your app's restart count over time (using the PromQL query from Section 4, adapted to your app's label).
Trigger a couple of restarts on purpose (an OOM kill from Week 14's exercise works well here) and confirm your panel actually reflects it.

Hint

If your query returns no data, check that Prometheus is actually scraping your namespace — the "Targets" page under Prometheus's own UI (also reachable via port-forward on its Service) shows exactly what's being scraped and whether it's succeeding.

6. Knowledge Check

Four quick questions. Expand each to check your answer.

What's the key difference in what happens when a liveness probe fails vs. a readiness probe?

A failed liveness probe causes the container to be RESTARTED. A failed readiness probe removes the Pod from the Service's Endpoints — no traffic is routed to it — but the container keeps running, untouched, until readiness passes again.

Why is it dangerous for a liveness probe to check a downstream dependency like a database?

If the database goes down, EVERY replica's liveness probe fails simultaneously, triggering a mass restart of the entire fleet — which does nothing to fix the database outage and adds a self-inflicted cascading failure on top of the original problem. Dependency checks belong in the readiness probe instead.

After a Pod crash-loops and restarts, why might kubectl logs show nothing useful about the crash?

By default, kubectl logs shows the CURRENT (freshly restarted) container instance's logs, not the previous one that actually crashed. Use kubectl logs --previous to see the logs from the instance that crashed, which is usually where the useful error is.

What does Prometheus provide that kubectl top does not?

Metrics stored over time, not just a current snapshot — enabling historical trend analysis, dashboards showing behavior over hours or days, and alert rules that fire based on sustained conditions (like restarts over a rolling window), none of which a point-in-time kubectl top reading can provide on its own.