1. SLIs, SLOs & the Error Budget
An SLI (Service Level Indicator) is a specific, measured metric — "the proportion of HTTP requests that return within 300ms," not a vague notion of "fast." An SLO (Service Level Objective) is a target for that SLI over a time window — "99.9% of requests over a rolling 28 days." An SLA (Service Level Agreement) is the same idea with a contractual consequence attached (a refund, a credit) if it's missed — most internal services only need an SLO, not a customer-facing SLA.
# SLI: proportion of requests completing under 300ms, last 28 days
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
The gap between 100% and the SLO target is the error budget — a 99.9% SLO means 0.1% of requests are allowed to fail the target before the SLO itself is breached, which is not a bug tolerance, it's a deliberately spent resource. Over 28 days at roughly 40,000 requests/day, a 99.9% target allows about 1,120 slow or failed requests — that's the budget the team has to spend on deploys, experiments, and the ordinary risk of shipping change.
Chasing 100% reliability trades away velocity for marginal, often unnoticeable reliability gains — the difference between 99.9% and 99.99% is real engineering effort for most services, for a difference in outage time most users will never perceive. The error budget makes that trade-off explicit and lets a team spend it deliberately: an aggressive feature-shipping pace when the budget is healthy, a deliberate slowdown to stabilize when it's nearly spent.
2. Burn-Rate Alerting
A raw threshold alert ("error rate > 1% for 5 minutes," the pattern from Week 13) either fires too often on brief, self-correcting blips or misses a slow, steady degradation that never crosses the threshold in any single 5-minute window but still burns through the entire month's error budget by day 10. Burn-rate alerting fixes this by asking a different question: "at the current rate of failure, how fast is the error budget being consumed?"
groups:
- name: slo-burn-rate
rules:
# Fast burn: would exhaust the entire 28-day budget in under 1 day.
# Requires agreement across BOTH a 5m and 1h window to avoid alerting on a blip.
- alert: ErrorBudgetFastBurn
expr: |
(
error_rate_5m > (14.4 * 0.001)
and
error_rate_1h > (14.4 * 0.001)
)
labels: { severity: page }
annotations:
summary: "Burning error budget 14.4x too fast — will exhaust it in ~2 days"
# Slow burn: steady degradation that would take a week to exhaust the budget.
- alert: ErrorBudgetSlowBurn
expr: |
(
error_rate_1h > (3 * 0.001)
and
error_rate_6h > (3 * 0.001)
)
labels: { severity: ticket }
annotations:
summary: "Burning error budget 3x too fast — will exhaust it in ~10 days"
Two windows per alert, not one, is the deliberate design here: requiring agreement between a short window (5m/1h) and a longer one (1h/6h) filters out a transient spike that self-corrects before it's real, while still catching sustained degradation quickly. The fast-burn alert pages someone immediately because the budget will be gone in about two days at that rate; the slow-burn alert opens a ticket instead of paging, because there's realistically a week to investigate before it matters.
An alert that pages someone at 3am but doesn't correspond to a real, agreed reliability target is a common cause of on-call burnout — it trains people to ignore pages. Burn-rate alerting tied to an actual SLO gives every page a concrete justification: "this specific promise to users is genuinely at risk," which is a much higher bar than "a graph moved."
3. On-Call, Runbooks & Blameless Postmortems
A page at 3am is useless without a clear next step. A runbook is a short, specific document tied to a particular alert: what it means, the first three things to check, and how to mitigate immediately (not necessarily fully fix) the problem. It's written calmly in advance, not improvised under pressure.
1. Check the Grafana "orders-service overview" dashboard for the affected window.
2. Is a deploy correlated with the start of the burn?
-> yes: `kubectl rollout undo deployment/orders-service` (Week 12), then investigate.
-> no: check downstream dependencies (payments-service, inventory-service) for their
own error rates first — this may be a cascading failure, not local.
3. Check recent NetworkPolicy or IAM changes (Week 14) for anything that could
silently be denying valid traffic.
4. If unresolved in 15 minutes, escalate to #incident-orders and declare an incident.
5. After mitigation: file a postmortem doc within 48 hours (see below).
After the incident is mitigated, a blameless postmortem documents what happened, the timeline, the root cause, and concrete follow-up actions — deliberately framed around "what in the system allowed this to happen" rather than "who made the mistake." The distinction isn't just tone: a postmortem that assigns blame teaches people to hide mistakes and route around the process next time; one that doesn't makes it safe to report a near-miss honestly, which is precisely the information a team needs to actually prevent a recurrence.
A beautifully written postmortem that identifies a real gap — no alert existed for this failure mode, a runbook step was missing — but whose follow-up items never get prioritized against feature work has taught the team nothing that sticks. Track postmortem action items with the same rigor as any other ticket, and revisit unresolved ones at the next incident review.
4. Hands-on Exercise
Define an SLO, alert on its burn rate, and write a runbook for it
Apply the full SRE loop to an app you've deployed earlier in this course.
Requirements:
- Define one latency-based and one availability-based SLI for a deployed app, each backed by a real Prometheus query against your Week 13 metrics.
- Set an SLO for each (e.g. 99.5% availability, 95% of requests under 500ms) over a rolling 7-day window, and calculate the resulting error budget in allowed failed/slow requests.
- Write a two-window burn-rate alert (fast + slow) for the availability SLO, matching the structure in Section 2.
- Write a one-page runbook for that alert, following the format in Section 3.
- Deliberately break the app (kill Pods, introduce latency) to trigger the fast-burn alert, then write a short blameless postmortem: timeline, root cause, and at least two concrete follow-up actions.
Google's Site Reliability Engineering book (free to read online) has a full worked burn-rate table by percentile and window that's worth skimming before picking your own multipliers — 14.4x and 3x above are the commonly used defaults for a 99.9% monthly SLO, not universal constants.
5. Knowledge Check
Three quick questions. Expand each to check your answer.
Q1
What's the difference between an SLI and an SLO?
What's the difference between an SLI and an SLO?
An SLI is the measured metric itself — the actual proportion of requests meeting some condition, computed continuously. An SLO is a target set against that SLI over a specific time window, like "99.9% over 28 days." The SLI is the measurement; the SLO is the goal for that measurement.
Q2
Why does a burn-rate alert require agreement across two time windows (e.g. 5m and 1h) instead of just one?
Why does a burn-rate alert require agreement across two time windows (e.g. 5m and 1h) instead of just one?
A single short window can trigger on a brief spike that self-corrects and never meaningfully threatens the error budget, causing a false-positive page. Requiring the elevated burn rate to hold across both a short and a longer window filters out transient blips while still catching genuinely sustained degradation quickly, rather than waiting for the long window alone to confirm it.
Q3
Why does a blameless postmortem focus on "what in the system allowed this" rather than "who made the mistake"?
Why does a blameless postmortem focus on "what in the system allowed this" rather than "who made the mistake"?
Assigning individual blame teaches people to hide mistakes or under-report near-misses to avoid consequences, which starves the team of exactly the information needed to prevent a recurrence. A blameless framing makes it safe to report what actually happened honestly, so the postmortem can identify the real systemic gap — a missing alert, an unclear runbook step — rather than just a person to point at.