1. The Incident Response Lifecycle
Incident response follows a well-established, repeatable structure — not because it's bureaucratic, but because trying to improvise all of this for the first time during an actual, live incident is exactly how a bad situation gets worse.
1. Prepare -- BEFORE anything happens: logging (Week 12), an IR plan,
defined roles, contact lists, tooling ready to go
2. Detect -- an alert fires (Week 12), or someone reports something odd
3. Contain -- stop the bleeding: isolate affected systems, without
destroying evidence in the process
4. Eradicate -- remove the actual cause: the malware, the compromised
account, the vulnerable code path that was exploited
5. Recover -- restore normal operation, with confidence the root
cause is actually gone, not just hidden
6. Lessons Learned -- the post-incident report (Section 5) -- what
happened, what worked, what needs to change
Notice how much of this lifecycle is Week 12's work paying off: "Prepare" is largely the logging and monitoring setup from last week, done in advance. An incident that hits a team with no logs, no plan, and no defined roles turns "contain and eradicate" into a much slower, much more improvised process.
Real incidents often loop: containing one system reveals a second compromised system, sending you back to "detect" for that new scope before you can truly move to "eradicate." Treating the lifecycle as a rigid checklist to complete once, rather than a framework you cycle through as understanding of the incident's scope grows, is a common and costly mistake.
2. Preserving Evidence
The instinct during an incident is often to fix things immediately — reboot the server, delete the suspicious file, restore from backup. Every one of those actions can destroy evidence needed to understand what actually happened, which matters both for a proper fix and, in serious cases, for legal/compliance reasons (Week 15).
Most volatile (capture FIRST, lost on reboot/shutdown):
1. CPU registers, cache
2. RAM contents (running processes, network connections, decrypted data)
3. Network state (active connections, ARP cache, routing tables)
Less volatile (survives longer, but still capture before "cleaning up"):
4. Disk contents
5. Remote logs (Week 12's centralized store -- this is WHY it's separate)
6. Physical configuration, network topology
# Rebooting a compromised server BEFORE capturing RAM throws away
# exactly the evidence most likely to show what's actively running
A disk image — a bit-for-bit copy of a compromised system's storage — is standard practice before any remediation touches the original, so investigation can continue on the copy while the (possibly still-compromised) original is handled separately.
Evidence: Disk image of web-server-03, sda1
Collected by: [name], [timestamp]
Method: dd if=/dev/sda1 of=/evidence/webserver03.img
Hash (SHA256): a1b2c3... (recorded at collection, re-verified before analysis
-- confirms the image hasn't changed since it was collected)
Stored: [access-restricted location, who has access]
Recording who collected evidence, when, and hashing it (Week 4's integrity guarantee, applied here) proves the evidence wasn't altered after collection — valuable for internal confidence in the investigation's findings, not only for the rarer cases that end up in legal proceedings. A finding nobody can trust because the evidence handling was sloppy is a wasted investigation either way.
3. Building a Forensic Timeline
The central deliverable of an investigation is usually a timeline: exactly what happened, in order, stitched together from every available log source — this is precisely why Week 12's centralized, correlated logging exists.
14:02:11 [Firewall log] Inbound connection from 203.0.113.42 to web-server-03:22
14:02:47 [Auth log] Failed SSH login: admin@web-server-03 from 203.0.113.42
14:03:15 [Auth log] Failed SSH login: admin@web-server-03 from 203.0.113.42 (x9 more)
14:04:02 [Auth log] SUCCESSFUL SSH login: admin@web-server-03 from 203.0.113.42
14:04:30 [auditd, Week 3] /etc/passwd modified by admin
14:05:12 [CloudTrail, Wk 8] New IAM access key created by admin
14:06:45 [S3 access log] GetObject on customer-data-backups/* from 203.0.113.42
# Each individual entry, from a single source, tells you almost nothing.
# Correlated across sources, in order, this is a complete story: brute-
# forced SSH, gained a foothold, escalated, and began exfiltrating data
This is the concrete payoff of every logging decision made across Weeks 3, 8, 10 and 12 — a timeline is only as complete as the logs actually available, and a gap in coverage becomes a gap in the story you can tell about what happened.
Correlating events across sources only works if every system's clock actually agrees — a server whose clock has drifted by even a few minutes can make an accurate timeline genuinely impossible to reconstruct precisely, or worse, subtly wrong in a way that misleads the investigation. NTP (Network Time Protocol), correctly configured on every system, is a small, easy-to-overlook control that this entire section depends on.
4. Containment & Eradication Decisions
Containment and eradication involve real tradeoffs, not a single obviously correct move — isolating a compromised system stops further damage, but also potentially alerts an attacker who's still watching, and might trigger destructive behavior (a "kill switch") the attacker built in for exactly this scenario.
Isolate immediately (pull network cable / disable the instance):
+ Stops further damage / exfiltration right away
- May alert the attacker, may lose volatile evidence (Section 2) if done
via a hard shutdown rather than a careful network-level isolation
- Right choice when active exfiltration is confirmed and ongoing
Monitor covertly first (leave it running, watch closely):
+ Preserves the ability to observe attacker behavior, gather more
evidence about scope and technique
- Real, ongoing risk while you watch -- more time for damage
- Right choice when scope is still unclear and the immediate risk
is judged tolerable for a short, deliberate window
There's no universal right answer — the decision depends on confirmed scope, active risk, and what's actually at stake, made by whoever the incident response plan (from Section 1's "Prepare" stage) designates as having authority to make that call, not improvised in the moment by whoever happens to notice first.
Deleting a malicious file without also revoking the compromised credential that let the attacker in leaves the actual entry point open — the attacker can simply come back through the same door. A real eradication step requires knowing, from Section 3's timeline, exactly how access was gained, not just what was done once inside.
5. Writing a Blameless Post-Incident Report
The final IR stage is the one most often skipped once the immediate crisis is over — and the one that determines whether the same incident happens again. A blameless report focuses entirely on the system and process failures that allowed the incident, explicitly avoiding assigning fault to any individual.
# Blame-focused (produces silence and cover-ups):
"Alice left the S3 bucket public, causing the breach."
-> Alice, and everyone watching, learns the lesson: hide near-misses,
don't report mistakes, don't ask for help when unsure
# Blameless (produces honest reporting and real fixes):
"A bucket was created without the org's default 'block public access'
setting, because that setting isn't enforced automatically at account
creation. Fix: enforce it as an account-level default (Week 8), not
a per-bucket manual step anyone could reasonably forget."
-> The actual system gap gets fixed, and nobody is incentivized to hide
the next near-miss out of fear
1. Summary: what happened, in 2-3 sentences, for a reader with no context
2. Timeline: the Section 3 timeline, in full
3. Root cause: the underlying system/process gap, not "who did what"
4. Impact: what was actually affected -- data, systems, users, duration
5. What went well: detection speed, an existing control that DID work
6. What needs to change: specific, assigned, trackable action items --
not vague aspirations like "be more careful"
A report that only lists failures can read as purely punitive even when it avoids naming names, and misses reinforcing the controls that DID work — the alert that fired correctly, the segmentation that limited the blast radius. Naming what worked is both honestly informative and helps the report land as genuinely blameless in practice, not just in wording.
6. Hands-on Exercise
Run a tabletop incident, build a timeline from real logs, and write the post-incident report
Simulate a real incident against your Week 12 monitoring setup, then produce the full set of IR deliverables from it.
Part 1 — Simulate an incident:
- Using the lab environment from Week 12 (or a fresh similar setup), simulate an attack chain across multiple log sources: a brute-force SSH attempt (multiple failed logins), followed by a successful login, followed by a deliberate change to a watched file (triggering your Week 3 auditd rule).
- Before touching anything to "fix" it, capture a disk image or file-level snapshot of the affected system (a VM snapshot is sufficient for this exercise) and record a chain-of-custody entry (Section 2's format) for it.
Part 2 — Build the timeline:
- Pull every relevant log entry from your simulated attack across all sources (firewall/auth logs, auditd, your SIEM/centralized store from Week 12) and assemble them into a single, ordered timeline (Section 3's format).
- Identify the exact moment containment should have happened, and justify why, using Section 4's tradeoffs (what was confirmed at that point, what was still unknown).
If your timeline has gaps or inconsistent timestamps, that's a genuine, useful finding in itself — note it explicitly in your report as a real gap in logging coverage, exactly the kind of thing a real post-incident report would flag as something to fix.
Part 3 — Write the blameless post-incident report:
- Using Section 5's structure, write a full post-incident report for your simulated incident: summary, timeline, root cause, impact, what went well, and specific action items.
- Rewrite one sentence from your first draft that names a specific person or role in a blaming way, converting it to focus on the system/process gap instead — following Section 5's example transformation.
- For each action item, make it specific and trackable (a concrete change, not "be more careful") — and note which earlier week's control it maps back to (e.g. "enforce S3 block-public-access at the account level, Week 8").
A useful test for "is this action item specific enough": could a different person, with no memory of this incident, read the action item alone and know exactly what to do? "Improve monitoring" fails this test; "add a detection rule for repeated auditd file-modification alerts within 5 minutes of a new SSH session" passes it.
7. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
Why is "reboot the compromised server immediately" often the wrong first move during an active incident?
Why is "reboot the compromised server immediately" often the wrong first move during an active incident?
Rebooting destroys the most volatile evidence first — RAM contents, active network connections, running processes — exactly the data most likely to reveal what's actively happening on the system. The order-of-volatility principle says to capture the most fragile evidence before it's lost, not to "fix" the visible problem before understanding what actually occurred.
Q2
Why does hashing a disk image at the moment of collection, and re-verifying that hash before analysis, matter for chain of custody?
Why does hashing a disk image at the moment of collection, and re-verifying that hash before analysis, matter for chain of custody?
It proves the evidence hasn't been altered since collection — if the hash matches, the image is provably identical to what was originally captured. Without this, there's no way to confirm the evidence analyzed later is genuinely the same data collected at the time of the incident, which undermines any conclusion drawn from it.
Q3
Why does building an accurate forensic timeline depend on every system's clock being properly synchronized (NTP)?
Why does building an accurate forensic timeline depend on every system's clock being properly synchronized (NTP)?
A timeline is built by correlating events across multiple log sources in time order. If different systems' clocks disagree — even by a few minutes — events that actually happened in one order can appear in a different, misleading order once merged, or seem unrelated when they were actually part of the same sequence. Accurate correlation across sources requires all sources to agree on what time it is.
Q4
Why does a blameless post-incident report tend to produce better long-term outcomes than one that names which person made the mistake?
Why does a blameless post-incident report tend to produce better long-term outcomes than one that names which person made the mistake?
Naming an individual's mistake teaches everyone watching that reporting a mistake or near-miss leads to blame — the rational response becomes hiding problems rather than surfacing them early. A blameless report instead treats the incident as evidence of a system or process gap (a missing default, a manual step someone could reasonably forget) and fixes that gap directly, which prevents the same class of incident regardless of which specific person is involved next time.