1. What to Log, What to Alert On, and What Just Becomes Noise
Logging everything sounds safe until the volume itself becomes the problem — a genuinely important event buried in gigabytes of routine traffic logs is functionally invisible. The useful question isn't "should I log this?" (usually yes, storage is cheap) but "does this deserve an alert?" (usually no, most logs don't).
LOG, don't alert:
- Every successful, routine authentication
- Every normal HTTP request to a public page
- Routine, expected configuration changes made through normal process
LOG and ALERT:
- Repeated failed logins against one account (Week 5's credential-stuffing pattern)
- A login from a new country/impossible-travel pattern for that user
- A change to IAM policy, firewall rules, or a security group (Week 8's
exact CloudTrail example)
- A process spawning an unexpected child process on a server (a classic
post-exploitation indicator)
- Any auditd rule from Week 3 firing on a watched file
The distinction is deliberate: log broadly for forensic completeness (Week 13 needs a full record after the fact), but alert narrowly, on patterns specific enough to actually indicate something worth a human's immediate attention.
A log entry missing a timestamp, a source identifier (user/IP/service), and enough context to know what actually happened is close to useless during an investigation, no matter how many of them you have. Structured logging (consistent fields, machine-parseable format like JSON) beats unstructured free-text log lines specifically because it's searchable at scale later — exactly what Section 2 and 3 depend on.
2. Centralizing Logs
A log sitting only on the individual server that generated it is fragile in exactly the moment it matters most: if that server is compromised, the attacker may be able to delete or alter its own local logs, destroying the evidence of the compromise itself. Shipping logs to a separate, centralized, access-restricted location closes that gap.
# A lightweight shipper on each host forwards logs to a central store
# (Filebeat, Fluent Bit, or a cloud-native equivalent like CloudWatch Logs)
[Web server] --logs--> [Shipper] --> [Central log store] <-- [DB server]
^
[App server] -- logs shipped here too
# The central store lives on infrastructure the individual app/web/db
# servers can WRITE to but not modify or delete from -- so a compromised
# web server can't retroactively erase its own incriminating log entries
This is Week 8's "ship CloudTrail logs to a separate, access-restricted account" principle, generalized to any log source — the specific architectural detail that makes centralized logging a real security control rather than just operational convenience is that write access flows one way, and the servers generating logs can't delete their own history from the central store.
Keeping logs forever is expensive; deleting them too quickly means an investigation into something discovered weeks after the fact has nothing to work with. A common pattern: full-detail logs retained for 30-90 days for active investigation, with a compressed/summarized archive retained much longer (often required by Week 15's compliance frameworks) for historical audit purposes.
3. A First Look at SIEM Tooling
A SIEM (Security Information and Event Management) system takes centralized logs a step further: ingesting, normalizing, correlating and alerting on them in near-real-time, rather than being just a searchable archive.
Plain centralized logs: "here's every event, go search it"
A SIEM adds:
- Normalization: different log formats (firewall, app, cloud, OS) mapped
to a common schema, so you can search across ALL of them at once
- Correlation: connecting related events across different sources
(a failed login on the VPN + a successful login on an internal
service, from the same IP, within 5 minutes -- individually routine,
together suspicious)
- Real-time alerting: rules (Section 4) that fire the moment a pattern
matches, rather than only being discoverable via a manual search
Open-source options like the Elastic Stack (ELK) or Wazuh, and commercial platforms like Splunk, all do some version of this. The specific tool matters less at this stage than understanding the shape of the problem it solves: turning scattered logs from dozens of sources into one place you can ask "did anything suspicious happen across the whole environment" and get a real answer.
A powerful correlation engine with no logs from your VPN, your cloud audit trail, or your application layer simply can't detect anything happening in those places — it can only correlate across sources it's actually receiving. Section 1's "what to log" decision directly determines what a SIEM built on top of it can ever hope to catch.
4. Writing Detection Rules for Common Attack Patterns
A detection rule expresses a specific, suspicious pattern as a query the SIEM evaluates continuously against incoming logs — turning "we log failed logins" into "we get paged when failed logins look like an attack."
RULE: excessive-failed-logins
WHEN: event.type == "auth_failure"
GROUP BY: source_ip, target_account
THRESHOLD: more than 10 failures within 5 minutes
THEN: fire alert "possible brute-force / credential-stuffing attempt"
severity: HIGH
include: source_ip, target_account, attempted usernames
# A single failed login is routine (Section 1). Ten from the same IP,
# against the same account, in five minutes, is a specific, well-known
# attack SHAPE -- not just "a failure happened"
Good detection rules target the shape of an attack, not a single event — this is what separates a useful rule from noise. A rule that fires on every failed login individually would be useless (everyone mistypes a password sometimes); a rule that fires on the specific pattern of many rapid failures is a genuine signal.
RULE: impossible-travel
WHEN: two successful logins for the SAME account
FROM: two different countries/regions
WITHIN: a time window shorter than physically possible travel between them
THEN: fire alert "impossible travel -- possible compromised credential"
# A user logging in from New York, then from a different continent
# 20 minutes later, is either a VPN, or a stolen session/credential
# being used from a second location -- both worth a human look
The MITRE ATT&CK framework catalogs real-world attacker techniques (credential access, lateral movement, exfiltration) with documented detection strategies for each. Mapping your detection rules to specific ATT&CK techniques, rather than inventing patterns from scratch, means you're building coverage against documented, real attacker behavior — and gives you a concrete way to identify gaps in what you're not yet detecting at all.
5. Alert Fatigue & Tuning
A detection rule that fires constantly on benign activity trains the humans watching it to ignore alerts — alert fatigue is a genuinely dangerous failure mode, because it means the one real alert, buried among hundreds of false ones, gets dismissed along with the noise.
# The brute-force rule from Section 4 fires constantly -- turns out a
# legitimate internal service retries failed API auth automatically,
# looking identical to a brute-force pattern
# Bad fix: disable the rule entirely -- loses real detection coverage
# Good fix: exclude the known, legitimate source, or raise the threshold
# specifically for that known pattern, while keeping the rule sensitive
# for everything else
RULE: excessive-failed-logins
WHEN: event.type == "auth_failure"
AND source_ip NOT IN known_internal_service_ips # <- the tuning
GROUP BY: source_ip, target_account
THRESHOLD: more than 10 failures within 5 minutes
Tuning is ongoing, not a one-time setup step — new legitimate traffic patterns emerge, new attack techniques appear, and a rule set that was well-tuned six months ago drifts out of sync with reality unless it's periodically reviewed against what's actually firing and why.
Without measuring it, "this rule is noisy" is a feeling, not a fact — and rules that quietly get ignored because everyone "knows" they're noisy are effectively disabled anyway, just without anyone deciding that on purpose. Track outcomes (true positive, false positive, needs investigation) per alert, and use that data to decide which rules genuinely need tuning versus which are working as intended.
6. Hands-on Exercise
Ship logs to a local ELK/Wazuh stack, write a detection rule, then trigger and tune it
Build a small, real monitoring pipeline and prove your detection rule actually fires on the attack pattern it's meant to catch.
Part 1 — Centralize logs:
- Set up a local Elastic Stack (Elasticsearch + Kibana, via Docker Compose is fastest) or Wazuh instance.
- Configure a shipper (Filebeat, or Wazuh's agent) on a test VM or your own machine to forward system/auth logs to it.
- Confirm logs are arriving by searching for a recent, known event (e.g. your own last login) in the tool's UI.
Part 2 — Write and trigger a detection rule:
- Write a detection rule (following Section 4's shape) that fires when more than 5 failed SSH login attempts occur from the same source IP within 2 minutes.
- Deliberately trigger it: from a test machine, attempt to SSH into your monitored host 6+ times with a wrong password in quick succession.
- Confirm the alert fires, and inspect exactly what fields it captured (source IP, target account, timestamp, attempted usernames).
If your rule doesn't fire, check the raw log entries first — confirm SSH failures are actually being logged with a source IP field your rule can group by. A rule written against a field that doesn't exist (or is named differently) in your actual log format will silently never match, with no error to point you at the problem.
Part 3 — Create and fix a false positive:
- Write a small script that legitimately retries a failed API call 6+ times in under 2 minutes (simulating the "legitimate internal service" scenario from Section 5) against the same rule's trigger condition, and confirm it also fires the alert — a genuine false positive.
- Tune the rule to exclude this specific, known-legitimate source (Section 5's pattern) without disabling the rule entirely.
- Re-run both your Part 2 attack simulation and your Part 3 legitimate-retry script, and confirm the rule now fires only on the former.
This last confirmation step is the actual point of the exercise — a rule that "looks tuned" but hasn't been re-tested against both the real attack pattern and the false-positive scenario is exactly the kind of change that quietly breaks detection coverage while looking like an improvement.
7. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
Why is "log broadly, alert narrowly" a better default than alerting on everything that gets logged?
Why is "log broadly, alert narrowly" a better default than alerting on everything that gets logged?
Most logged events are routine and expected — alerting on every one would produce overwhelming noise that trains responders to ignore alerts entirely (Section 5's alert fatigue). Logging broadly still preserves the full record for later investigation (Week 13), while alerting only on specific, meaningful patterns keeps alerts actionable and worth a human's immediate attention.
Q2
Why does shipping logs to a separate, centralized store matter specifically for security, not just operational convenience?
Why does shipping logs to a separate, centralized store matter specifically for security, not just operational convenience?
If logs only exist on the server that generated them, an attacker who compromises that server can potentially delete or alter the local logs, erasing evidence of their own activity. A centralized store the source server can only write to (not modify or delete from) preserves the record even if the originating system is later fully compromised.
Q3
Why does a good brute-force detection rule fire on a pattern (many failures, grouped by source and account, within a time window) rather than on any single failed login?
Why does a good brute-force detection rule fire on a pattern (many failures, grouped by source and account, within a time window) rather than on any single failed login?
A single failed login is routine — everyone mistypes a password occasionally — and alerting on every instance would be pure noise. The specific shape of many rapid failures from one source against one account is a well-known attack pattern (credential stuffing, brute force) that's genuinely rare in legitimate use, which is exactly what makes it a meaningful signal worth an alert.
Q4
A detection rule is firing constantly on a known, legitimate internal service's behavior. What's the risk of simply disabling the rule to stop the noise?
A detection rule is firing constantly on a known, legitimate internal service's behavior. What's the risk of simply disabling the rule to stop the noise?
Disabling the rule entirely removes detection coverage for the real attack pattern it was written to catch, not just the false positive from the known service. The correct fix is scoping the rule to exclude the specific known-legitimate source while keeping it sensitive to everything else — tuning the rule's precision, rather than removing its coverage altogether.