Week 14: Security & DevSecOps

Every course so far has quietly deferred a security question: the Secrets you hand-waved in Week 11, the IAM policies you kept broad in Week 8 to keep things moving, the fact that every Pod in your cluster can currently talk to every other Pod. This week closes those gaps for real. You'll manage secrets properly instead of base64-hoping nobody looks, wire dependency and image scanning into the GitHub Actions pipeline from Week 6, tighten the IAM policies from Week 8 down to least privilege, and use Kubernetes NetworkPolicies to restrict which Pods can talk to which. This is also your last stop in the "core" half of the course — Week 15 onward moves into more advanced, closer-to-production territory (stateful workloads, service mesh, GitOps, multi-cloud, SRE practice) before the capstone in Weeks 25–26 pulls all of it — this week's security practices included — into one pipeline.

Module 11 of 22 Week 14 of 26 ~3–4 Hours Hands-on Exercise Included

By the end of this week, you'll be able to

  • Manage secrets with a dedicated secrets manager instead of committing or base64-hiding them
  • Scan dependencies and container images for vulnerabilities inside a CI pipeline
  • Write least-privilege IAM policies and Kubernetes NetworkPolicies

1. Secrets Management

The single most common real-world security incident in this entire course's toolchain isn't a zero-day exploit — it's a credential committed to Git. Once a secret lands in a commit, it's in the repository's history permanently, even after you delete the file in a later commit; anyone who ever cloned the repo, and anyone who can read a public fork, has it.

.gitignore — the first, weakest line of defense
.env
.env.*
*.pem
*.key
kubeconfig
secrets/

.gitignore only stops new accidental commits; it does nothing once a secret is already in history, and it does nothing to prevent an app from simply reading a static credential out of a config file forever. A real secrets manager — HashiCorp Vault, AWS Secrets Manager, or a Kubernetes-native option like External Secrets Operator — solves both problems: secrets live in one encrypted, access-controlled, audited store, and are fetched at runtime instead of baked into config.

terminal — AWS Secrets Manager
aws secretsmanager create-secret \
  --name prod/web-app/db-password \
  --secret-string '{"username":"app_svc","password":"S7!kq2vLpR"}'

aws secretsmanager get-secret-value \
  --secret-id prod/web-app/db-password \
  --query SecretString --output text

Inside Kubernetes, the External Secrets Operator pattern is the cleanest fit for what you built in Week 11: it syncs a secret from AWS Secrets Manager (or Vault) into a native Kubernetes Secret automatically, so your Deployment still consumes a normal secretKeyRef while the actual plaintext credential lives in a properly access-controlled, auditable system, and rotates without you touching a manifest.

external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: web-app-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: web-app-db   # the native k8s Secret this creates
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod/web-app/db-password
        property: password
If a secret ever gets committed, rotate it — don't just delete the commit

Rewriting Git history to remove a leaked credential (git filter-repo, force-push) doesn't undo the exposure — anyone with an existing clone, a cached fork, or a CI log that printed it still has the old value. The only real fix is treating the credential as burned and rotating it at the source; history cleanup is optional hygiene after that, not the fix itself.

2. Dependency & Container Image Scanning in CI

Every third-party package your app depends on, and every layer in your Docker base image, is code you didn't write and are trusting anyway. Week 5 covered scanning images manually with Trivy; the real value comes from running that scan automatically on every pull request in the GitHub Actions pipeline from Week 6, so a vulnerable dependency is caught before it merges, not discovered in production.

.github/workflows/ci.yml (excerpt)
jobs:
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Scan dependencies (npm audit)
        run: npm audit --audit-level=high

      - name: Build image
        run: docker build -t web-app:${{ github.sha }} .

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: web-app:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: "1"          # fail the job if matches are found
          ignore-unfixed: true    # skip CVEs with no available patch yet

exit-code: "1" is what actually gives this teeth — without it, Trivy prints a report and the job passes anyway, which is a scan nobody will ever read. The tradeoff is that a hard fail on every CRITICAL/HIGH finding can block a merge over a vulnerability with no available fix yet, which is why ignore-unfixed matters: there's no useful action a team can take on an unpatched CVE except accept the risk, so failing the build on it just trains people to ignore the scan.

terminal — the same scan run locally before pushing
trivy image --severity CRITICAL,HIGH web-app:local
trivy fs --severity CRITICAL,HIGH .   # scan source + lockfiles for known-vulnerable deps
Pin base image versions and rebuild on a schedule

A Dockerfile that doesn't pin its base image (FROM node:20 instead of FROM node:20.15.1-slim) can silently pull in new vulnerabilities between builds even with no code change. Pin the tag for reproducibility, and add a weekly scheduled workflow (on: schedule) that rebuilds and rescans the image anyway, so you catch newly disclosed CVEs in an unchanged image instead of only scanning at merge time.

3. Least-Privilege IAM Policies

Week 8 introduced IAM users, roles and policies and told you to prefer roles over long-lived access keys; this week is about tightening what those roles are actually allowed to do. Least privilege means a policy grants exactly the permissions a workload needs to do its job, nothing more — not because a wildcard policy is guaranteed to be exploited, but because it makes the blast radius of any single compromised credential as small as possible.

overly-broad-policy.json — what NOT to ship
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:*", "Resource": "*" }
  ]
}

That single statement lets the role read, overwrite, or delete every object in every S3 bucket in the account — including buckets that have nothing to do with the app it was written for. A least-privilege version names the exact actions and scopes the resource down to the specific bucket and prefix the app actually touches:

least-privilege-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::acme-web-app-uploads/user-content/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-web-app-uploads",
      "Condition": {
        "StringLike": { "s3:prefix": "user-content/*" }
      }
    }
  ]
}

Inside a Kubernetes cluster on EKS, this same idea is called IRSA (IAM Roles for Service Accounts): a specific Kubernetes ServiceAccount, not the whole cluster's worker nodes, gets bound to a specific IAM role, so a Pod compromise doesn't automatically hand over broad AWS credentials shared by everything running on that node.

Use IAM Access Analyzer to generate a policy from real usage

Writing a least-privilege policy from scratch means guessing every API call a workload will ever make. AWS IAM Access Analyzer can generate a policy from a role's actual CloudTrail activity over a time window — run the workload broadly-permissioned in a test account first, then tighten the policy down to exactly what Access Analyzer observed it actually calling.

4. Kubernetes NetworkPolicies

By default, every Pod in a Kubernetes cluster can send traffic to every other Pod, across every namespace — there's no network isolation unless you add it yourself. That's the Kubernetes equivalent of every EC2 instance sharing one security group that allows all traffic. A NetworkPolicy restricts which Pods can talk to a given Pod, and it requires a CNI plugin that enforces them (Calico, Cilium; the default kindnet in a plain kind cluster does not).

NetworkPolicies are additive and default-deny once applied: the moment any policy selects a Pod, all traffic not explicitly allowed by some policy is blocked for that Pod. A common first step is a deny-all baseline, then explicit allow rules layered on top:

deny-all.yaml — default-deny baseline for a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}       # selects every Pod in the namespace
  policyTypes:
    - Ingress
    - Egress
allow-web-to-db.yaml — explicit allow on top of the deny-all
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - protocol: TCP
          port: 5432

This pair means: nothing can reach postgres Pods by default, except Pods labeled app: web, and only on port 5432. A compromised Pod elsewhere in the namespace — even one running as root inside its container — has no network path to the database at all, which limits exactly the kind of lateral movement that turns a single compromised container into a full data breach.

A default-deny policy also blocks DNS unless you allow it

A namespace-wide egress deny-all silently breaks every Pod's ability to resolve DNS, because it blocks the outbound query to kube-dns/CoreDNS too — a very common "why is everything suddenly failing to connect anywhere" moment. Always pair a default-deny egress policy with an explicit allow rule for UDP/TCP port 53 to the cluster's DNS Pods.

5. Hands-on Exercise

Hands-on

Lock down secrets, CI, and cluster networking for a real app

Apply this week's four security practices to the app you've been building across the Kubernetes and CI/CD weeks.

Requirements:

  1. Audit your existing repo and Kubernetes manifests for any committed credentials; move any you find into --from-literal-generated Secrets or, if you have AWS access, an actual AWS Secrets Manager entry, and add a .gitignore entry to prevent recurrence.
  2. Add a security-scan job to your GitHub Actions workflow from Week 6 that runs npm audit (or your language's equivalent) and Trivy against your built image, failing the build on any CRITICAL/HIGH finding with a fix available.
  3. Take the broadest IAM policy attached to any role you created in Week 8 or Week 9's Terraform, and rewrite it to name specific actions and a specific resource ARN instead of using wildcards.
  4. Deploy a namespace-wide default-deny-all NetworkPolicy to a Calico- or Cilium-backed cluster (or note in comments that kind's default CNI doesn't enforce it, if that's your environment), then add an explicit allow rule permitting only your web tier to reach your database tier on its port.
  5. Add an explicit DNS-egress allow rule alongside the deny-all so Pods can still resolve service names, and confirm with kubectl exec that a Pod in the web tier can reach the database while a Pod outside the allowed selector cannot.
Hint

Test NetworkPolicy behavior with a throwaway debug Pod: kubectl run tmp --rm -it --image=busybox --labels="app=untrusted" -- wget -qO- --timeout=2 postgres:5432. A timeout confirms the deny is working; swap the label to match an allowed selector and re-run to confirm the allow rule works too.

6. Knowledge Check

Four quick questions. Expand each to check your answer.

Q1

Why isn't deleting a commit that contained a leaked API key enough to fix the exposure?

The key already exists in anyone's local clone, any fork, any CI log that echoed it, and possibly in cached views on the hosting platform, regardless of what later commits do. Deleting or rewriting the commit only affects the canonical repository going forward; it doesn't retroactively remove the value from copies that already exist. The only real fix is rotating the credential at the source system so the leaked value stops working.

Q2

Why does ignore-unfixed: true make sense on a Trivy scan that also has exit-code: "1"?

A CVE with no available patch can't actually be resolved by any change to the Dockerfile or dependencies, so failing the build on it blocks every merge indefinitely for a risk the team can only accept, not fix. Combining a hard failure on fixable findings with a pass on unfixed ones keeps the gate meaningful — it blocks merges that could genuinely be resolved, without training the team to routinely bypass or ignore a perpetually-red pipeline.

Q3

What's the actual security benefit of narrowing an IAM policy from "s3:*"/"Resource": "*" down to specific actions on one bucket prefix?

It shrinks the blast radius if that specific credential is ever compromised. A wildcard policy means a stolen credential can read, overwrite, or delete objects in every bucket in the account, including ones unrelated to the app; a scoped policy limits the same compromised credential to the exact prefix and actions the app legitimately needs, so an attacker who obtains it still can't touch unrelated data.

Q4

After applying a namespace-wide default-deny egress NetworkPolicy, Pods can no longer resolve any service by name. What's the likely cause and fix?

A default-deny egress policy blocks all outbound traffic by default, including the DNS queries Pods send to CoreDNS/kube-dns to resolve service names — without an explicit allow, that traffic is dropped along with everything else. The fix is adding an explicit egress allow rule permitting UDP and TCP port 53 to the cluster's DNS Pods alongside the deny-all baseline, so name resolution keeps working while everything else stays restricted.