Week 15: Advanced Kubernetes — StatefulSets, Jobs & Operators

1. StatefulSets

A Deployment's Pods are interchangeable — any replica can be killed and replaced by an identical one with a random new name and, if you're not using a PersistentVolume, no memory of what came before. That's exactly the right model for a stateless API. It's the wrong model for something like a Postgres primary or a Kafka broker, where "which specific instance is this" and "which specific disk does it own" both matter. A StatefulSet exists for that case: each Pod gets a stable, predictable name (web-0, web-1, web-2, not a random suffix), Pods are created and scaled in strict order, and each Pod's storage follows it through restarts and rescheduling instead of being reassigned.

apiVersion: v1
kind: Service
metadata:
  name: pg
spec:
  clusterIP: None        # headless — required for StatefulSet DNS
  selector: { app: pg }
  ports:
    - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pg
spec:
  serviceName: pg          # must match the headless Service above
  replicas: 3
  selector:
    matchLabels: { app: pg }
  template:
    metadata:
      labels: { app: pg }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:      # one PVC PER POD, not shared
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests: { storage: 10Gi }

Two things here don't exist on a Deployment. First, the headless Service (clusterIP: None) doesn't load-balance at all — instead it gives each Pod its own stable DNS name, pg-0.pg, pg-1.pg, pg-2.pg, so other services can address a specific replica (the primary, say) instead of "whichever Pod the Service happens to route to." Second, volumeClaimTemplates provisions a distinct PersistentVolumeClaim per Pod — data-pg-0, data-pg-1, data-pg-2 — and if pg-1 is deleted and rescheduled, Kubernetes reattaches the same data-pg-1 volume to the new Pod rather than provisioning a fresh empty one.

2. Jobs & CronJobs

A Deployment assumes its Pod should run forever and restarts it if it exits — exactly wrong for a database migration, a report generator, or a one-off data backfill, all of which are supposed to finish. A Job runs a Pod (or several) to completion and stops; a CronJob creates a new Job on a schedule.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  backoffLimit: 3          # retry up to 3 times on failure
  activeDeadlineSeconds: 300
  template:
    spec:
      restartPolicy: Never   # Jobs cannot use "Always"
      containers:
        - name: migrate
          image: web-app:latest
          command: ["./manage.py", "migrate"]

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"          # 02:00 UTC every day
  concurrencyPolicy: Forbid       # skip a run if the previous one is still going
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: web-app-backup-tool:latest
              command: ["./backup.sh"]

concurrencyPolicy: Forbid matters more than it looks: without it, a backup job that runs long past its next scheduled trigger will simply stack a second one on top, and now two processes are racing against the same backup target. Also worth internalizing — a Job's restartPolicy can only be Never or OnFailure, never Always; a Pod that's supposed to finish can't also be a Pod Kubernetes keeps resurrecting forever.

3. The Operator Pattern

A human running Postgres by hand doesn't just start the process — they also handle failover when the primary dies, take backups on a schedule, and safely apply minor version upgrades. Codifying that operational knowledge into software that runs inside the cluster and watches over a specific kind of workload is the Operator pattern. An Operator is built from two pieces: a Custom Resource Definition (CRD), which extends the Kubernetes API with a new object type, and a controller, which watches objects of that type and continuously reconciles real cluster state toward what they declare.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-db
spec:
  instances: 3
  storage:
    size: 20Gi
  backup:
    barmanObjectStore:
      destinationPath: s3://acme-backups/orders-db
      s3Credentials:
        accessKeyId: { name: backup-creds, key: ACCESS_KEY_ID }
        secretAccessKey: { name: backup-creds, key: SECRET_ACCESS_KEY }
    retentionPolicy: "30d"

That's the entire spec — no StatefulSet, no manually wired-up replication, no backup CronJob. The CloudNativePG (or Zalando, or Crunchy) operator installed in the cluster sees this Cluster object, and its controller does the rest: provisions the underlying StatefulSet and PersistentVolumeClaims, configures streaming replication between the three instances, promotes a replica automatically if the primary fails, and runs the scheduled backups to S3 — continuously reconciling reality against this ~15-line declaration, the same reconciliation loop idea behind every built-in Kubernetes controller, just aimed at a domain (Postgres operations) Kubernetes itself knows nothing about.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.