Week 15: StatefulSets, Jobs & CronJobs

1. StatefulSets

A Deployment's Pods are interchangeable — my-app-7d9f8-abc12 could be destroyed and replaced by my-app-9c2e1-jkl78, and nothing outside the Pod cares. A StatefulSet gives each Pod a stable, predictable identity that survives across restarts.

what "stable identity" actually means

# Deployment Pod names are random suffixes:
my-app-7d9f8-abc12, my-app-7d9f8-def34

# StatefulSet Pod names are PREDICTABLE and ORDERED:
postgres-0, postgres-1, postgres-2

# Delete postgres-1 -> its replacement is named postgres-1 again, not a new
# random name, and it keeps the SAME PersistentVolumeClaim it had before.

statefulset.yaml — the shape

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres    # a "headless" Service, giving each Pod its own DNS entry
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
  volumeClaimTemplates:      # <- one PVC PER REPLICA, automatically
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

volumeClaimTemplates is the key difference from Week 11's Deployment + single PVC pattern — instead of one shared volume, each replica (postgres-0, postgres-1, postgres-2) gets its own dedicated PVC, created automatically, and keeps that exact same PVC across restarts.

This is the correct fix for Week 11's single-replica limitation

Week 11 deliberately used replicas: 1 because a shared PVC doesn't safely support multiple writers. A StatefulSet's per-replica storage is exactly what makes running a genuinely replicated stateful service (a Postgres cluster with replicas, a Kafka cluster, Elasticsearch) possible.

2. Ordered Deployment & Scaling

StatefulSets also start, stop, and scale Pods in strict order — by default, postgres-0 must be Running and Ready before postgres-1 is even created, and scaling down happens in reverse order.

watching ordered rollout

kubectl apply -f statefulset.yaml
kubectl get pods -w
# postgres-0   Pending -> ContainerCreating -> Running
# (only once postgres-0 is Ready...)
# postgres-1   Pending -> ContainerCreating -> Running
# (only once postgres-1 is Ready...)
# postgres-2   Pending -> ContainerCreating -> Running

This matters for real clustered databases: a replica often needs to join and sync against an already-running primary, so starting them out of order could mean a replica trying to sync against nothing. Ordering isn't a nice-to-have here — it's often a correctness requirement of the software itself.

Real clustered databases usually need an Operator, not just a StatefulSet

A StatefulSet gives you stable identity, storage and ordering — but genuine cluster logic (electing a primary, handling failover, running replication setup commands) usually needs a purpose-built Kubernetes Operator on top. Understanding StatefulSets is the prerequisite for understanding what an Operator adds.

3. Jobs

A Job runs a Pod to completion — unlike a Deployment, which expects its Pods to run forever, a Job expects its Pod's process to exit successfully and considers that success, not a failure to restart from.

job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 3          # retry up to 3 times on failure before giving up
  template:
    spec:
      containers:
        - name: migrate
          image: myrepo/my-app:1.5.0
          command: ["npm", "run", "migrate"]
      restartPolicy: Never   # Jobs require Never or OnFailure, not Always

terminal

kubectl apply -f job.yaml
kubectl get jobs                 # COMPLETIONS column shows e.g. 1/1 once done
kubectl logs job/db-migration    # see the migration's actual output
kubectl delete job db-migration  # Jobs aren't auto-cleaned up — remove manually or with ttlSecondsAfterFinished

This is exactly the right tool for a database migration, a one-off data backfill, or any batch task that needs to run exactly once (or a fixed number of times) and then be done — not something a long-running Deployment models correctly.

Set ttlSecondsAfterFinished to avoid clutter

By default, completed Jobs stick around indefinitely. Add spec.ttlSecondsAfterFinished: 3600 to auto-clean a Job (and its Pods) an hour after it finishes, instead of accumulating dozens of old, finished Job objects over time.

4. CronJobs

A CronJob creates a new Job on a recurring schedule, using standard cron syntax — the exact tool for backups, report generation, or cleanup tasks that need to run periodically, not continuously.

cronjob.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"        # every day at 2:00 AM, standard cron syntax
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          containers:
            - name: backup
              image: myrepo/backup-tool:1.0
              command: ["/backup.sh"]
          restartPolicy: OnFailure

terminal

kubectl apply -f cronjob.yaml
kubectl get cronjobs
kubectl get jobs                       # a new Job appears each time the schedule fires
kubectl create job manual-run --from=cronjob/nightly-backup   # trigger one immediately, for testing

That last command is genuinely useful during development — you don't want to wait until 2 AM to find out your backup script has a bug. Triggering an on-demand Job from the same template lets you test the exact same logic on your own schedule.

Looking ahead

You now have every core workload type Kubernetes offers: Deployments for stateless services, StatefulSets for ordered/stateful ones, Jobs for run-once tasks, CronJobs for scheduled ones. Week 16 turns to making all of them observable — knowing when any of them are actually healthy, not just running.

5. Hands-on Exercise

Hands-on

Run a 3-replica StatefulSet, then a Job and a CronJob doing real work

See ordered, stable identity firsthand, then build the run-once and scheduled workload types on top of something concrete.

Part 1 — A StatefulSet with per-replica storage:

Deploy a simple StatefulSet (nginx or any small image is fine — the point is observing behavior, not the app itself) with 3 replicas and a volumeClaimTemplates block.
Watch kubectl get pods -w during creation and confirm the strict ordering — each Pod fully Running before the next starts.
Run kubectl get pvc and confirm 3 separate PVCs were created automatically, one per replica.
Delete <name>-1 specifically and confirm its replacement comes back with the exact same name and reattaches to the same PVC (check the PVC's "used by" via kubectl describe pvc).

Hint

You'll also need a headless Service (clusterIP: None) matching the StatefulSet's serviceName for it to apply cleanly — this is what gives each Pod its own individually-addressable DNS entry, distinct from a normal Service's single virtual IP.

Part 2 — A Job and a CronJob:

Write a Job that runs a container performing some real, verifiable one-off task (even something simple like writing a timestamped file to a mounted volume, or making an API call and logging the result).
Apply it, confirm it reaches 1/1 completions, and check its logs to verify the work actually happened.
Convert it into a CronJob scheduled to run every few minutes (for testing — e.g. */3 * * * *), apply it, and wait for at least two scheduled runs to fire.
Manually trigger an extra on-demand run with kubectl create job --from=cronjob/..., then list all Jobs it has spawned so far and note the naming pattern Kubernetes used to keep them distinct.

Hint

CronJob schedules use your cluster's configured timezone (UTC by default on most setups) — if a run doesn't fire when you expect, double check what time it actually is from the cluster's perspective before assuming something's broken.

6. Knowledge Check

Four quick questions. Expand each to check your answer.

What does a StatefulSet give a Pod that a Deployment doesn't?

A stable, predictable name and a dedicated PersistentVolumeClaim that persist across restarts — a replaced Pod comes back with the same name and reattaches to the same storage, rather than getting a fresh random name and (in a plain Deployment) sharing storage with its siblings.

Why does StatefulSet Pod startup order matter for something like a replicated database?

A replica often needs an already-running primary to sync against when it starts. Starting Pods out of order could mean a replica attempting to join a cluster before the primary exists — ordered, sequential startup (each Pod Ready before the next starts) avoids that failure mode.

Why is restartPolicy: Never (or OnFailure) required for a Job, instead of the default Always?

A Job's whole model is "run to completion, then stop" — a container exiting successfully is the expected, desired outcome. Always restart policy is meant for long-running processes that should never intentionally exit, which conflicts directly with what a Job is for.

What's the relationship between a CronJob and a Job?

A CronJob is a template plus a schedule — on each scheduled trigger, it creates a new, ordinary Job (with its own Pods) using that template. It doesn't run continuously itself; it's purely a scheduler that spawns fresh Jobs at the configured cron interval.