Week 24: Databases & Stateful Workloads in the Cloud

1. Managed Databases vs. Self-Hosting

Week 15's CloudNativePG Operator makes self-hosting Postgres in Kubernetes genuinely viable — replication, failover, and backups are all handled. That doesn't make it the default right answer. RDS (or Aurora) hands the same operational concerns to AWS entirely: automated patching, automated failover, and automated backups, with none of it running as workloads your cluster has to schedule, monitor, or upgrade.

The honest trade-off: a managed database costs more per unit of compute than the equivalent self-hosted instance, and it's less flexible — a specific Postgres extension your team needs might not be supported on RDS. In exchange, a small team gets a production-grade database without needing deep, ongoing database administration expertise on staff. A larger platform team with that expertise, or a workload with unusual requirements a managed service can't meet, is where self-hosting with an Operator starts to make more sense.

resource "aws_db_instance" "orders" {
  identifier              = "orders-db"
  engine                  = "postgres"
  engine_version          = "16.3"
  instance_class          = "db.t3.medium"
  allocated_storage       = 50
  storage_type            = "gp3"
  multi_az                = true         # synchronous standby in a second AZ
  backup_retention_period = 14           # days -- required for PITR (Section 2)
  deletion_protection     = true
  skip_final_snapshot     = false
}

multi_az = true is the managed equivalent of what Week 15's Operator built manually — a standby replica AWS keeps synchronously up to date and automatically promotes if the primary fails, with the entire failover mechanism as one boolean instead of CRD configuration you own and maintain.

2. Backups, Point-in-Time Recovery & Tested Restores

A daily snapshot answers "can I get back to yesterday." Point-in-time recovery (PITR) answers a sharper question: "can I get back to 2:47pm today, one minute before someone ran a bad UPDATE with no WHERE clause." RDS achieves this by continuously archiving transaction logs between snapshots, so a restore can replay to any specific second within the retention window, not just to the last snapshot boundary.

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier orders-db \
  --target-db-instance-identifier orders-db-restored \
  --restore-time "2026-08-10T14:47:00Z"

# always restores to a NEW instance -- never overwrites the original

That restore always creates a separate new instance rather than overwriting the original — deliberately, so a bad restore doesn't compound a bad situation by destroying the only other copy of the data. The single most important habit this section teaches isn't configuring backups at all, though — it's this:

3. Read Replicas, Connection Pooling & Zero-Downtime Migrations

A single database instance eventually becomes the bottleneck under enough read traffic. A read replica — an asynchronously-updated copy AWS maintains automatically — lets read-heavy queries (a dashboard, a reporting job) run against the replica instead of competing with writes on the primary. Asynchronous replication means a small, usually sub-second lag is a normal trade-off: a replica is the wrong place to read data that was just written in the same request.

resource "aws_db_instance" "orders_replica" {
  identifier          = "orders-db-replica"
  replicate_source_db = aws_db_instance.orders.identifier
  instance_class      = "db.t3.medium"
}

At higher connection counts, RDS Proxy sits between the app and the database and pools connections — the same problem Week 18's callout about Lambda connections flagged, generalized: a fleet of Pods or Lambda invocations opening connections directly can exhaust the database's connection limit long before it runs out of CPU or memory.

Schema changes are the other place state quietly breaks a deploy. Adding a NOT NULL column to a large table with a default value can lock it for the entire rewrite in some databases — a multi-second lock is an outage for every request hitting that table. The standard safe pattern is an expand-contract migration, spread across separate deploys instead of one:

# Deploy 1 (expand): add the column as NULLABLE, no lock, app doesn't use it yet
ALTER TABLE orders ADD COLUMN status TEXT;

# Deploy 2: app writes to BOTH the old and new column; backfill existing rows
UPDATE orders SET status = 'unknown' WHERE status IS NULL;

# Deploy 3 (contract): once backfilled and app fully cut over, enforce it
ALTER TABLE orders ALTER COLUMN status SET NOT NULL;

No single step in that sequence requires the old and new application code to disagree about the schema at any point — which is exactly what makes it compatible with Week 15's rolling updates, where old and new Pods briefly run side by side during a deploy. A single-step migration that adds a NOT NULL column the application immediately depends on breaks the moment an old Pod, still running mid-rollout, tries to insert a row without it.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.