Week 24: CI/CD & Progressive Delivery for FastAPI

1. A Real Multi-Stage Pipeline

Week 13's pipeline built an image and deployed it in essentially one step. A production-grade pipeline separates concerns into distinct, sequentially-gated stages, so a failure at any point stops the pipeline before it reaches production.

name: Test, Scan & Deploy

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install poetry && poetry install
      - run: poetry run pytest --cov

  build-and-scan:
    needs: test
    runs-on: ubuntu-latest
    outputs:
      image: ${{ steps.image.outputs.image }}
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t acme/task-service:${{ github.sha }} .
      - name: Scan image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: acme/task-service:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: "1"
          ignore-unfixed: true
      - run: docker push acme/task-service:${{ github.sha }}
      - id: image
        run: echo "image=acme/task-service:${{ github.sha }}" >> "$GITHUB_OUTPUT"

  deploy-canary:
    needs: build-and-scan
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: ./scripts/deploy-canary.sh ${{ needs.build-and-scan.outputs.image }}

Each job's needs: is what makes this a real gate rather than decoration — build-and-scan cannot start if test fails, and deploy-canary cannot start if the Trivy scan fails. This is worth verifying directly: deliberately break a test and separately reference a known-vulnerable dependency, and confirm the deploy job never runs in either case, rather than trusting the YAML at a glance.

2. Blue-Green & Canary Deploys

A plain Kubernetes rolling update replaces Pods gradually, but the new version still eventually receives all traffic with no way to hold it at a fixed, small percentage first. Two stronger patterns for a production FastAPI service:

Blue-green runs two complete, identical environments — "blue" (currently live) and "green" (the new version) — and switches all traffic from one to the other atomically, typically by repointing a load balancer or Kubernetes Service selector:

apiVersion: v1
kind: Service
metadata:
  name: task-service
spec:
  selector:
    app: task-service
    version: green   # was "blue" -- this single line switch is the whole cutover
  ports:
    - port: 80
      targetPort: 8000

Both environments run simultaneously before and briefly after the switch, which means rollback is just as instant as the deploy — flip the selector back to blue — but it doubles resource usage for the duration, and gives no opportunity to observe the new version under a small slice of real traffic before it takes everything at once.

Canary deploys address exactly that gap by shifting a small, controlled percentage of traffic first:

# Two Deployments, "task-service-stable" (10 replicas) and
# "task-service-canary" (1 replica), both labeled app=task-service
# and selected by the same Service -- Kubernetes load-balances
# across all matching Pods roughly proportional to replica count,
# giving the canary ~9% of traffic with a 10:1 replica ratio.

kubectl scale deployment/task-service-canary --replicas=1
# watch error rate and p99 latency on the canary specifically
# (tag metrics with a "version" label from Week 12's observability)
kubectl scale deployment/task-service-canary --replicas=5   # ramp up once it looks healthy

3. Feature Flags

Every deployment strategy so far ties a code change directly to user-visible behavior changing at the same moment. A feature flag breaks that coupling deliberately: the code for a new feature ships and runs in production behind a conditional, but stays off until explicitly turned on — separating deploy (the code exists and is running) from release (users can actually reach it).

@router.get("/tasks/{task_id}/summary")
async def get_summary(
    task_id: int,
    tenant: Tenant = Depends(get_current_tenant),
    flags: FeatureFlags = Depends(get_feature_flags),
) -> TaskSummaryResponse:
    if flags.is_enabled("ai-generated-summaries", tenant.id):
        return await ai_summary_service.generate(task_id)   # the new, riskier path
    return await legacy_summary_service.generate(task_id)     # the existing, proven path

With a real flagging service (Unleash, LaunchDarkly, or a simple database-backed toggle table for a smaller service), that flag can be enabled for 1% of tenants, or only internal accounts, or only a specific customer who asked for early access — controlled at runtime, with no new deploy required to change who sees it:

class FeatureFlags:
    def __init__(self, db: AsyncSession):
        self.db = db

    async def is_enabled(self, flag_name: str, tenant_id: int) -> bool:
        flag = await flag_repository.get(self.db, flag_name)
        if flag is None:
            return False
        return (
            flag.globally_enabled
            or tenant_id in flag.enabled_tenant_ids
            or (flag.rollout_percentage > 0 and tenant_id % 100 < flag.rollout_percentage)
        )

This is the direct answer to a real limitation in Sections 1–2: a blue-green switch or a canary's replica ratio controls which version of the code serves a request, but both are coarse and infrastructure-level. A feature flag controls which behavior within a single running version a specific user sees, which is a finer-grained, faster, and reversible tool for exactly the case where you want to validate one new feature's behavior, not roll out an entire new build.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.