Week 17: GitOps — Declarative Delivery with ArgoCD

1. Push-Based CI/CD vs. Pull-Based GitOps

In the pipeline you built for the Week 15 capstone target, GitHub Actions pushes changes into the cluster: the runner authenticates outward-in with a kubeconfig secret and issues the commands that change cluster state. Two problems follow directly from that shape. First, credential exposure — a CI runner, often running third-party Actions, holds standing access broad enough to modify production. Second, and more insidious: nothing stops someone from running kubectl edit by hand later and changing a running Deployment directly, at which point the cluster silently disagrees with what's in Git, and nobody finds out until it causes an incident.

A GitOps agent like ArgoCD installed inside the cluster inverts this: it pulls from a Git repository on a polling interval (or via a webhook), computes the diff between what's declared in Git and what's actually running, and reconciles the difference automatically. CI's job shrinks to exactly one thing — build, test and push an image, then update an image tag in a Git repo. The rest of the deploy is ArgoCD's reconciliation loop, running with credentials that never leave the cluster.

Manual kubectl changes become visible, not just forbidden

The real value of pull-based reconciliation isn't that manual changes are impossible — someone with cluster access can still run kubectl edit. It's that ArgoCD detects the drift on its next sync cycle and either reports it as "OutOfSync" or actively reverts it (depending on your sync policy), turning a silent, undiscovered discrepancy into a visible, auditable one.

2. ArgoCD Applications & Sync Policies

ArgoCD's core object is the Application — it declares which Git repository and path to watch, and which cluster and namespace to reconcile into.

application.yaml — an ArgoCD Application for the orders service

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/orders-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true        # delete resources removed from Git
      selfHeal: true      # revert manual drift automatically
    syncOptions:
      - CreateNamespace=true

automated.selfHeal: true is the setting that makes ArgoCD actively correct drift rather than merely reporting it — any manual change to a resource ArgoCD manages gets reverted back to what's declared in Git on the next reconciliation pass, typically within seconds. prune: true is equally important and equally dangerous if misunderstood: it means deleting a manifest file from the Git repo deletes the corresponding resource from the cluster, which is exactly the GitOps ideal — Git fully describes desired state — but it also means a mistaken git rm is now a production deletion, not a no-op.

terminal — watching and manually triggering a sync

argocd app get orders-service
argocd app sync orders-service         # force an immediate reconcile
argocd app history orders-service      # every synced Git revision, in order
argocd app rollback orders-service 4   # roll back to a specific prior revision

argocd app rollback is worth sitting with for a moment — a rollback in a GitOps model isn't a special deploy mechanism at all, it's just telling ArgoCD to reconcile to a Git revision you've already been at before. There's no separate rollback tooling to maintain because the entire deploy history already lives in Git's commit log.

Start with selfHeal: false on a new Application

Automatic self-healing is powerful once you trust your manifests, but on a freshly onboarded Application it can turn a manual emergency fix (a quick kubectl scale during an incident) into an immediate, confusing revert. Run new Applications with manual sync or selfHeal: false until the team is confident the Git source is genuinely always correct, then turn it on.

3. Multi-Environment Promotion with Kustomize

Real deployments need staging and production to run mostly the same manifests with a few deliberate differences — a lower replica count in staging, different resource limits, a different image tag. Kustomize (built into kubectl) expresses that as a shared base plus small, environment-specific overlays, instead of copy-pasting near-duplicate YAML per environment.

directory layout

manifests/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml
    │   └── replica-patch.yaml
    └── production/
        ├── kustomization.yaml
        └── replica-patch.yaml

overlays/production/kustomization.yaml

resources:
  - ../../base
patches:
  - path: replica-patch.yaml
images:
  - name: orders-service
    newTag: v1.8.3      # the only thing that changes on a normal deploy

With this layout, promoting a change from staging to production is a small, reviewable diff — usually just bumping newTag in overlays/production/kustomization.yaml — rather than rewriting a whole manifest. Point two ArgoCD Applications at overlays/staging and overlays/production respectively, and the promotion workflow becomes: a PR that changes one line in the production overlay, reviewed and merged like any other code change, after which ArgoCD picks it up and reconciles automatically.

A production promotion PR is itself an audit trail

Because the only change needed to promote a version is a one-line diff to the production overlay, "who deployed what to production, and when, and who approved it" is answered entirely by git log and the PR review history — no separate deployment-tracking system needed on top of what Git already gives you for free.

4. Hands-on Exercise

Hands-on

Deploy an app through ArgoCD with a staging/production overlay split

Install ArgoCD locally and put both self-healing and Kustomize promotion to work.

Requirements:

Install ArgoCD on your local cluster and log in to its UI or CLI.
Structure a small app's manifests as a Kustomize base plus staging and production overlays, differing at minimum in replica count.
Create two ArgoCD Applications pointing at each overlay, with automated.selfHeal: true and prune: true on staging only.
Manually edit a running staging Deployment with kubectl edit and confirm ArgoCD reverts it back to the Git-declared state within a sync cycle.
Bump the image tag in the production overlay's kustomization.yaml via a Git commit, and confirm ArgoCD reconciles production to the new version without any kubectl command from you.
Use argocd app rollback to revert production to the previous revision and confirm the old image tag is running again.

Hint

kustomize build overlays/production renders the fully patched manifest locally, without touching a cluster — run it before committing to confirm the overlay produces exactly the YAML you expect.

5. Knowledge Check

Three quick questions. Expand each to check your answer.

What's the core difference in credential exposure between push-based CI/CD and pull-based GitOps?

Push-based CI/CD requires a runner outside the cluster to hold credentials capable of authenticating in and applying changes, which means that access has to leave the cluster's trust boundary. Pull-based GitOps reverses the direction — an agent running inside the cluster pulls from Git, so the credentials that can modify cluster state never need to exist outside it.

Why start a new ArgoCD Application with `selfHeal: false` rather than enabling it immediately?

Self-healing immediately reverts any manual change back to the Git-declared state, including a legitimate emergency fix applied by hand during an incident. Running with manual sync first gives the team a chance to confirm the Git manifests genuinely reflect intended state before letting ArgoCD enforce them automatically and unconditionally.

Why does a Kustomize base-plus-overlays structure make a production promotion easier to review than separate full manifests per environment?

With a shared base and small overlays, promoting a change to production is a one- or two-line diff — usually just an image tag — that a reviewer can read in seconds. Full, separately maintained manifests per environment mean the meaningful change is buried inside a much larger file, and there's no structural guarantee that staging and production haven't already silently diverged in ways nobody intended.