Drift, Self-Heal, and Failure Modes: When ArgoCD's Reconciliation Loop Fights Something Else

Auto-sync and self-heal are usually presented as unambiguous wins. They are — right up until something else in the cluster has a legitimate reason to change the same resource ArgoCD thinks it owns exclusively. When that happens, the result isn't a clear error. It's a quiet oscillation that looks like ArgoCD misbehaving, when it's actually doing exactly what it was told.

This article maps the three most common shapes of that collision, why each one happens, and the precise fix for each.

What Self-Heal Actually Does

ArgoCD's OutOfSync status means live cluster state doesn't match what's declared in Git. With selfHeal: true, ArgoCD doesn't just refuse to accept the drift — it actively overwrites it back to the Git-declared state on every reconciliation pass.

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

That's the right default for almost everything a Deployment touches directly. The failure modes below all share one shape: a resource ArgoCD considers itself the sole owner of is also, legitimately, being modified by something else that has no idea ArgoCD exists.

HPA vs. Self-Heal: The Replica Count Fight

This is the most common collision. A HorizontalPodAutoscaler manages spec.replicas dynamically. Git, in a naive GitOps setup, also declares spec.replicas. The result:

Load spikes ──► HPA scales Deployment to 8 replicas
                        │
                        ▼
        ArgoCD reconciliation runs
                        │
                        ▼
        Live replicas (8) != Git-declared replicas (3)
                        │
                        ▼
        selfHeal reverts Deployment back to 3 replicas
                        │
                        ▼
        HPA scales back to 8
                        │
                        └──── repeats every reconciliation interval

At a 3-minute reconciliation interval during a traffic spike, this produces roughly ten full oscillation cycles over 30 minutes — the service spends a meaningful fraction of the spike window under-provisioned, and the sawtoothing replica count reads as ArgoCD being broken rather than working exactly as configured.

The fix is ignoreDifferences scoped to that field:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-service
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

ArgoCD stops treating spec.replicas as part of what it reconciles for this resource. The HPA becomes the sole owner of that field. Worth auditing across every HPA-managed Deployment in a fleet before selfHeal: true is enabled fleet-wide — not reactively after the first incident.

Sync Waves and Partial Failure

Sync-wave ordering is a sequencing guarantee, not a transactional one. A wave-0 database migration Job that fails partway through doesn't roll back what wave-1 already applied — the Deployment's Pods may already be running against a schema mid-migration before ArgoCD retries wave 0.

For anything where partial application genuinely can't be tolerated, a PreSync hook with hook-failure-policy: Abort is the safer pattern:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-failure-policy: Abort

Abort stops the entire sync — including later waves — the moment the hook fails, rather than leaving later waves to apply against a database that's still mid-migration. Self-heal's retry behavior on a failed wave is useful for transient failures; it's the wrong tool when partial application has real consequences.

Operators Mutating ArgoCD-Owned Resources

The third shape is structurally identical to the HPA case: a Kubernetes operator — a cert-manager controller, a service mesh sidecar injector, a mutating admission webhook — modifying a field on a resource ArgoCD also tracks. ArgoCD sees a live-vs-Git diff, self-heal reverts it, the operator re-mutates it, and the oscillation repeats.

The fix is identical — ignoreDifferences scoped to the contested field:

spec:
  ignoreDifferences:
    - group: ""
      kind: Pod
      jsonPointers:
        - /spec/containers/0/resources

The harder part isn't the fix — it's noticing the collision exists. Unlike the HPA case, which announces itself through a visibly oscillating replica count, a webhook fight over a less visible field presents as intermittent OutOfSync with no obvious pattern. When an Application shows unexplained intermittent drift, check kubectl get events on the affected resource and the list of registered mutating webhooks before assuming the drift is a Git or ArgoCD problem.

Decision Framework

Read the Full Article

The full article covers:

• The precise mechanism of selfHeal: true and why it's correct by default
• The HPA oscillation failure at real numbers — 10 oscillation cycles over a 30-minute traffic spike, what it looks like on a dashboard, and why it reads as ArgoCD being broken
• The sync-wave partial failure scenario in detail — a backfill Job, a lock timeout, and application code reading nulls it wasn't written to handle
• The service mesh sidecar injector collision — why it presents as intermittent OutOfSync with no obvious trigger, and how to trace it to the actual controller
• Series takeaways across all six articles — the thread connecting hub-and-spoke architecture, ApplicationSets, CI handoff, secrets management, progressive delivery, and failure modes into one coherent GitOps discipline

👉 Drift, Self-Heal, and Failure Modes — Full Article