There's a moment every platform team hits: Argo CD is syncing beautifully, deployments are declarative, and yet moving a release from dev to staging to production still means a human editing a values file and hoping. The deployment problem is solved; the promotion problem isn't.
In this article, I want to walk through how we close that gap on AWS: separate EKS clusters per environment, ECR as the registry, OIDC federation everywhere instead of static credentials, ApplicationSets instead of copy-pasted manifests, and — critically — automated verification gates backed by real metrics before anything gets promoted.
This is the architecture my team runs variations of in production, and the reasoning behind each decision.
The core idea: promotion is a Git commit, not a deployment
Before touching any tool, it's worth being precise about what "promotion" means in a GitOps world.
Argo CD's job is narrow and it does it extremely well: make the cluster match what Git says. It does not decide when a new version should move from dev to staging to production. That decision — historically made by a human editing a values file, or by a brittle CI job running sed against your repo — is the promotion problem.
Kargo exists to own exactly that gap. It watches your artifact sources (ECR, Git, Helm repos), models each environment as a Stage, packages new artifact versions as Freight, and promotes that Freight between stages by writing commits to your GitOps repo. Argo CD then does what it always does: reconcile.
The result is a clean separation of concerns:
- GitHub Actions builds, tests, signs, and pushes — it never touches a cluster.
- Kargo decides what version belongs in which environment, and records that decision in Git.
- Argo CD makes each cluster converge on what Git declares.
Every environment's state is a commit. Every promotion is auditable. Rollback is git revert. That's the whole philosophy.
Environment isolation: clusters and accounts, not namespaces
The single biggest gap between tutorial GitOps and production GitOps is environment isolation.
Namespaces on a shared cluster do not give you blast-radius isolation. A misbehaving controller, a noisy neighbor exhausting node resources, a cluster upgrade gone wrong — all of these take dev and prod down together. In a real AWS setup:
- One EKS cluster per environment, ideally in separate AWS accounts (dev, staging, prod) under AWS Organizations. Account boundaries are the strongest isolation primitive AWS gives you — IAM, billing, service quotas, and security tooling all scope naturally.
- A shared services account hosting Argo CD and Kargo on a management cluster, registering the workload clusters as deployment targets. Alternatively, run an Argo CD instance per cluster and let Kargo (centralized) drive all of them through Git — this keeps the prod cluster from needing inbound access from anywhere.
- ECR in the shared account, with cross-account pull permissions granted to each workload account via registry policies. One image, one immutable digest, promoted by reference — never rebuilt per environment.
That last point deserves emphasis: the artifact that reaches production must be byte-identical to the one validated in staging. Rebuilding "the same" image per environment silently invalidates everything your pipeline verified. Promote digests, not tags.
CI on GitHub Actions: OIDC or nothing
If your GitHub Actions workflow authenticates to AWS with an AWS_ACCESS_KEY_ID stored in repository secrets, that's a standing credential waiting to leak. GitHub's OIDC provider lets each workflow run exchange a short-lived, cryptographically verifiable token for a scoped IAM role session:
permissions:
id-token: write
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::<SHARED_ACCT>:role/gha-ecr-push-frontend
aws-region: eu-west-1
- uses: aws-actions/amazon-ecr-login@v2
- name: Build, tag, push
run: |
IMAGE=$ECR_REGISTRY/frontend
docker build -t $IMAGE:$VERSION .
docker push $IMAGE:$VERSION
The IAM role's trust policy pins the exact repository and branch (repo:digitalzone/frontend:ref:refs/heads/main), so a compromised fork or a rogue workflow in another repo simply cannot assume it. Scope one role per service, with push rights to only that service's ECR repository.
Two additions that cost little and pay off in audit season:
- Sign images with cosign (keyless, via the same OIDC identity) and verify signatures at admission time with a policy controller on each cluster. Provenance becomes enforceable, not aspirational.
- Generate an SBOM at build time and attach it as an OCI artifact. When the next log4shell lands, you'll answer "are we affected?" in minutes.
Semantic tags (v1.4.2) remain useful for humans, but your promotion machinery should carry the digest end to end.
Repository and manifest structure that scales
I follow a polyrepo model for application code — each microservice owns its repo, its CI, and its release cadence — with a single GitOps repo as the deployment control plane:
platform-gitops/
├── charts/ # One Helm chart per service
│ └── frontend/
├── envs/
│ ├── dev/frontend/values.yaml
│ ├── staging/frontend/values.yaml
│ └── prod/frontend/values.yaml
├── argocd/
│ └── applicationsets/
└── kargo/
└── frontend/ # Warehouse, Stages, PromotionTasks
A rule I hold firm on: environments are directories, not branches. Long-lived dev/staging/prod branches turn every promotion into a merge with drift and conflict potential. A single main branch with per-environment values directories means the diff between environments is always visible in one git diff — which is exactly what you want at 2 a.m. during an incident.
Instead of hand-maintaining an Argo CD Application per service per environment (that's N×M YAML files that will drift), generate them with an ApplicationSet:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: services
spec:
generators:
- matrix:
generators:
- git:
repoURL: https://github.com/digitalzone/platform-gitops
directories:
- path: envs/*/*
- list:
elements:
- env: dev
cluster: https://dev-cluster-endpoint
- env: staging
cluster: https://staging-cluster-endpoint
- env: prod
cluster: https://prod-cluster-endpoint
template:
metadata:
name: '{{path.basename}}-{{env}}'
spec:
project: platform
source:
repoURL: https://github.com/digitalzone/platform-gitops
path: charts/{{path.basename}}
helm:
valueFiles:
- ../../envs/{{env}}/{{path.basename}}/values.yaml
destination:
server: '{{cluster}}'
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
Adding a new service to every environment becomes: add a chart, add three values files, done. No new Application YAML, ever.
One deliberate asymmetry: I keep selfHeal and prune fully automated in dev and staging, but in production I pair automated sync with sync windows and require promotion (not sync) to be the gated step. The cluster should always converge on Git — the control point is what gets into Git.
Kargo on AWS: warehouses, stages, and IRSA
Kargo's Warehouse polls ECR for new images. On EKS, don't feed it credentials — bind its controller service account to an IAM role via IRSA (or EKS Pod Identity) with read-only ECR permissions:
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
name: frontend
namespace: kargo-platform
spec:
subscriptions:
- image:
repoURL: <ACCT>.dkr.ecr.eu-west-1.amazonaws.com/frontend
semverConstraint: ">=1.0.0"
strictSemvers: true
Each environment is a Stage. Dev subscribes directly to the warehouse; staging subscribes to dev's verified freight; prod subscribes to staging's:
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
name: staging
namespace: kargo-platform
spec:
requestedFreight:
- origin:
kind: Warehouse
name: frontend
sources:
stages: [dev] # Only freight verified in dev is eligible
promotionTemplate:
spec:
steps:
- uses: git-clone
- uses: yaml-update
config:
path: envs/staging/frontend/values.yaml
updates:
- key: image.tag
value: ${{ imageFrom("...frontend").Tag }}
- uses: git-commit
- uses: git-push
- uses: argocd-update
That sources.stages: [dev] line is the promotion graph. An image physically cannot reach staging without having passed dev, and cannot reach prod without passing staging. The pipeline topology is declarative and enforced — not a convention someone can forget under deadline pressure.
Verification: the part that makes promotion trustworthy
Automated promotion without automated verification is just automated blast radius. This is where most write-ups wave their hands, and where the real engineering lives.
Kargo integrates with Argo Rollouts' AnalysisTemplates, which means each stage can run metric-backed verification against your observability stack before its freight becomes eligible for the next stage. We run the LGTM stack, so verification queries Prometheus directly:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: frontend-health
spec:
metrics:
- name: error-rate
interval: 1m
count: 10
failureLimit: 1
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{app="frontend",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{app="frontend"}[5m]))
successCondition: result[0] < 0.01
- name: p99-latency
interval: 1m
count: 10
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{app="frontend"}[5m])) by (le))
successCondition: result[0] < 0.5
A release now soaks in dev, gets measured against real SLO-shaped queries for ten minutes, and only then becomes promotable. The same pattern in staging, ideally alongside synthetic traffic or smoke tests, gives you two independent metric-verified gates before production.
For the prod stage itself, pair the promotion with Argo Rollouts canary steps — shift 10% of traffic behind the ALB, re-run the same analysis against the canary subset, then progress. Verification and progressive delivery share templates, which keeps your definition of "healthy" in exactly one place.
The final gate — staging to prod — stays manual by policy in our setup: a human clicks approve in Kargo's UI (or via the CLI, from Slack). Not because the automation isn't trusted, but because production timing is a business decision. During a regional sales event on our ticketing platform, "verified and ready" and "deploy right now" are very different statements.
What the full flow looks like
- Engineer merges a PR in the service repo. GitHub Actions lints, tests, builds, signs, generates the SBOM, and pushes
v1.4.2to ECR via OIDC. - Kargo's warehouse discovers the digest and creates Freight.
- Freight auto-promotes to dev: Kargo commits the tag bump to
envs/dev/frontend/values.yaml, Argo CD syncs the dev cluster. - The dev AnalysisRun watches error rate and latency for ten minutes. Pass → freight is verified.
- Freight auto-promotes to staging; same commit-sync-verify cycle, plus smoke tests.
- A human approves prod. Kargo commits to
envs/prod/frontend/values.yaml; Argo CD executes a canary rollout with in-flight analysis; full traffic shift on success. - Every state transition exists as a commit, a Kargo promotion record, and an Argo CD sync — three independent audit trails that agree with each other.
Mean time from merge to verified-in-staging: about twenty-five minutes, with zero human involvement. Mean human effort per production release: one approval click.
Lessons from running this for real
Promote digests, pin everything. Tags are for humans; digests are for machines. Immutable ECR tags plus digest-based promotion eliminates an entire category of "but it worked in staging" incidents.
Verification queries are product code. Treat AnalysisTemplates with the same review rigor as application code. A query with a subtly wrong label selector is a green light that means nothing.
Keep the emergency path inside GitOps. When you need to ship a hotfix at 3 a.m., the answer is Kargo's manual promotion of a specific freight — not kubectl edit. If your break-glass procedure bypasses Git, your audit trail is fiction precisely when you'll need it most.
Budget for ECR lifecycle policies early. A warehouse polling a repository with ten thousand untagged images is slow and expensive. Expire untagged images aggressively; keep the last N releases per service.
Don't over-gate. Every manual approval you add is a place where releases queue and context evaporates. Gate production. Let metrics gate everything else.
The tooling here — Actions, Argo CD, Helm, Kargo — is the same stack you'll find in a weekend tutorial. What separates a demo from a platform is everything around it: account-level isolation, federated identity end to end, generated rather than hand-written manifests, and promotion gates that measure reality instead of assuming it.
Git as the source of truth is the principle. Verified, auditable, boring promotions are the payoff.
How is your team handling environment promotion today — Git commits, CI scripts, or still a human with kubectl? I'd genuinely like to hear what's working (and what isn't) in the comments.
