Your Image Tag Says ‘latest’. Your Rollback Says Good Luck.

Over on r/devops, a thread titled “How does your organisation handle CI/CD and image tagging for microservices?” asked a simple question — how do you tag your container images across a fleet of services? It was crossposted to r/kubernetes, and the answers split into two camps: teams with a scheme, and teams with a habit. The habit, almost every time, is one word: latest .

Picture the scene — illustrative, but you have lived a version of it. A deploy goes out at night. Something is wrong, nobody knows what yet, and the call is the sensible one: roll back. Somebody re-applies the manifest from before the deploy. The manifest says image: api:latest. The pods come back green. Everyone exhales.

Except the rollback rolled nothing back. Between the good deploy and the bad one, another build pushed to latest. The tag moved. The cluster pulled whatever latest pointed at that minute — the broken build, or a third build nobody had tested. The team did the right thing, quickly, and production still got a surprise. That is not a people problem. That is a naming problem.

A tag is a nickname, not a name

Start from the floor: this whole mess is two definitions wearing one coat.

A container image (the packaged app — code, runtime, and files bundled into one runnable unit) lives in a registry (the store that holds images and hands them out when a cluster asks). When you push an image, you give it a tag: a short human-readable label after the colon, like api:1.4.2 or api:latest.

Here is the part nobody reads on the box: a tag is just a pointer. By default, tags are mutable (changeable). Push a new image with the same tag and the tag quietly leaves the old image and points at the new one. The old image is still in the registry, but it has lost its label. Kubernetes documentation says it plainly: tags can be moved to point to different images, but digests are fixed.

The digest is the image’s other identity. It is a SHA-256 hash (a fingerprint computed from the content itself — change even one byte and you get a completely different fingerprint) written as sha256: followed by 64 characters. You reference it with an @: api@sha256:.... A digest does not point at the content. In a real sense, it is the content’s name, because it is calculated from it.

And latest? latest is not special software magic. It is simply the tag you get when you do not specify any tag at all. Push or pull an image with no tag and everyone — Docker, Kubernetes — assumes you mean latest. It carries no promise about being newest, best, or tested. It only means “whatever this tag pointed at when you asked.”

One more uncomfortable truth: a tidy tag does not save you. api:1.4.2 looks like a version and reads like a promise, but unless the registry locks it, that tag is exactly as movable as latest. A nickname that looks formal is still a nickname.

A tag tells you what someone called it. A digest tells you what it actually is.

Why ‘latest’ breaks rollbacks first

A rollback (returning production to the previous, known-good version) has one job: run the exact code that ran before. Not similar code. Not code with the same label. The exact code.

With a moving tag, you cannot even name that code. “Roll back to latest“ says nothing, because latest is a question, not an answer — and the registry will answer it fresh every time you ask. If a build lands mid-incident, your rollback target moves while you are rolling back. It is like giving someone directions to “the blue house” on a street where a painter is working his way down the block.

Kubernetes makes the trap spring faster. The imagePullPolicy setting (the rule that decides when the kubelet — the agent on each node that runs pods — downloads an image) is defaulted for you when you leave it out. Leave it out with :latest, or with no tag at all, and the default is Always: every new pod re-asks the registry what the tag means right now. Leave it out with a specific tag or a digest, and the default is IfNotPresent: pull only if the node does not already have it. So a latest deployment does not even fail consistently — old pods keep the old image, new pods get the new one, and for a while your fleet is running a mixture that exists in no manifest anywhere. Kubernetes’ own guidance is blunt: avoid :latest in production, because it is harder to track which version is running and harder to roll back properly.

Then comes the audit question: what code was running at 2 PM? With digests recorded, that is a lookup. With latest, it is archaeology — push logs, pipeline timelines, and hope. Auditors are not famous for accepting hope as evidence.

Tag like you mean it: SHA plus semver

So what should CI/CD (continuous integration and continuous delivery — the automated pipeline that builds, tests, and ships every change) stamp on each image? Two tags, doing two different jobs.

First, every build gets the full Git SHA as a tag. The Git SHA (the fingerprint Git gives each commit — each saved snapshot of the source code) is unique to that exact source. Tag api:a91f... (use the full SHA in real life, not the short form) and anyone can walk from a running image back to the precise code that produced it. No two different builds ever share it, because no two different commits share it. This is the tag for machines and for tracing.

Second, a release also gets a semver tag, for humans. Semver (semantic versioning) is the three-number scheme MAJOR.MINOR.PATCH: raise MAJOR for changes that break compatibility, MINOR for features that break nothing, PATCH for backward-compatible bug fixes. The spec also carries a rule that fits this whole post: once a version is released, its contents must not be modified — changes ship as a new version. v1.4.2 tells a human what kind of change this is at a glance. A real job — just not the job of proving identity.

Here is the shape of it in a pipeline. Read the comments — the last step is the one most pipelines skip, and it is the one that matters most:

IMAGE=registry.example.com/shop/api
SHA=$(git rev-parse HEAD)
VERSION=1.4.2

docker build -t $IMAGE:$SHA -t $IMAGE:v$VERSION .
docker push $IMAGE:$SHA
docker push $IMAGE:v$VERSION

# Capture the digest - this is the identity you deploy
docker inspect --format='{{index .RepoDigests 0}}' $IMAGE:$SHA
# Illustrative output:
# registry.example.com/shop/api@sha256:1a2b3c...(full 64-character hash in real output)

Two tags go up: the SHA tag, unique forever, and the release tag, readable by humans. Then the pipeline captures the digest from the pushed image and records it with the build — in the build output, the release notes, wherever your team actually looks. That recorded digest is now the deployable fact. Everything after this point should use it.

Deploy the digest, not the nickname

Now the deploy side. A Deployment (the Kubernetes object that keeps a set of identical pods running and replaces them when the spec changes) should carry the image in its full form: name, tag, and digest, joined like this: name:tag@sha256:digest.

Why keep the tag in the string at all? For the human reading the manifest at 2 AM — the tag says “release 1.4.2.” But when tag and digest are both present, Kubernetes pulls only by the digest. The tag is a comment living inside the address. Humans read the nickname; the machine obeys the fingerprint.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: shop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/shop/api:1.4.2@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Notice imagePullPolicy: IfNotPresent sitting happily next to the digest pin. That pairing makes sense precisely because a digest never changes: if the node already holds that exact fingerprint, there is nothing newer to fetch — newer does not exist. Compare that with Always plus latest, where every pod start is a fresh roll of the dice.

And rollback? Rollback becomes the most boring operation in your runbook: take the previous recorded digest, put it back in the manifest, apply. No re-pushing old images. No re-tagging anything to make yesterday’s build wear today’s label. No guessing. Boring is not a side effect here. Boring is the feature.

Lock the registry door — and sign it, if you can

Everything above is discipline. Discipline is good, but discipline that a tired human can accidentally break at release time needs a lock behind it. Registries sell exactly that lock: immutable tags (tags the registry itself refuses to move once pushed).

On Amazon ECR (Amazon’s container registry), the tag setting defaults to MUTABLE — overwrites allowed. Set ImageTagMutability to IMMUTABLE and a second push under an existing tag is refused, with an ImageTagAlreadyExistsException. The mistake stops being silent and becomes a loud, cheap failure in the pipeline, where failures belong. On Google Artifact Registry (Google Cloud’s registry), immutable tags mean you cannot delete a tagged image, remove a tag, move a tag to another digest, or push a tag another version already uses. Docker Hub offers the same idea — all tags immutable, or only specific ones.

Two honest warnings before you flip the switch on everything you own. First, flip it repository by repository, and tell the team before you do — the first person to learn about it should not learn from a failed release push. If your flow genuinely needs one moving label in development, keep that in a development repository and keep production locked. Second —

Staging warning: test the whole flow in staging first. Turn immutability on in a staging repository, push the same tag twice on purpose, and watch the second push fail. Then practise a digest rollback there. A lock you have never tested is just a new way to be surprised, and production charges full price for surprises that staging would have given you free.

If you can go one step further, sign the image. Signing (attaching a cryptographic signature — a tamper-evident seal proving who produced an artifact and that it has not changed since) is what cosign, from the Sigstore project, does for container images. The rule that matters: always sign the digest, never just the tag — sign a tag and you might seal something you did not intend, because the tag can move after you sign it. Verification then checks that the signature’s digest matches the image in front of it. Signing does not fix tagging. It sits on top of digest pinning and answers the next question: not just “is this the exact image?” but “is this the exact image we built?”

# Sign the digest, never just the tag
cosign sign --key cosign.key registry.example.com/shop/api@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD

# Verify before anything deploys it
cosign verify --key cosign.pub registry.example.com/shop/api@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD

The tagging checklist

Pin this where your pipeline lives. Work it top to bottom for one service first — one service done properly teaches the whole fleet the pattern.

1. Never deploy latest : Not in production, not in staging — latest, or no tag at all, is for a quick test on your own laptop and nothing else.

2. Tag every CI build with the full Git SHA: Unique, automatic, and traceable back to the exact source that produced the image.

3. Add a semver tag on releases, for humans: v1.4.2 tells people what kind of change shipped — MAJOR breaks, MINOR adds, PATCH fixes.

4. Record the digest from every build: Capture it in the build output and keep it with the release record. No record, no rollback.

5. Deploy by digest: The manifest carries name:tag@sha256:digest — humans read the tag, Kubernetes pulls the digest.

6. Roll back by restoring the previous digest: Never by re-pushing or re-tagging an old image to make it look current.

7. Turn on immutable tags in the registry: ECR IMMUTABLE, Artifact Registry immutable tags, or Docker Hub’s setting — repository by repository.

8. Test in staging first: Push the same tag twice and watch it fail; practise the digest rollback before production needs it.

9. Sign the digest if your pipeline can: cosign sign the digest, verify before deploy — proof of what it is and whose it is.

10. Keep latest in its place: If you keep it at all, it is a convenience label in development only — never a deploy reference, never a rollback plan.

Make rollback boring again

None of this is exotic: a SHA tag, a semver tag, a recorded digest, a manifest that deploys the digest, a registry that refuses to move tags, a signature if you can. An afternoon of pipeline work — mostly deleting habits, not adding machinery.

What you get back is what latest quietly stole: certainty. What is running? That digest. What ran yesterday? The previous digest. Roll it back — done, because the previous digest was written down before anyone needed it.

Let latest live in development, where surprises are cheap tuition. In production, let the manifest say the fingerprint. Nicknames are for friends. Rollbacks need names.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Kubernetes, Docker, CI/CD, DevOps, SRE, and platform engineering — explained the practical way.

Quick check before you close this tab: what do your production manifests reference right now — latest, a tag, or a digest? Tell me in the comments, and if it is latest, tell me what your rollback plan calls it.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.