The Rollback Worked. My Next Deploy Could Break It Again.

My application was working again. That should have been the end of the problem.

It was not.

In [the first article in this series(https://blog.lalitbagga.com/posts/why-i-chose-k3s-debian-homelab), I built an always on Kubernetes lab on my home server. I deployed a small application, changed it from Version 2 to Version 3, and then practised a rollback.

Kubernetes restored Version 2 successfully. But the YAML file on my computer the file containing the instructions for the application still said Version 3.

I had repaired what was running, but I had not repaired the instructions that I would use next time. If I applied that file again, I could accidentally bring Version 3 back.

I corrected the YAML manually and confirmed that the file and cluster agreed. That solved the immediate problem, but it also exposed a weakness in my process: recovery depended on me remembering to repair two places.

That became the starting point for this project.

The Real Problem Was Two Different Answers

Kubernetes always has a live answer to the question, “What should run now?” My
YAML file had a saved answer to the same question.

After the rollback, those answers were different:

Running in Kubernetes: Version 2
Saved in YAML:          Version 3

Neither answer was hidden. The weakness was that nothing continuously checked
whether they agreed.

I wanted one durable place to record the version I had chosen. If somebody changed the cluster directly, the system should notice the difference and restore the recorded choice. If the recorded choice itself was bad, recovery should mean changing that record not making another temporary cluster edit.

That is the problem GitOps is meant to address.

I Considered Who Should Be Allowed to Change the Cluster

My first idea was the most direct one: let the application pipeline deploy the new version.

A pipeline, usually called continuous integration or CI, is an automated worker. It runs tests and builds the application whenever code changes. I could also give it the password like Kubernetes credentials needed to change the cluster.

That would be convenient, but it would give the application build process a second, much more powerful job. A mistake or compromised workflow could move directly from building code to changing the cluster.

I wanted a smaller boundary:

CI may build and publish a release.
CI may propose which release should run.
CI may not change Kubernetes directly.

The proposal would become a pull request. I would review it and merge it into Git, the same version history system used for the application code. Git would therefore hold the durable answer to “Which version should this environment run?”

This was not about removing every credential from CI. It still needed limited GitHub permission to publish an image and open a pull request. The important restriction was precise: application CI would not receive a Kubernetes credential or permission to deploy to the cluster.

I Designed the Process Before Choosing the Tools

At this point, I could describe the process without naming a Kubernetes tool:

Developer changes the application
        ↓
CI tests it and builds a release
        ↓
CI opens a pull request with the exact release
        ↓
A human reviews and merges the change
        ↓
A worker inside Kubernetes reads Git
        ↓
That worker makes the cluster match Git

This operating model is called GitOps. Git stores the desired state the version we intend to run. Software inside the cluster repeatedly compares that record with reality and corrects differences.

I chose Argo CD for that inside-the-cluster worker. Argo CD watches the Git repository and reconciles the cluster. “Reconcile” simply means compare what is running with what should be running, then work to make them agree.

This separation gave each part one main responsibility:

• CI proves that the application can be built.
• Git records the reviewed version choice.
• Argo CD makes Kubernetes follow that choice.
• Kubernetes reports whether the application becomes healthy.

I Needed an Exact Version, Not a Moving Name

A release also needed an identity that could not silently change.

Container images often use names such as latest or version-3. These are tags: convenient labels that can be moved to different image contents later. That would weaken Git as the record of exactly what I approved.

I therefore chose an image digest. A digest is a fingerprint calculated from the image contents. If the contents change, the fingerprint changes. Git would store that exact fingerprint, so the selected release could not change while keeping the same name.

CI would publish the image, learn its digest, and open a pull request changing only that value. It would then stop and wait for review.

I Used Helm to Keep the Kubernetes Instructions Together

The application needed several Kubernetes instructions: how to run it, how to reach it, how to check its health, and which image digest to use.

I packaged those instructions with Helm. Helm is a way to create a reusable Kubernetes template. The structure stays the same while selected values such as the image digest or number of copies can change.

Before allowing Argo CD to use the Helm package, I planned to test it on its own: install it, verify the application, upgrade it, roll it back, remove it, and install it again in a clean location.

This separated two questions:

• Does the Kubernetes package work correctly?
• Does the automated delivery process use it correctly?

If both failed at the same time, troubleshooting would be much harder.

I Chose a Small Application That Could Fail Safely

A successful first deployment would not prove much. I needed an application that could help test both healthy and unhealthy releases without risking real data.

I built a small Node.js service that could answer four simple questions:

/health   Is the program alive?
/ready    Should Kubernetes send it user traffic?
/version  Which source version is running?
/metrics  How many requests has it handled?

The health and readiness checks were deliberately separate. This allowed me to create a safe bad release: the program would keep running, but its readiness check would fail, so Kubernetes would refuse to send new traffic to it.

The version response would help trace one change from source code, through the built image and Git, to the running application.

I Kept Argo CD Private

My home server already ran other services and had limited free memory. I did not need another public management screen.

I chose the smaller Argo CD Core installation. Its management components would remain reachable only inside the cluster, and I gave them explicit CPU and memory limits.

Argo CD still needed meaningful power inside Kubernetes to do its job. Keeping it private did not make it harmless. I therefore limited this application to its repository, its Kubernetes namespace, and the types of resources its Helm package created.

I Defined the Failure Tests Before Claiming Success

A green installation screen would only prove that the tools started. I wrote the tests before implementing the system:

The two failure cases were intentionally different.

Changing the cluster by hand would make the cluster wrong while Git remained correct. Argo CD should restore the version recorded in Git. This disagreement is called drift.

The readiness experiment would test the opposite problem: Git itself would contain a harmful setting. Argo CD should apply it faithfully, while Kubernetes should report that the new copy was not ready. Recovery would require reverting the Git change.

That distinction became the most important result of the project.

What the Design Did Not Promise

This design did not make Git automatically correct. A reviewer could still approve a bad value, and Argo CD could apply it perfectly. Argo CD also still needed permission to change the cluster.

The single server lab could not prove high availability. One successful request also could not prove that no user request was ever lost.

The goal was smaller and testable: keep direct cluster access out of application CI, record the exact selected release in Git, and observe how the system behaved when either the cluster or Git was wrong.

What Came Next

The rollback in my first article taught me that recovering the running application and correcting its saved instructions were separate jobs. This design joined them through one reviewable path.

Part 2
implements that path. It follows one release from CI to Git and into Kubernetes, then records what happened when I changed the cluster by hand, approved a bad readiness setting, and recovered by reverting Git.

The complete implementation and sanitized evidence are in the public
pull-based-kubernetes-delivery repository.