Two-step control plane upgrades in GKE: How minor version rollbacks work under the hood

Kubernetes control plane minor version upgrades have historically been an all-or-nothing proposition. In standard Kubernetes clusters, upgrading a control plane from one minor version to the next—such as moving from 1.33 to 1.34—commits changes to storage schemas immediately. If an unexpected regression emerges after upgrading the API server, rolling back to the previous minor version was impossible without restoring etcd snapshots.

To eliminate this operational risk, the GKE team drove upstream contributions in Kubernetes Enhancement Proposal KEP-4330 (Compatibility Versions) and introduced two-step control plane upgrades. Following public preview validation with enterprise customers, this capability is now Generally Available (GA) across all GKE release channels.

In this article, I will explain how two-step upgrades work under the hood, how automated rollouts leverage canary analysis, and how to manage rollback-safe upgrades using the Google Cloud CLI and Terraform.

The problem with minor version upgrades

In Kubernetes, minor version releases introduce storage schema changes, deprecated API removals, and modified controller behaviors. When the kube-apiserver binary starts on a newer minor version, it writes resources using newer internal schemas.

Because earlier binary versions cannot parse data stored in newer schemas, Kubernetes prohibits control plane downgrades across minor versions. If an organization encountered an issue after upgrading, platform operators had to either run the degraded control plane or rebuild the cluster.

Two-step upgrades decouple binary execution from API capability enablement. By separating the upgrade into two distinct phases, GKE provides an observation period (also known as a soak window) during which operators or automated systems can monitor cluster behavior and roll back the control plane to the previous minor version with zero data loss.

Decoupling binary execution from emulated versions

The foundation of two-step upgrades is running a newer control plane binary in an emulated compatibility mode.

When a two-step upgrade begins, GKE advances the control plane through two sequential stages:

• Step 1: Binary upgrade (emulated mode): GKE upgrades the control plane binary to the target minor version (e.g., 1.34), but configures the API server to emulate the previous minor version (1.33). In this state, the control plane executes the new binary logic, while API schemas match the older version. APIs removed in 1.34 remain accessible. During this soak period, you can safely roll back to 1.33.
• Step 2: Emulated version upgrade (finalization): Once the soak window completes without incident, GKE updates the emulated version to match the binary version. This step permanently enables the new minor version API schemas and feature deprecations. After this point, rollback is no longer possible.

During the soak period, worker node pools cannot be upgraded beyond the emulated version to preserve Kubernetes version skew rules.

Automated orchestration with CPRS and Canary Analysis Service

For clusters configured for auto-upgrades, two-step upgrades are enabled out of the box with zero manual configuration required. The GKE rollout engine, known as Control Plane Rollout Service (CPRS), orchestrates the staged lifecycle natively:

• CPRS upgrades the control plane binary while locking the emulated version to the previous minor release.
• CPRS initiates an automated 24-hour soak window.
• The Canary Analysis Service (CAS) monitors cluster health signals throughout the soak duration, including API latency, error rates, and Pod health metrics.
• If CAS detects unexpected regressions, the rollout halts to allow automated or user-driven rollbacks.
• Once the cluster passes all CAS evaluations and satisfies the soak timer, CPRS triggers step 2 to finalize the emulated version.

This validation framework has helped GKE control plane upgrades achieve a 99.999% (five nines) rolling 30-day success rate across the global fleet.

Initiating manual two-step upgrades

If your team manages upgrades manually, you can execute two-step upgrades using the Google Cloud CLI or Terraform.

To initiate a two-step upgrade with a custom soak duration using gcloud:

gcloud beta container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.34.1-gke.1829001 \
    --control-plane-soak-duration=48h \
    --master

The --control-plane-soak-duration flag defines the rollback-safe window, supporting values from 6 hours up to 7 days (e.g., 48h or 2d).

For infrastructure-as-code workflows, Terraform includes official support for managing two-step control plane upgrades declaratively by specifying the target version and soak parameters in your GKE cluster resources.

Verifying upgrade status and executing a rollback

While the cluster is soaking in emulated mode, you can inspect the active rollback state using the CLI:

gcloud container clusters describe my-cluster \
    --location=us-central1 \
    --format="yaml(rollbackSafeUpgradeStatus)"

The output includes a rollbackSafeUpgradeStatus block with the target binary version, emulated version, remaining soak time, and previousVersion string.

If your monitoring tools uncover a regression during the soak window, you can roll back the control plane to the previous minor patch:

gcloud container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.33.5-gke.1080000 \
    --master

Because the control plane ran in emulated mode, no new data formats were written to etcd. GKE downgrades the control plane binary back to the specified version without risking data corruption.

Completing the upgrade early

If your validation tests pass and you want to unlock new minor version features immediately without waiting for the soak duration to expire, you can finalize the upgrade manually:

gcloud beta container clusters clusters complete-control-plane-upgrade my-cluster \
    --location=us-central1

Once executed, GKE promotes the emulated version to match the binary version, completing the upgrade.

Operational requirements and limits

When planning two-step control plane upgrades, keep the following operational rules in mind:

• Two-step upgrades apply to minor version upgrades to GKE 1.33 and later.
• Control plane upgrades must proceed one minor version at a time.
• Maintenance windows and exclusions are strictly respected.
• Autopilot and regional Standard clusters maintain continuous control plane availability during both phases.

Next steps

Two-step control plane upgrades eliminate the risk of irreversible minor version updates in Kubernetes, providing platform engineers with automated safety and an emergency rollback mechanism.

To learn more about configuring two-step upgrades, read the official GKE cluster upgrade documentation and explore KEP-4330: Compatibility Versions.