The Stack Observer
Cloud Native

How Immutable OS Pipelines Are Replacing the 2 AM Kubernetes Upgrade

•Stackxx•Cloud Native, Kubernetes

Upgrading a Kubernetes control plane used to mean staying awake for it. SSH into every node, run the upgrade by hand, watch etcd health during reboots, and hope quorum holds. In 2026, with the pace of CVEs accelerating across the container stack, that manual approach is no longer just tedious — it is a liability.

A recent build by Olivier Calzi, a CNCF Golden Kubestronaut, demonstrates what the alternative looks like: a fully automated, self-healing pipeline that upgrades an entire three-node Kubernetes control plane in 11 minutes with zero human intervention. The pipeline uses Kairos, an immutable Linux distribution, combined with Renovate, Kyverno, ArgoCD, and cosign-signed images. The result is not a proof-of-concept. It is a production-grade pattern that any platform team can adapt.

The Problem: Patching Velocity vs. Manual Friction

Kubernetes itself does not ship with an automated node operating system upgrade mechanism. When the OS underneath the cluster needs a kernel patch, a CVE fix, or a base image refresh, the operational burden falls on the platform team. That burden compounds quickly.

In June 2026, the Kubernetes Security Response Committee updated records for four older CVEs that remain unfixed in upstream code because fixing them would break existing functionality. Datadog’s security team noted that vulnerability scanners began surfacing these findings in clusters where they were not previously detected. The takeaway is stark: affected versions do not necessarily mean exposure, but they do mean investigation, verification, and often, patching.

For platform teams managing dozens or hundreds of clusters, the math is unforgiving. Each node that requires manual intervention adds latency, risk, and opportunity for human error. When a critical CVE drops on a Friday afternoon, the difference between a pipeline that runs itself and one that requires an engineer to babysit reboot sequences is the difference between a secure cluster and a weekend incident.

The Architecture: Six Tools, One Pipeline

Calzi’s pipeline is built on Kairos Hadron, an immutable Linux distribution designed specifically for Kubernetes workloads. Kairos uses an A/B partition scheme: the active partition runs the current OS image, while the inactive partition receives the new image. A reboot swaps the two. If something goes wrong, rolling back is as simple as rebooting back into the old partition. There is no in-place patching, no drift, and no accumulation of configuration cruft.

The pipeline itself stitches together six tools, each with a single responsibility:

  • Gitea hosts every manifest, policy, and upgrade specification under version control, running on its own Kairos-backed Kubernetes cluster.
  • Renovate monitors the upstream Kairos image registry on Quay. When a new release appears, Renovate opens a pull request bumping the image tag and the custom resource name in the upgrade specification.
  • Kyverno acts as an admission gate, rejecting any upgrade custom resource that does not match the expected image path quay.io/kairos/hadron:*. This blocks misconfigurations, typosquatting attempts, and unauthorized image references before they ever reach a node.
  • Cosign verifies the image signature against the upstream GitHub Actions OIDC identity. This is not just checking the tag — it is cryptographically confirming that the actual CI pipeline produced the artifact.
  • ArgoCD detects the merged pull request as drift and applies the new manifest. No engineer runs kubectl apply.
  • kairos-operator watches for NodeOpUpgrade custom resources, cordons one node at a time, pulls the image, writes the new A/B slot, triggers a reboot, waits for the node to rejoin the cluster, and then proceeds to the next node.

The Run: What 11 Minutes Actually Looks Like

On July 7, 2026, the pipeline executed a real upgrade from Hadron v0.3.0 to v0.4.0. The timeline tells the story:

  • 11:40:20 — Baseline established: three nodes Ready, all running kernel 7.0.10-hadron.
  • 11:42:00 — Renovate’s pull request is merged. The diff is two lines: the image tag and the metadata name of the upgrade custom resource.
  • 11:42:30 — ArgoCD detects the cluster is OutOfSync.
  • 11:42:37 — ArgoCD syncs the change, deleting the old custom resource and creating the new one.
  • 11:43:09 — The image pulls in 25.5 seconds (324 MB).
  • 11:43:33 — The upgrade job completes: the A/B slot is written, and the reboot is triggered.
  • ~11:47 — Node 0 rejoins the cluster with kernel 7.1.0-hadron.
  • ~11:49 — Node 1 completes the same cycle.
  • 11:51:36 — Node 2 finishes. All three nodes are running the new kernel.

Total wall-clock time: 11 minutes, 16 seconds. etcd quorum was never broken. Workloads were not disrupted. No human touched a terminal after the pull request merged.

The Bug That Almost Broke It

The pipeline was not flawless on the first real run. Calzi discovered that Renovate’s custom regex manager used a field called extractVersionTemplate, which does not exist in Renovate’s custom manager schema. It silently did nothing. The result: Renovate correctly bumped the image tag but left the custom resource name unchanged.

Here is why that matters. The NodeOpUpgrade custom resource is one-shot. The kairos-operator marks it complete and never reprocesses it. Patching the spec.image field on an existing resource does nothing because the operator considers that object finished. The name change is what forces ArgoCD to delete the old resource and create a genuinely new one, which is the actual trigger for the upgrade.

The fix was switching to currentValueTemplate, which correctly converts the dash-formatted version in the resource name into comparable SemVer, then reconstructs the dash format for the new tag. This experience underscores a broader lesson: automation that looks correct but is silently half-executed is more dangerous than no automation at all. The human review step in this pipeline exists not to execute the upgrade, but to catch the places where automation quietly does the wrong thing while appearing to succeed.

Why This Pattern Matters Beyond One Cluster

The Kairos pipeline is not about one team’s home lab. It is a template for how platform engineering should think about node lifecycle management in 2026.

First, immutability is a security primitive. An OS image that is never patched in place, only atomically replaced and verified, eliminates an entire class of drift and tampering risks. The supply chain is short and auditable: the image is signed by the CI pipeline that built it, verified by cosign before deployment, and gated by Kyverno at admission.

Second, sequenced rolling upgrades with quorum preservation are non-negotiable for control plane nodes. Calzi’s earlier iteration used concurrency: 0 in the upgrade spec, which he assumed meant “one node at a time.” It actually meant “all nodes simultaneously.” Three control plane nodes rebooted at once during a test run. etcd survived — by luck, not design. Fixing this to concurrency: 1 is the kind of operational detail that separates a working prototype from a production-ready system.

Third, CVE response time is now a competitive advantage. When unfixed Kubernetes CVEs remain open for years because upstream fixes would break compatibility, the operational response shifts from “wait for the patch” to “verify your exposure, apply preventive controls, and patch everything you can.” A pipeline that turns a two-line diff into a fully executed, verified, rollback-capable node upgrade in 11 minutes is the difference between a security posture that scales and one that collapses under its own weight.

What Platform Teams Should Do Next

The Kairos pipeline is not the only way to solve this problem, but it is one of the most complete public implementations to date. For teams looking to reduce manual node patching, the actionable takeaways are:

  • Adopt immutable OS images where possible. Whether Kairos, Flatcar Container Linux, or another distribution, the A/B partition model removes the risk of in-place upgrade failures.
  • Automate the entire upgrade path from image detection to node reboot. The fewer human steps, the fewer opportunities for timing errors, skipped validations, or weekend page-outs.
  • Gate every upgrade with admission controls and signature verification. Kyverno and cosign (or equivalent policy engines and signing tools) should block any upgrade that does not pass both checks.
  • Validate concurrency and sequencing assumptions in your upgrade executor. Never assume defaults are safe. Test quorum behavior under failure conditions.
  • Build dry-run capability before production execution. Calzi’s next step is a CI stage that runs kairos-agent upgrade --recovery against the new image before the pull request merges, catching failures before they touch a live node.

The broader shift is clear. Kubernetes platform engineering is moving from “we upgrade nodes manually a few times a year” to “our nodes upgrade themselves within minutes of a signed image being published.” The teams that make that transition will be the ones who survive the next wave of CVE disclosures without a sleepless weekend. The teams that do not will find themselves staring at etcd health dashboards at 2 AM, hoping quorum holds — just like the old days.

Sources