# How Immutable OS Pipelines Are Replacing the 2 AM Kubernetes Upgrade - Link: https://thestackobserver.com/how-immutable-os-pipelines-are-replacing-the-2-am-kubernetes-upgrade/ - Published: 2026-08-17T09:51:49-04:00 - Author: Stackxx Upgrading a Kubernetes control plane used to mean staying awake for it. SSH into every node, run the upgrade by hand, watch etcd health during reboots, and hope quorum holds. In 2026, with the pace of CVEs accelerating across the container stack, that manual approach is no longer just tedious — it is a liability. A recent build by Olivier Calzi, a CNCF Golden Kubestronaut, demonstrates what the alternative looks like: a fully automated, self-healing pipeline that upgrades an entire three-node Kubernetes control plane in **11 minutes** with **zero human intervention**. The pipeline uses Kairos, an immutable Linux distribution, combined with Renovate, Kyverno, ArgoCD, and cosign-signed images. The result is not a proof-of-concept. It is a production-grade pattern that any platform team can adapt. ## The Problem: Patching Velocity vs. Manual Friction Kubernetes itself does not ship with an automated node operating system upgrade mechanism. When the OS underneath the cluster needs a kernel patch, a CVE fix, or a base image refresh, the operational burden falls on the platform team. That burden compounds quickly. In June 2026, the Kubernetes Security Response Committee updated records for four older CVEs that remain unfixed in upstream code because fixing them would break existing functionality. Datadog’s security team noted that vulnerability scanners began surfacing these findings in clusters where they were not previously detected. The takeaway is stark: **affected versions do not necessarily mean exposure**, but they do mean investigation, verification, and often, patching. For platform teams managing dozens or hundreds of clusters, the math is unforgiving. Each node that requires manual intervention adds latency, risk, and opportunity for human error. When a critical CVE drops on a Friday afternoon, the difference between a pipeline that runs itself and one that requires an engineer to babysit reboot sequences is the difference between a secure cluster and a weekend incident. ## The Architecture: Six Tools, One Pipeline Calzi’s pipeline is built on Kairos Hadron, an immutable Linux distribution designed specifically for Kubernetes workloads. Kairos uses an **A/B partition scheme**: the active partition runs the current OS image, while the inactive partition receives the new image. A reboot swaps the two. If something goes wrong, rolling back is as simple as rebooting back into the old partition. There is no in-place patching, no drift, and no accumulation of configuration cruft. The pipeline itself stitches together six tools, each with a single responsibility: * **Gitea** hosts every manifest, policy, and upgrade specification under version control, running on its own Kairos-backed Kubernetes cluster. * **Renovate** monitors the upstream Kairos image registry on Quay. When a new release appears, Renovate opens a pull request bumping the image tag and the custom resource name in the upgrade specification. * **Kyverno** acts as an admission gate, rejecting any upgrade custom resource that does not match the expected image path `quay.io/kairos/hadron:*`. This blocks misconfigurations, typosquatting attempts, and unauthorized image references before they ever reach a node. * **Cosign** verifies the image signature against the upstream GitHub Actions OIDC identity. This is not just checking the tag — it is cryptographically confirming that the actual CI pipeline produced the artifact. * **ArgoCD** detects the merged pull request as drift and applies the new manifest. No engineer runs `kubectl apply`. * **kairos-operator** watches for `NodeOpUpgrade` custom resources, cordons one node at a time, pulls the image, writes the new A/B slot, triggers a reboot, waits for the node to rejoin the cluster, and then proceeds to the next node. ## The Run: What 11 Minutes Actually Looks Like On July 7, 2026, the pipeline executed a real upgrade from Hadron v0.3.0 to v0.4.0. The timeline tells the story: * **11:40:20** — Baseline established: three nodes Ready, all running kernel 7.0.10- hadron. * **11:42:00** — Renovate’s pull request is merged. The diff is two lines: the image tag and the metadata name of the upgrade custom resource. * **11:42:30** — ArgoCD detects the cluster is OutOfSync. * **11:42:37** — ArgoCD syncs the change, deleting the old custom resource and creating the new one. * **11:43:09** — The image pulls in 25.5 seconds (324 MB). * **11:43:33** — The upgrade job completes: the A/B slot is written, and the reboot is triggered. * **~11:47** — Node 0 rejoins the cluster with kernel 7.1.0-hadron. * **~11:49** — Node 1 completes the same cycle. * **11:51:36** — Node 2 finishes. All three nodes are running the new kernel. Total wall-clock time: 11 minutes, 16 seconds. etcd quorum was never broken. Workloads were not disrupted. No human touched a terminal after the pull request merged. ## The Bug That Almost Broke It The pipeline was not flawless on the first real run. Calzi discovered that Renovate’s custom regex manager used a field called `extractVersionTemplate`, which does not exist in Renovate’s custom manager schema. It silently did nothing. The result: Renovate correctly bumped the image tag but left the custom resource name unchanged. Here is why that matters. The `NodeOpUpgrade` custom resource is one-shot. The kairos- operator marks it complete and never reprocesses it. Patching the `spec.image` field on an existing resource does nothing because the operator considers that object finished. The name change is what forces ArgoCD to delete the old resource and create a genuinely new one, which is the actual trigger for the upgrade. The fix was switching to `currentValueTemplate`, which correctly converts the dash- formatted version in the resource name into comparable SemVer, then reconstructs the dash format for the new tag. This experience underscores a broader lesson: ** automation that looks correct but is silently half-executed is more dangerous than no automation at all**. The human review step in this pipeline exists not to execute the upgrade, but to catch the places where automation quietly does the wrong thing while appearing to succeed. ## Why This Pattern Matters Beyond One Cluster The Kairos pipeline is not about one team’s home lab. It is a template for how platform engineering should think about node lifecycle management in 2026. First, **immutability is a security primitive**. An OS image that is never patched in place, only atomically replaced and verified, eliminates an entire class of drift and tampering risks. The supply chain is short and auditable: the image is signed by the CI pipeline that built it, verified by cosign before deployment, and gated by Kyverno at admission. Second, **sequenced rolling upgrades with quorum preservation** are non-negotiable for control plane nodes. Calzi’s earlier iteration used `concurrency: 0` in the upgrade spec, which he assumed meant “one node at a time.” It actually meant “all nodes simultaneously.” Three control plane nodes rebooted at once during a test run. etcd survived — by luck, not design. Fixing this to `concurrency: 1` is the kind of operational detail that separates a working prototype from a production- ready system. Third, **CVE response time is now a competitive advantage**. When unfixed Kubernetes CVEs remain open for years because upstream fixes would break compatibility, the operational response shifts from “wait for the patch” to “verify your exposure, apply preventive controls, and patch everything you can.” A pipeline that turns a two-line diff into a fully executed, verified, rollback-capable node upgrade in 11 minutes is the difference between a security posture that scales and one that collapses under its own weight. ## What Platform Teams Should Do Next The Kairos pipeline is not the only way to solve this problem, but it is one of the most complete public implementations to date. For teams looking to reduce manual node patching, the actionable takeaways are: * **Adopt immutable OS images** where possible. Whether Kairos, Flatcar Container Linux, or another distribution, the A/B partition model removes the risk of in- place upgrade failures. * **Automate the entire upgrade path** from image detection to node reboot. The fewer human steps, the fewer opportunities for timing errors, skipped validations, or weekend page-outs. * **Gate every upgrade with admission controls and signature verification**. Kyverno and cosign (or equivalent policy engines and signing tools) should block any upgrade that does not pass both checks. * **Validate concurrency and sequencing assumptions** in your upgrade executor. Never assume defaults are safe. Test quorum behavior under failure conditions. * **Build dry-run capability** before production execution. Calzi’s next step is a CI stage that runs `kairos-agent upgrade --recovery` against the new image before the pull request merges, catching failures before they touch a live node. The broader shift is clear. Kubernetes platform engineering is moving from “we upgrade nodes manually a few times a year” to “our nodes upgrade themselves within minutes of a signed image being published.” The teams that make that transition will be the ones who survive the next wave of CVE disclosures without a sleepless weekend. The teams that do not will find themselves staring at etcd health dashboards at 2 AM, hoping quorum holds — just like the old days. ## Sources * [CNCF Blog — “Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos”](https://www.cncf.io/blog/2026/08/14/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos/)( August 14, 2026) * [Datadog Blog — “Understanding unfixed Kubernetes CVEs: What you can and can’t detect”](https://www.datadoghq.com/blog/how-to-manage-unfixed-kubernetes-cves/)( August 10, 2026) * [OpenTelemetry Blog — “What can you do with OpenTelemetry entity events?”](https://opentelemetry.io/blog/2026/consuming-opentelemetry-entity-events/)( August 14, 2026) * [Prometheus v3.14.0-rc.0 Release Notes](https://github.com/prometheus/prometheus/releases/tag/v3.14.0-rc.0)( August 6, 2026) * [Fluent Bit 5.1.1 Release Notes](https://github.com/fluent/fluent-bit/releases/tag/v5.1.1)( August 15, 2026) * [Grafana Labs Blog — “Reflections on AI Week, and the future of solving problems with observability and AI”](https://grafana.com/blog/ai-week-recap/)( July 31, 2026)