High availability in Kubernetes often fails in the same place: the uncomfortable gap between “the scheduler wants to do the right thing” and “the infrastructure keeps changing underneath it.” Multi-AZ architectures help, but they don’t automatically address gray failures—intermittent packet loss, increasing latency, or a single Availability Zone (AZ) that’s just off enough to cause cascading timeouts.
AWS recently detailed an approach that tightens that gap for Amazon EKS: integrate Amazon Application Recovery Controller (ARC) zonal shifts with Karpenter via a purpose-built Kubernetes controller. The goal is simple: when ARC declares an AZ impaired and shifts traffic away, Karpenter should stop trying to provision capacity in the impaired zone and (critically) help the cluster converge on healthy topology quickly.
This is more than “yet another controller.” It’s an example of a pattern platform teams will need more of in 2026: control-plane signals (ARC) translated into cluster-level scheduling inputs (Karpenter node pool requirements) with clear rollback semantics.
Why ARC + Kubernetes isn’t enough when Karpenter is involved
Kubernetes already has strong primitives for resilience:
- Pod topology spread constraints and anti-affinity to distribute replicas across zones/nodes
- PDBs to prevent aggressive eviction during rollouts
- Readiness/liveness probes to keep traffic away from unhealthy pods
ARC’s zonal shift adds a higher-level “get out of the bad zone” lever by cordoning nodes and removing endpoints from routing in an impacted AZ. That’s great if your capacity already exists elsewhere. But when your cluster depends on Karpenter for elastic node provisioning, there’s a missing feedback loop: Karpenter doesn’t automatically learn that a specific zone should be avoided right now.
Without that signal, you can hit a failure mode like this:
- ARC detects an AZ problem and initiates a zonal shift.
- Workloads evict or stop receiving traffic in the impacted AZ.
- Demand rises elsewhere; pods go Pending due to insufficient capacity.
- Karpenter tries to satisfy the demand across all zones defined for the node pool—including the impaired one—wasting time and sometimes capacity on the wrong fault domain.
The integration pattern: rewrite node pool topology based on ARC events
The controller AWS describes listens for events emitted when a zonal autoshift is triggered (or when a manual zonal shift is initiated). Each event includes the impaired zone ID. The controller then “translates” that event into Karpenter inputs by updating node pool configuration so Karpenter won’t provision in the impaired zone.
The clever part is how it does this while keeping rollback simple:
- If a node pool explicitly includes the impaired AZ in its
topology.kubernetes.io/zonerequirement, the controller removes that AZ from the requirement. - It stores the removed zone in an annotation (AWS uses
zonal-autoshift.eks.amazonaws.com/away-zones) so it can restore the original requirement later. - If a node pool doesn’t specify zones, the controller can infer the available zones (for example from subnet tags) and then apply an “all healthy zones except the impaired one” rule.
Operationally, that means Karpenter’s provisioning decisions start aligning with ARC’s traffic-shift decisions, reducing the time to stabilize.
What platform teams should validate before adopting
Before you deploy anything that mutates node pool requirements, treat it like a production control loop. In particular:
1) Your workload topology assumptions
If your workloads are not zone-spread (or if your PDBs are too strict), “avoid the impaired zone” can turn into “can’t reschedule anywhere.” Validate that:
- critical Deployments are spread across zones,
- StatefulSets can tolerate zone movement (or have clear failover), and
- PDBs allow enough disruption during a zonal shift.
2) Karpenter node pool boundaries
If you have a single node pool powering multiple tiers (stateless, stateful, GPU), a topology rewrite could have surprising blast radius. Many teams will want to:
- split node pools by workload class,
- use explicit zone requirements so behavior is predictable,
- and maintain “minimum capacity” in each healthy AZ for critical services.
3) Rollback behavior and event ordering
Zonal issues can be transient. The integration needs a clean reversion path when ARC ends a shift. Confirm:
- how quickly annotations are reverted,
- what happens if multiple shifts overlap, and
- whether node pools converge back to the pre-shift topology deterministically.
A practical runbook: adopting ARC↔Karpenter safely
- Start in a staging cluster and simulate impaired-AZ behavior (or use manual zonal shift if your environment supports it).
- Observe Karpenter provisioning latency and Pending pod durations with and without the controller.
- Audit node pool mutations: log every change (old requirement → new requirement) and store it centrally.
- Add guardrails: restrict which node pools the controller is allowed to mutate (label/annotation allowlist).
- Document a manual override procedure if you need to temporarily “pin” a node pool to a reduced set of zones during a major incident.
Why this matters beyond EKS
The bigger takeaway is architectural: resilient platforms increasingly depend on bridging signals across layers. ARC is an infrastructure control plane. Karpenter is a cluster provisioning control plane. The integration is a “translation layer” that keeps their decisions consistent.
Expect more of this pattern as clusters run more mixed workloads (including GPUs) and as “agentic” systems start calling real tools that must remain available. When topology and traffic policy are managed by different controllers, you either create explicit glue—or you accept downtime during the exact moments you can least afford it.
Leave a Reply