Kubernetes became the default control plane for cloud infrastructure by treating pods as independently schedulable units. Distributed AI training challenges that assumption. A training job may comprise dozens or hundreds of workers that must start together and communicate continuously, yet the default scheduler evaluates pods one at a time. A cluster can therefore appear busy while expensive GPUs wait for missing peers or exchange gradients across a slow network path.
A recent CNCF technical analysis puts two complementary mechanisms at the center of the remedy: gang scheduling, which coordinates when a group of pods can run, and topology-aware scheduling, which constrains where those pods land. The practical lesson is broader than a new scheduler feature. For AI infrastructure, allocation is not enough. The control plane must understand that a training job is one coordinated unit and that the distance between its accelerators directly affects useful throughput.
Why pod-at-a-time scheduling breaks down
The ordinary Kubernetes scheduling model works well when replicas are interchangeable and largely independent. A web service can make progress with nine replicas while the tenth is pending. A stateless worker can be restarted on another node without coordinating a simultaneous restart of every other worker.
Distributed training behaves differently. Workers repeatedly synchronize through collective communications such as all-reduce, gather, reduce, and all-to-all. In data-parallel training, workers exchange gradients at each step. Tensor, pipeline, and expert parallelism can create even tighter communication dependencies. The slowest worker or link can delay the entire job because the next phase cannot proceed until the required participants finish the current one.
This produces two failure modes that a collection of individually valid pod placements does not reveal:
- Partial allocation: some workers receive GPUs while the cluster lacks capacity for the rest. The allocated devices may remain idle or underused because the gang cannot begin useful work.
- Poor locality: every pod starts, but workers are scattered across racks or network domains. Communication consumes more of every training step, lowering samples or tokens processed per unit of time.
Neither problem necessarily looks dramatic in a basic infrastructure dashboard. GPU memory may be allocated, utilization may fluctuate, and pods may be running. Those signals show activity, not training productivity. A job can keep hardware occupied while delivering far less useful work than its accelerator count implies.
Gang scheduling solves the “when” problem
Gang scheduling treats the workers of a distributed job as a group. Instead of allowing a subset to bind indefinitely while other members remain pending, the scheduler waits until enough resources are available for the required membership. The objective is all-or-nothing progress: either the gang can assemble within its policy and timing constraints, or its resources remain available for another workload.
The Kubernetes scheduler-plugins project implements this pattern through its Coscheduling plugin. Pods are associated with a PodGroup, which expresses the minimum number of members required. The plugin coordinates their scheduling and exposes a group-level state rather than leaving operators to infer job health from a long list of individual pods.
This matters economically. A partially scheduled eight-worker job with four allocated GPUs is not simply “half ready.” If the framework needs all eight workers before training can proceed, those four GPUs are producing no useful training output. On a shared cluster, that stranded capacity also blocks other jobs that could run successfully.
Gang scheduling does not manufacture capacity. A job may still wait because quota is exhausted, the requested accelerator type is unavailable, or other placement constraints cannot be satisfied. Its benefit is making that waiting explicit and coordinated. Operators gain a clearer distinction between a job that is queued for capacity and one that has been admitted but cannot complete assembly.
Topology awareness solves the “where” problem
Starting every worker at once is only half the scheduling requirement. The physical relationship between GPUs determines the bandwidth and latency available to the training framework. Communication within a server may use NVLink or NVSwitch. Communication between servers traverses host interfaces and the cluster fabric. Crossing racks can add another constrained hop.
The exact penalty depends on the model and parallelism strategy, but the direction is consistent: as communication becomes more frequent or more data-intensive, placement becomes part of application performance. Two allocations with the same number and type of GPUs can deliver materially different training throughput.
Kueue’s topology-aware scheduling operates at admission time. Administrators describe topology levels through node labels, such as block, rack, and host. Kueue evaluates whether a workload can fit within an appropriate domain and records a topology assignment in the workload’s admission status. It then injects scheduling constraints into the pod templates so the normal Kubernetes scheduler must place workers within the selected domain.
That division of responsibility is important. Kueue is not a replacement for the kube-scheduler, and it does not discover the physical network automatically. It makes a higher-level admission decision using quotas, priorities, resource flavors, and declared topology. The kube-scheduler still binds individual pods to eligible nodes. The result depends on accurate labels and a faithful model of the underlying hardware.
The control path has two distinct checkpoints
Combining Kueue and Coscheduling creates a pipeline rather than a single scheduling event. First, Kueue decides whether a job should enter the schedulable population and whether a valid topology domain exists. It represents the job as a Workload, accounts for the total resource request across its pod sets, and records the admission decision. Once admitted, the job controller can create pods with the constraints Kueue selected.
Coscheduling then coordinates the binding of those pods as a gang. This separation gives platform teams a useful diagnostic boundary:
- If a workload remains inadmissible, investigate queue quota, priority, resource flavor, or topology capacity.
- If the workload is admitted with a topology assignment but its PodGroup remains pending, investigate scheduler constraints, taints, node availability, and gang assembly.
- If the gang is running but training throughput is low, inspect the network fabric, accelerator interconnect, drivers, and framework-level communication behavior.
Without these checkpoints, every delay risks being labeled “GPU shortage.” The layered model reveals whether the cluster lacks capacity, cannot find a legal placement, cannot assemble the group, or delivered a technically valid placement whose hardware path performs poorly.
Observability must follow the scheduling pipeline
Infrastructure teams should instrument each stage rather than relying on a single utilization graph. At the admission layer, Kueue exposes queue depth, admission wait time, and eviction metrics. A persistent rise in inadmissible workloads points toward quota or constraint problems, while increasing admission latency can show that demand is outpacing the capacity available to a queue.
The authoritative proof that topology-aware scheduling ran is the Kueue Workload status, specifically the topology assignment recorded for its pod sets. After that, operators should validate the result against the actual hardware path. A correct assignment can still underperform because of fabric degradation, driver trouble, or an inaccurate topology model.
At the gang layer, PodGroup state shows whether the minimum membership is assembling and becoming runnable. A group that stays pending or scheduling beyond its expected window should be correlated with pod events and scheduler logs. This avoids confusing a pre-admission queueing problem with a post-admission binding problem.
Finally, the application must supply the outcome metric. Steps per second, samples per second, tokens per second, collective-operation duration, and job completion time say whether GPU activity is producing training progress. The most useful dashboard connects those measures to placement and queue signals. Utilization remains useful, but only as one input to a productivity model.
What platform teams should change
The shift from independent pods to coordinated accelerator workloads requires several operational decisions.
Model the physical topology deliberately
Node labels become performance-critical configuration. Platform owners need a consistent vocabulary for hosts, racks, blocks, and other network domains, plus automation that keeps labels aligned with reality. A stale label can produce a placement that satisfies policy on paper while violating the intended bandwidth boundary.
Make job shape part of admission policy
Queues should account for the total gang, not just the next pod requesting a device. Priorities, quotas, and preemption policies need to reflect business value and job characteristics. Smaller fine-tuning jobs may tolerate a broader topology; large pretraining runs may justify waiting longer for tightly connected capacity.
Define timeouts as product decisions
A readiness or scheduling timeout determines how long scarce capacity can remain reserved while a gang struggles to assemble. Set it too high and GPUs stay stranded; set it too low and large jobs churn through eviction and requeue cycles. The correct value depends on image-pull times, node provisioning latency, workload size, and the cost of restarting admission.
Measure completed work per accelerator-hour
Cluster efficiency should be expressed in terms of useful training output, not merely allocated or busy devices. Tracking tokens, samples, or steps per accelerator-hour makes topology mistakes visible and helps teams compare placement policies. It also creates a common language between ML engineers, platform operators, and finance teams.
The remaining constraint is operational maturity
Gang scheduling and topology-aware admission make Kubernetes more suitable for distributed training, but they add a control layer that must be operated. Teams need compatible workload integrations, accurate resource requests, reliable labels, queue governance, and dashboards that cross controller boundaries. They also need to test failure paths such as a node loss inside an assigned topology domain.
The approach is therefore not a switch that automatically improves every AI job. Small jobs may gain little from tight placement, and over-constraining them can reduce overall cluster utilization. The scheduler needs enough flexibility to backfill capacity while preserving the topology requirements of communication-heavy jobs. Policies should encode workload sensitivity rather than applying the strictest placement to everything.
The larger lesson is that AI training changes the unit of scheduling. A pod can be healthy while its job is economically unhealthy. Kubernetes platforms that expose queue admission, gang assembly, physical topology, and framework throughput as one operational story will make better use of scarce accelerators. Those that optimize only for device allocation may keep GPUs busy while leaving much of their performance—and budget—on the table.


