# Kubernetes AI Training Exposes the Limits of Pod-at-a-Time Scheduling

- Link: https://thestackobserver.com/kubernetes-ai-training-exposes-the-limits-of-pod-at-a-time-scheduling/
- Published: 2026-11-13T12:56:37-05:00
- Author: Stackxx

Kubernetes became the default control plane for cloud infrastructure by treating
pods as independently schedulable units. Distributed AI training challenges that
assumption. A training job may comprise dozens or hundreds of workers that must 
start together and communicate continuously, yet the default scheduler evaluates
pods one at a time. A cluster can therefore appear busy while expensive GPUs wait
for missing peers or exchange gradients across a slow network path.

A recent CNCF technical analysis puts two complementary mechanisms at the center
of the remedy: **gang scheduling**, which coordinates when a group of pods can run,
and **topology-aware scheduling**, which constrains where those pods land. The practical
lesson is broader than a new scheduler feature. For AI infrastructure, allocation
is not enough. The control plane must understand that a training job is one coordinated
unit and that the distance between its accelerators directly affects useful throughput.

## Why pod-at-a-time scheduling breaks down

The ordinary Kubernetes scheduling model works well when replicas are interchangeable
and largely independent. A web service can make progress with nine replicas while
the tenth is pending. A stateless worker can be restarted on another node without
coordinating a simultaneous restart of every other worker.

Distributed training behaves differently. Workers repeatedly synchronize through
collective communications such as all-reduce, gather, reduce, and all-to-all. In
data-parallel training, workers exchange gradients at each step. Tensor, pipeline,
and expert parallelism can create even tighter communication dependencies. The slowest
worker or link can delay the entire job because the next phase cannot proceed until
the required participants finish the current one.

This produces two failure modes that a collection of individually valid pod placements
does not reveal:

 * **Partial allocation:** some workers receive GPUs while the cluster lacks capacity
   for the rest. The allocated devices may remain idle or underused because the 
   gang cannot begin useful work.
 * **Poor locality:** every pod starts, but workers are scattered across racks or
   network domains. Communication consumes more of every training step, lowering
   samples or tokens processed per unit of time.

Neither problem necessarily looks dramatic in a basic infrastructure dashboard. 
GPU memory may be allocated, utilization may fluctuate, and pods may be running.
Those signals show activity, not training productivity. A job can keep hardware 
occupied while delivering far less useful work than its accelerator count implies.

## Gang scheduling solves the “when” problem

Gang scheduling treats the workers of a distributed job as a group. Instead of allowing
a subset to bind indefinitely while other members remain pending, the scheduler 
waits until enough resources are available for the required membership. The objective
is all-or-nothing progress: either the gang can assemble within its policy and timing
constraints, or its resources remain available for another workload.

The Kubernetes scheduler-plugins project implements this pattern through its Coscheduling
plugin. Pods are associated with a `PodGroup`, which expresses the minimum number
of members required. The plugin coordinates their scheduling and exposes a group-
level state rather than leaving operators to infer job health from a long list of
individual pods.

This matters economically. A partially scheduled eight-worker job with four allocated
GPUs is not simply “half ready.” If the framework needs all eight workers before
training can proceed, those four GPUs are producing no useful training output. On
a shared cluster, that stranded capacity also blocks other jobs that could run successfully.

Gang scheduling does not manufacture capacity. A job may still wait because quota
is exhausted, the requested accelerator type is unavailable, or other placement 
constraints cannot be satisfied. Its benefit is making that waiting explicit and
coordinated. Operators gain a clearer distinction between a job that is queued for
capacity and one that has been admitted but cannot complete assembly.

## Topology awareness solves the “where” problem

Starting every worker at once is only half the scheduling requirement. The physical
relationship between GPUs determines the bandwidth and latency available to the 
training framework. Communication within a server may use NVLink or NVSwitch. Communication
between servers traverses host interfaces and the cluster fabric. Crossing racks
can add another constrained hop.

The exact penalty depends on the model and parallelism strategy, but the direction
is consistent: as communication becomes more frequent or more data-intensive, placement
becomes part of application performance. Two allocations with the same number and
type of GPUs can deliver materially different training throughput.

Kueue’s topology-aware scheduling operates at admission time. Administrators describe
topology levels through node labels, such as block, rack, and host. Kueue evaluates
whether a workload can fit within an appropriate domain and records a topology assignment
in the workload’s admission status. It then injects scheduling constraints into 
the pod templates so the normal Kubernetes scheduler must place workers within the
selected domain.

That division of responsibility is important. Kueue is not a replacement for the
kube-scheduler, and it does not discover the physical network automatically. It 
makes a higher-level admission decision using quotas, priorities, resource flavors,
and declared topology. The kube-scheduler still binds individual pods to eligible
nodes. The result depends on accurate labels and a faithful model of the underlying
hardware.

## The control path has two distinct checkpoints

Combining Kueue and Coscheduling creates a pipeline rather than a single scheduling
event. First, Kueue decides whether a job should enter the schedulable population
and whether a valid topology domain exists. It represents the job as a `Workload`,
accounts for the total resource request across its pod sets, and records the admission
decision. Once admitted, the job controller can create pods with the constraints
Kueue selected.

Coscheduling then coordinates the binding of those pods as a gang. This separation
gives platform teams a useful diagnostic boundary:

 * If a workload remains inadmissible, investigate queue quota, priority, resource
   flavor, or topology capacity.
 * If the workload is admitted with a topology assignment but its PodGroup remains
   pending, investigate scheduler constraints, taints, node availability, and gang
   assembly.
 * If the gang is running but training throughput is low, inspect the network fabric,
   accelerator interconnect, drivers, and framework-level communication behavior.

Without these checkpoints, every delay risks being labeled “GPU shortage.” The layered
model reveals whether the cluster lacks capacity, cannot find a legal placement,
cannot assemble the group, or delivered a technically valid placement whose hardware
path performs poorly.

## Observability must follow the scheduling pipeline

Infrastructure teams should instrument each stage rather than relying on a single
utilization graph. At the admission layer, Kueue exposes queue depth, admission 
wait time, and eviction metrics. A persistent rise in inadmissible workloads points
toward quota or constraint problems, while increasing admission latency can show
that demand is outpacing the capacity available to a queue.

The authoritative proof that topology-aware scheduling ran is the Kueue Workload
status, specifically the topology assignment recorded for its pod sets. After that,
operators should validate the result against the actual hardware path. A correct
assignment can still underperform because of fabric degradation, driver trouble,
or an inaccurate topology model.

At the gang layer, PodGroup state shows whether the minimum membership is assembling
and becoming runnable. A group that stays pending or scheduling beyond its expected
window should be correlated with pod events and scheduler logs. This avoids confusing
a pre-admission queueing problem with a post-admission binding problem.

Finally, the application must supply the outcome metric. Steps per second, samples
per second, tokens per second, collective-operation duration, and job completion
time say whether GPU activity is producing training progress. The most useful dashboard
connects those measures to placement and queue signals. Utilization remains useful,
but only as one input to a productivity model.

## What platform teams should change

The shift from independent pods to coordinated accelerator workloads requires several
operational decisions.

### Model the physical topology deliberately

Node labels become performance-critical configuration. Platform owners need a consistent
vocabulary for hosts, racks, blocks, and other network domains, plus automation 
that keeps labels aligned with reality. A stale label can produce a placement that
satisfies policy on paper while violating the intended bandwidth boundary.

### Make job shape part of admission policy

Queues should account for the total gang, not just the next pod requesting a device.
Priorities, quotas, and preemption policies need to reflect business value and job
characteristics. Smaller fine-tuning jobs may tolerate a broader topology; large
pretraining runs may justify waiting longer for tightly connected capacity.

### Define timeouts as product decisions

A readiness or scheduling timeout determines how long scarce capacity can remain
reserved while a gang struggles to assemble. Set it too high and GPUs stay stranded;
set it too low and large jobs churn through eviction and requeue cycles. The correct
value depends on image-pull times, node provisioning latency, workload size, and
the cost of restarting admission.

### Measure completed work per accelerator-hour

Cluster efficiency should be expressed in terms of useful training output, not merely
allocated or busy devices. Tracking tokens, samples, or steps per accelerator-hour
makes topology mistakes visible and helps teams compare placement policies. It also
creates a common language between ML engineers, platform operators, and finance 
teams.

## The remaining constraint is operational maturity

Gang scheduling and topology-aware admission make Kubernetes more suitable for distributed
training, but they add a control layer that must be operated. Teams need compatible
workload integrations, accurate resource requests, reliable labels, queue governance,
and dashboards that cross controller boundaries. They also need to test failure 
paths such as a node loss inside an assigned topology domain.

The approach is therefore not a switch that automatically improves every AI job.
Small jobs may gain little from tight placement, and over-constraining them can 
reduce overall cluster utilization. The scheduler needs enough flexibility to backfill
capacity while preserving the topology requirements of communication-heavy jobs.
Policies should encode workload sensitivity rather than applying the strictest placement
to everything.

The larger lesson is that AI training changes the unit of scheduling. A pod can 
be healthy while its job is economically unhealthy. Kubernetes platforms that expose
queue admission, gang assembly, physical topology, and framework throughput as one
operational story will make better use of scarce accelerators. Those that optimize
only for device allocation may keep GPUs busy while leaving much of their performance—
and budget—on the table.

## Sources

 * [CNCF: Gang scheduling and topology-aware placement for AI training in Kubernetes](https://www.cncf.io/blog/2026/11/12/gang-scheduling-and-topology-aware-placement-for-ai-training-in-kubernetes/)
 * [Kueue documentation: Topology-aware scheduling](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/)
 * [Kubernetes scheduler-plugins: Coscheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling)
