For the past three years, the dominant narrative in AI has been about building bigger, better models. In 2026, that conversation has shifted. The most important engineering problem in artificial intelligence is no longer how to train a better model — it’s how to serve one efficiently at scale. From autoscaling endpoints that understand LLM-specific signals, to schedulers that prevent agentic workloads from thrashing GPU memory, to zero-egress storage systems that decouple compute from cloud vendor lock-in, the entire stack beneath the model is being rebuilt. The race hasn’t stopped. It has just moved down a layer.
The Bottleneck Moved From Models to Compute
When GPT-3 was trained in 2020, Microsoft assembled a dedicated supercomputer with over 10,000 GPUs and 285,000 CPU cores for OpenAI. At the time, it was one of the five largest systems in the world. Six years later, that number reads like a starting point. Anthropic now runs simultaneous multi-gigawatt commitments across four separate hardware platforms — Amazon, Google, Microsoft, and AMD — while Meta has signed comparable deals of its own. Spreading commitments across four vendors is what compute scarcity looks like when you have effectively unlimited capital and still can’t get enough from a single source.
But here’s what changed: capability is no longer the binding constraint. The models are good enough. The question is whether you can afford to run them.
In a recent Hugging Face analysis, the authors draw a compelling parallel between enterprise AI and commercial aviation. An aircraft’s costs accrue by the calendar hour — financing, depreciation, insurance, maintenance — while its revenue only accrues by the flight hour. Every hour on the ground shrinks the output side while the cost side keeps running. Two airlines with comparable fleets can end up with radically different economics, and most of that gap traces back to one number: utilization.
GPUs work the same way. They accrue cost by the calendar hour — financing, depreciation, power, cooling — whether they’re doing anything useful or not. Their output only accrues by the compute hour. Two companies with comparable GPU budgets increasingly diverge based on how much of that hardware is actually doing useful work, not on how much of it either one owns.
The scarcity didn’t disappear as AI scaled. It moved up the chain and landed on a different resource entirely. Intelligence carried the industry this far. Utilization is where the next real constraint is forming.
Why “Just Autoscale It” Doesn’t Work for LLMs
For stateless web services, autoscaling is a solved problem. For LLM inference, it breaks two fundamental assumptions.
First, GPU-style metrics lie about load. A GPU can read 60% utilized while the engine’s request queue is already backing up. Utilization measures arithmetic intensity, not pressure. Scaling deployments on the wrong signal means the system responds to a number that doesn’t describe the actual problem. As Together AI explained in a recent deep dive, this creates a vicious cycle: under-provision and your p95 latency degrades sharply the moment traffic exceeds what replicas can batch. LLM serving degrades nonlinearly — a replica at its concurrency limit doesn’t get “a bit slower,” it starts queueing, and time-to-first-token (TTFT) can blow up from 200ms to 15 seconds.
Second, cold starts can take several minutes. A new replica has to be placed on a GPU node, pull tens of gigabytes of weights, load them into VRAM, and warm up. By the time a spike triggers scale-up, it’s already too late. The autoscaler’s real job is to act early on leading signals, not trailing ones.
Together AI’s solution is a catalog of inference-native metrics: in-flight requests (a leading indicator that rises before latency visibly degrades), TTFT and end-to-end latency (SLO-driven trailing signals), and GPU/token utilization (efficiency-driven cost optimizers). The key insight is asymmetric timing: an eager scale-up window (mistakes cost dollars) and a patient scale-down window (mistakes cost latency and dollars, because you’ll just scale back up, paying the cold start again).
Agentic Workloads Break the Engine Itself
If autoscaling is hard, agentic inference is harder. Systems like Claude Code, Codex, and OpenClaw reason, call tools, read results, and reason again — often for dozens of turns before completing a task. Training these agents requires large-scale synthetic data generation, which in turn requires running agentic inference at high concurrency.
Existing inference engines schedule at the request level. Each LLM call is an independent unit with no awareness that it belongs to a longer, multi-turn workflow. When an agent pauses for a tool call, its KV cache can be evicted to make room for other requests, only to be recomputed from scratch when the agent resumes. At high concurrency, this cascade of evictions and recomputations creates KV cache thrashing — a vicious cycle where memory pressure forces constant recomputation, which generates more memory pressure.
Together AI’s ThunderAgent, accepted as an ICML 2026 Spotlight paper, solves this by adding a missing abstraction layer. It treats each agentic workflow as a schedulable program, tracking execution phase, KV cache footprint, and node placement. Under memory pressure, it selectively pauses low-priority workflows to reduce competition for cache. With fewer concurrent programs, the remaining active workflows achieve significantly higher KV cache hit rates and lower latency. On an 8-node cluster, ThunderAgent delivers 2.4× speedup with near-linear scaling from 16 to 64 GPUs.
The drop-in integration is elegant: one program_id field, OpenAI-compatible, works with existing engine-level optimizations like speculative decoding. The system sits between agentic clients and inference backends without requiring changes to either.
The Convergence of Training and Inference Infrastructure
Another signal that the infrastructure layer is maturing: the once-sharp boundary between training and inference code is blurring.
Hugging Face recently announced that its transformers library can now serve as a native-speed modeling backend inside vLLM — no custom port required. Using torch.fx for static graph analysis and AST manipulation for runtime layer fusion, the transformers backend dynamically applies inference-specific optimizations that match (or exceed) hand-written vLLM implementations. Tested on Qwen3 models ranging from 4B dense to 235B-parameter FP8 MoE, the backend meets or beats native throughput on every configuration.
The practical implication is significant: model authors write once for transformers, and immediately get ultra-fast vLLM inference for free. The same model code can be used for training, evaluation, RL rollouts, and production serving. This eliminates the duplicate integration work that previously slowed every new architecture’s path to production.
Meanwhile, Hugging Face and SkyPilot have partnered to eliminate the cross-cloud storage tax. Most teams now hold GPU capacity across multiple vendors — a block on AWS, a cluster on Lambda, maybe an on-prem rack. But object stores are regional and per-cloud, meaning feeding a GPU in a different vendor’s datacenter means either keeping copies everywhere or paying egress fees. Hugging Face Storage now mounts directly into SkyPilot tasks via an hf:// URL, with no egress or CDN fees. Reading your model weights onto any GPU cluster costs nothing, regardless of which cloud it’s on. Storage stops deciding where you run.
Microsoft Bets on Heterogeneity
While the open-source ecosystem converges on portable abstractions, the hyperscalers are expanding their hardware portfolios. Microsoft announced in July 2026 that it is bringing AMD’s latest Helios AI platform and next-generation EPYC datacenter processors to Azure in three new VM families:
- HDv2 VMs — Nearly 500 physical 6th Gen AMD EPYC CPU cores, 4TB RAM, 32TB NVMe, designed for data preparation, search, reinforcement learning, and agent coordination at scale.
- HXv2 VMs — 176 EPYC cores clocked above 5GHz with 3D V-cache, targeting silicon design and technical computing workloads.
- ND MI455X v7 VMs — Powered by AMD’s Helios rackscale solution, designed for large-scale inference on reasoning, search, and agentic workloads.
The message is clear: no single architecture wins every workload. Customer choice is the core design principle, and Azure is positioning itself as the neutral ground where NVIDIA, AMD, and custom silicon coexist.
Voice AI Forces a Rethink of the Entire Inference Stack
Perhaps the most radical infrastructure rethink comes from an unexpected place: voice. OpenAI’s GPT-Live, the company’s third-generation voice system, removes the traditional turn detector from the audio path entirely. Its voice model is full-duplex — it listens and speaks simultaneously. This eliminates the latency of waiting for a “turn detector” to decide when the user has stopped speaking, which in previous systems could add hundreds of milliseconds before the LLM even began processing.
But delivering this at scale required reworking every layer of the inference stack. The system separates media flow from application logic: audio moves between client and voice model on a dedicated fast path, while deeper reasoning and tool use happen asynchronously on a separate RPC boundary. A slow tool call can delay its own result, but cannot stall the flow of speech. The media frontend was rewritten in Go (replacing a Python asyncio implementation), improving p95 frame delivery smoothness to match the previous system’s p50.
Stateful inference introduces its own operational complexity. A voice session may remain active for a long time, but its context continuously grows. To handle this, OpenAI built a seamless handoff mechanism: when a transition is needed, a replacement model instance is warmed alongside the existing one, pre-filled with current session context, and cut over when ready. Context compaction — reducing accumulated history to fit within model limits — is handled the same way: compact off the live path, prepare a replacement instance, switch over without audible interruption.
These are not model improvements. They are infrastructure inventions, built specifically for a new class of real-time, stateful, streaming workloads that traditional request-response inference was never designed to handle.
What This Means for the Industry
The pattern across all these developments is consistent: the AI infrastructure stack is being rebuilt from the scheduling layer up. Classic autoscalers, request-level inference engines, and regional storage systems were built for a world where AI workloads looked like web traffic — discrete, stateless, and tolerably slow. The workloads arriving in 2026 look nothing like that. They are conversational, agentic, multi-turn, memory-intensive, and latency-sensitive in ways that break every assumption the previous generation of infrastructure was built on.
For enterprises, the implication is strategic. Buying more GPUs adds capacity, but not efficiency. The airlines that survived weren’t the ones with the biggest fleets — they were the ones that kept their aircraft in the air. Similarly, the enterprises that extract value from AI in 2026 won’t be the ones with the most compute. They’ll be the ones with the orchestration layer to keep it busy with the right work at the right time.
The bottleneck has moved. The solutions are arriving. And the winners will be the teams that build their infrastructure for the workloads that are coming, not the ones that have already arrived.
