The Stack Observer
AI

The Agentic Shift: How AI Infrastructure Is Being Rebuilt for Long-Running Agents

•Stackxx•AI

The transition from single-turn chatbots to long-running, multi-step AI agents is reshaping the entire AI infrastructure stack. What started as a simple API call to a large language model has evolved into complex workflows where agents spawn sub-agents, maintain massive context windows, call tools, and execute code over minutes or even days. This shift is creating new demands for inference engines, operational tooling, and hardware optimization that the industry is racing to meet.

The Agentic Inference Bottleneck

Agentic workloads introduce a fundamentally different access pattern to AI infrastructure. In traditional chatbot inference, each request is largely independent. In agentic systems, the same conversation prefix gets reused dozens or even hundreds of times across sequential tool calls, plan revisions, and sub-agent delegations. NVIDIA’s analysis of production agent workloads reveals a write-once-read-many (WORM) pattern where KV cache reads dramatically outpace writes.

Take Claude Code as an example. After the initial conversation prefix is computed and cached, subsequent API calls hit 85–97% cache reuse. In multi-agent swarms, aggregate cache hit rates reach 97.2% across teammate agents. This creates an 11.7x read-to-write ratio — the system reads from cache nearly 12 times for every token it writes. Maximizing cache reuse across all workers and keeping KV blocks warm and routable has become the central optimization target for agentic inference infrastructure.

But this level of cache optimization isn’t available out of the box for teams running open-source models on their own GPUs. Managed API providers control prefix matching, cache placement, and eviction policies. For everyone else, new infrastructure layers are needed to bridge this gap.

NVIDIA Dynamo: Building Agent-Native Inference

NVIDIA’s answer is Dynamo, a full-stack inference platform designed specifically for agentic workloads. Rather than treating inference as stateless request/response pairs, Dynamo introduces three layers of optimization: the frontend API, the router, and KV cache management.

At the frontend, Dynamo serves multiple protocols — including the newer v1/responses and v1/messages APIs — through a common internal representation. This matters because these newer APIs use typed content blocks, allowing the orchestrator to see boundaries between thinking, tool calls, and text. The router can then apply different cache and scheduling policies per block type.

Perhaps most innovative is Dynamo’s agent hints extension. Agent harnesses like Claude Code, Codex, and OpenClaw have global context that traditional inference servers never see: which agents are blocked on tool calls, how many turns remain, and whether a call is a quick lookup or long synthesis. Agent hints allow harnesses to attach structured metadata — including estimated output length, priority, and speculative prefill signals — directly to requests. The router uses this to make agent-aware scheduling and caching decisions.

The router itself maintains a global index of KV cache blocks across workers, enabling KV-aware placement that routes requests to workers with the highest cache overlap rather than round-robin. This is critical because without cache-aware routing, turn 2 of a conversation has only a ~1/N chance of landing on the same worker as turn 1 — each miss requiring a full prefix recomputation.

Nemotron 3 Ultra: A Model Built for Agents

Inference optimization alone isn’t sufficient; the models themselves must be architected for agentic workloads. NVIDIA recently released Nemotron 3 Ultra, a 550B-parameter Mixture-of-Experts model with only 55B active parameters — roughly one-tenth the compute of a dense model of equivalent capacity.

Nemotron 3 Ultra achieves 5x higher throughput than comparable open models while delivering frontier accuracy. On benchmark suites like PinchBench, EnterpriseOps-Gym, and Ruler @ 1M context, it matches or exceeds models like GLM 5.1, Kimi K2.6, and Qwen3.5 — despite being significantly smaller and faster.

Several architectural innovations power this efficiency:

  • Hybrid Mamba-Transformer architecture: Mamba layers improve sequence efficiency for long-context workloads, while Transformer layers preserve precise recall when agents need to retrieve specific facts from massive context windows.
  • NVFP4 precision: The same checkpoint runs across NVIDIA Hopper, Blackwell, and Ampere GPUs, delivering up to 5x higher throughput at the same interactivity compared to BF16 on Blackwell.
  • LatentMoE: More efficient expert routing for workflows spanning reasoning, code generation, tool calls, and domain-specific logic.
  • Multi-token prediction (MTP): Predicts multiple future tokens in a single forward pass, improving throughput for long outputs and multi-turn workflows.
  • Multi-Teacher On-Policy Distillation (MOPD): The model learns from over 10 specialized teacher models across domains, with student rollout generation, teacher scoring, and student optimization fully pipelined for efficiency.

Perhaps most importantly for infrastructure costs, Nemotron 3 Ultra completes agentic benchmarks using up to 30% fewer total tokens than comparable models. For teams running agents at scale, this directly translates to lower inference costs and faster task completion.

Quantization and Inference Engine Optimization

While architectural innovations squeeze more performance from each GPU, quantization techniques are reducing the resources required per inference call. NVIDIA’s recent work on FP8 quantization with TensorRT demonstrates how to convert FP8 checkpoints into production-ready inference engines.

In benchmarks on an NVIDIA RTX 6000 Ada GPU, FP8 quantization delivered a 48% reduction in engine size for image encoders and 34% for text encoders, with latency speedups of 1.39x and 1.45x respectively. The same savings carry over to GPU VRAM usage at inference time, enabling larger batch sizes or more concurrent agents on the same hardware.

TensorRT-LLM’s AutoDeploy feature further simplifies this by automating the process of building optimized inference engines for new model architectures, reducing the manual tuning typically required when deploying new models.

Enterprise Operational Maturity

As AI infrastructure moves from development labs into enterprise production, operational maturity is becoming as important as raw performance. NVIDIA’s DGX Spark Enterprise Manageability provides a complete operational framework from first provisioning to end-of-life retirement.

The framework delivers agentless SSH execution with bounded standard JSON output, integrating directly into existing CMDB, SIEM, and monitoring pipelines. It includes six operational lifecycle phases: procurement and receiving, initial provisioning, ongoing monitoring, maintenance windows, incident response, and end-of-life cascade and redeployment.

Two diagnostic tools address common pain points in AI infrastructure:

  • spark_diagctl.py operates in L1 mode for fast health posture checks (disk, network, drivers) and L2 mode for deep evidence bundles including GPU telemetry, kernel logs, and PCIe state.
  • reset_reason_reporter.py correlates multiple evidence sources to produce structured root cause assessments for system reboots, deliberately using conservative classifications to avoid speculative conclusions.

The framework also supports fully air-gapped deployments through custom installation patterns using cloud-init and on-premises mirrors — a critical capability for regulated industries and sovereign AI deployments.

The Open Source Ecosystem

While NVIDIA builds vertically integrated solutions, the open-source ecosystem continues to expand the frontier of what’s possible with commodity hardware. vLLM remains the dominant open inference engine, with recent releases adding support for new models like JetBrains’ Mellum v2 and AMD Zen CPU acceleration through zentorch kernels.

The rapid pace of open model releases — from Nemotron 3 Ultra to Cohere’s North Mini Code to JetBrains’ Mellum2 — means inference engines must adapt quickly. vLLM’s recent v0.22.1 patch release addressed model-loading regressions, multi-node Ray data-parallel hangs, and DeepSeek-V4 initialization issues, demonstrating the ongoing engineering effort required to keep pace with the model ecosystem.

Meanwhile, SGLang and other runtimes continue to push performance boundaries with features like radix cache eviction and priority-based memory management, creating healthy competition that benefits all inference workloads.

Looking Ahead

The convergence of agentic AI, efficient model architectures, and optimized inference infrastructure is creating a new category of AI factory — facilities purpose-built for manufacturing intelligence at scale. NVIDIA’s Vera CPU, Blackwell GPUs, and DOCA in-silicon security are all being positioned for this era.

For infrastructure teams, the key insight is that agentic workloads demand fundamentally different optimization targets than traditional LLM serving. Cache reuse rates matter more than raw throughput per request. Context window efficiency matters more than single-turn latency. And operational maturity — from air-gapped provisioning to automated diagnostics — matters as much as model accuracy.

The teams that recognize this shift early and invest in agent-native infrastructure will find themselves with dramatically lower costs per task completion and the ability to run longer, more capable agent workflows on the same hardware budget.

Sources

Next signal