The Race to Optimize AI Inference: From vLLM's Model Runner V2 to NVIDIA's DFlash and Cloud Coding Agents

Infrastructure for serving and scaling AI models has quietly become one of the most competitive spaces in technology. While the headlines chase the latest frontier models, the teams building the engines that actually run those models are shipping transformative improvements — often at a pace that outstrips model releases themselves. The past two weeks have delivered major advances across inference engines, speculative decoding, cloud-native agent runtimes, and data pipelines that will reshape how AI workloads are deployed in production.

vLLM v0.25.0: Model Runner V2 Becomes the Default, PagedAttention Is Retired

The vLLM project released version 0.25.0 in mid-July, a milestone release with 558 commits from 232 contributors, including 64 new contributors. The headline change is that Model Runner V2 (MRv2) is now the default execution path for all dense models, building on the quantized-model support introduced in the previous release. This shift represents the formal retirement of the legacy architecture and a bet on a more modular, extensible inference engine.

MRv2 brings several new capabilities beyond its predecessor: support for EVS (Evolving Video Streaming), realtime embeddings, prefix caching for Mamba hybrid models, and multimodal-prefix bidirectional attention. It also enables dynamic speculative decoding compatible with full CUDA graphs. In parallel, the legacy PagedAttention implementation has been deleted entirely — a decisive cleanup that removes one of the oldest attention paths in the codebase.

Version 0.25.0 also expands model support significantly. New architectures include LLaVA-OneVision-2, Unlimited OCR with a Triton R-SWA backend, MOSS-Transcribe-Diarize, and Hy3 with token-suffix and JSON Schema array support. The GLM-5 and DeepSeek-V3.2 families have been added to the model zoo, with GLM-5.2-specific tuning and pipeline parallelism support for MiniMax-M3.

Perhaps most notably, vLLM v0.25.1 followed quickly with two targeted bug fixes: a deferred error for TorchCodec when system FFmpeg is missing, and a dtype-match guard for mixed-precision allreduce RMSNorm quant fusions that had been corrupting hidden states in models like Gemma and Qwen on NVFP4, producing garbage output such as repeated exclamation tokens.

Hugging Face’s Transformers Backend Reaches Native vLLM Speed

In a development that could reshape the relationship between model authors and inference engines, the Hugging Face transformers library has achieved parity — and in some cases superiority — with vLLM’s hand-written native implementations. The transformers vLLM backend now meets or beats native throughput across a range of model architectures and sizes, from a 4B dense model on a single GPU to a 235B-parameter FP8 MoE running on data plus expert parallelism across eight H100s.

The technical approach is sophisticated. The backend now uses torch.fx to perform static graph analysis, identifying patterns that can be fused into optimized vLLM kernels. Then it uses AST manipulation to rewrite the model’s source code in place, applying runtime layer fusions that previously required custom vLLM implementations.

This matters because it collapses a long-standing workflow bottleneck. Previously, model authors integrated a new architecture once for transformers (focused on readability and correctness), then again for vLLM (focused on performance). Now, a single transformers implementation automatically gains native vLLM speed through the --model-impl transformers flag. Compatible architectures benefit from fused operations mapped to vLLM’s optimized kernels, including expert parallelization for MoE models.

As a side effect, this reduces the maintenance burden on the vLLM team for new model support. Models that use linear attention are not yet supported but are expected to follow. Custom models living in private Hub repos may need updates for compliance, but the broad implication is clear: transformers is becoming the canonical modeling implementation, with vLLM providing the optimized inference runtime rather than duplicating modeling code.

NVIDIA DFlash Delivers Up to 15x Inference Speedups on Blackwell

NVIDIA published detailed benchmarks for DFlash, an open-source block-diffusion speculative decoding drafter that extends conventional speculative decoding by generating entire token blocks in a single forward pass rather than token-by-token. This block-parallel approach preserves the target model’s output quality through verification while dramatically improving GPU utilization.

The results on NVIDIA Blackwell are striking. For gpt-oss-120b, DFlash increases inference throughput by up to 15x at the same interactivity level. For Llama 3.1 8B, it nearly doubles interactivity at the same concurrency compared to the state-of-the-art EAGLE-3 speculative decoder. The research team has released 20 DFlash checkpoints on Hugging Face with recipes for both Blackwell and Hopper GPUs.

DFlash is also expanding beyond TensorRT-LLM into broader inference stacks. NVIDIA is working with the SGLang and vLLM communities to integrate the drafter, which adds a DFlash option alongside existing speculative decoding approaches like EAGLE-3 and the new DSpark drafter introduced in vLLM v0.25.0. The convergence of block-diffusion drafters with high-throughput inference engines points toward a near-term future where speculative decoding becomes the default rather than an optimization.

Mistral Medium 3.5: Cloud Coding Agents and the Rise of Remote Agent Runtimes

Mistral released Mistral Medium 3.5, a 128B dense model with a 256k context window that merges instruction-following, reasoning, and coding into a single weight set. The model scores 77.6% on SWE-Bench Verified and 91.4 on τ³-Telecom, placing it ahead of Devstral 2 and Qwen3.5 397B on real-world coding benchmarks. It is released as open weights under a modified MIT license and can be self-hosted on as few as four GPUs.

But the model release was paired with something equally significant: Mistral Vibe remote agents. Coding agents have historically lived locally on developer laptops. Mistral is moving them to the cloud, where they run asynchronously, in parallel, and notify the user when complete. Sessions can be spawned from the Vibe CLI or directly within Le Chat, and a local CLI session can be “teleported” to the cloud mid-task.

Le Chat also gained a new Work mode — a multi-step agent that calls tools in parallel until a complex task is finished, powered by Medium 3.5. Reasoning effort is now configurable per request, so the same model can handle quick chat replies and deep agentic runs without switching endpoints. This architecture — a single powerful model with configurable reasoning tiers, paired with cloud agent runtimes and human oversight for sensitive actions — represents a template other vendors are likely to follow.

Ollama v0.32.0 Pivots Toward Interactive Agents

Ollama shipped version 0.32.0 with a significant UX pivot: running ollama without arguments now launches an interactive agent experience for coding, web search, and task delegation. The default model is glm-5.2:cloud, suggesting closer integration with cloud-hosted inference rather than purely local execution.

The release also rebrands the Codex App integration as ChatGPT, adds deprecation warnings for older agent models like CodeLlama and Qwen2.5-coder, and simplifies the integration selection menu to surface only the most popular options. On the inference side, v0.31.2 enabled flash attention on older NVIDIA GPUs (compute capability 6.x) and allowed iGPUs to offload vision models with memory-fit padding.

Ollama’s trajectory mirrors the broader shift from local inference tooling to agent runtimes. What began as a convenient way to run quantized models locally is evolving into a client for cloud-backed agentic workflows, with the local engine remaining as a fallback for privacy-sensitive or offline work.

SkyPilot and Hugging Face Zero-Egress Storage

A practical but significant infrastructure development came from the SkyPilot and Hugging Face partnership. Teams can now mount Hugging Face Storage buckets and Hub repositories into SkyPilot jobs via hf:// URLs, reading models and datasets directly from the Hub with zero egress fees. This means a SkyPilot task can run on GPUs in any of 20+ clouds without paying cross-cloud data transfer costs.

The integration uses Hugging Face’s Xet-backed deduplication, so incremental checkpoints and model variants only transfer changed chunks. The Hugging Face team upstreamed FUSE fixes for unprivileged containers, making this viable in shared Kubernetes environments. For teams running training or fine-tuning across multiple clouds for spot capacity, this eliminates one of the largest hidden costs in multi-cloud ML workflows.

LiteLLM Adds Signed Docker Images and UI Improvements

LiteLLM continues its rapid release cadence with v1.94.0-dev.1, which introduces cosign-signed Docker images for supply chain security. Every release is now signed with a pinned commit hash, allowing verification that the running container matches the published source. The release also includes UI improvements for the complexity auto-router test connection and a migration to OpenAPI React Query for the management interface.

LiteLLM’s role as a unified gateway for multi-provider inference is becoming more critical as organizations mix proprietary APIs with self-hosted endpoints. Signed artifacts and improved observability are table stakes for production deployments at scale.

NVIDIA’s Open Data Push for Agentic AI

Separately, NVIDIA published a detailed post on its open data strategy for agentic AI, emphasizing that weights alone are insufficient for reproducible agent behavior. The company released open datasets including Nemotron-CC-v2 (synthetic-enhanced Common Crawl for pretraining), Nemotron-CC-MATH-v1 (synthetic math for reasoning), and broader pretraining collections. The theme: if agents call tools, execute workflows, and retrieve information, developers need inspectable data, not just opaque model weights.

Synthetic data is positioned as the key scaling mechanism — preserving organizational signals without exposing proprietary sources. With nearly 145 ICML 2026 papers citing Nemotron models and datasets, NVIDIA is clearly investing in open data as a competitive moat alongside its hardware dominance.

What This Means for Production AI Infrastructure

Several converging trends emerge from these releases:

Inference engines are consolidating. vLLM’s MRv2, Hugging Face’s unified transformers backend, and Ollama’s agent pivot all point toward fewer, more capable inference runtimes rather than fragmented custom implementations.
Speculative decoding is becoming standard. DFlash, DSpark, and EAGLE-3 are all maturing simultaneously, and block-diffusion drafters may soon be expected rather than exotic.
Agent runtimes are moving to the cloud. Mistral’s remote agents and Ollama’s cloud-backed interactive mode signal that local-only agent execution is a transitional phase, not an end state.
Data pipelines matter as much as models. NVIDIA’s open data push and Hugging Face’s zero-egress storage integration reflect growing awareness that the infrastructure around the model — data access, synthetic generation, deduplication — is where competitive advantage increasingly lives.

The next few months will likely see further integration: vLLM’s transformers backend expanding to more architectures, DFlash landing in SGLang, and cloud agent runtimes standardizing around configurable reasoning tiers. For operators, the message is clear: the inference layer is where the action is.

The Race to Optimize AI Inference: From vLLM’s Model Runner V2 to NVIDIA’s DFlash and Cloud Coding Agents