# The Race to Optimize AI Inference: From vLLM’s Model Runner V2 to NVIDIA’s DFlash and Cloud Coding Agents

- Link: https://thestackobserver.com/the-race-to-optimize-ai-inference-from-vllms-model-runner-v2-to-nvidias-dflash-and-cloud-coding-agents/
- Published: 2026-07-15T11:43:24-04:00
- Author: Stackxx

Infrastructure for serving and scaling AI models has quietly become one of the most
competitive spaces in technology. While the headlines chase the latest frontier 
models, the teams building the engines that actually run those models are shipping
transformative improvements — often at a pace that outstrips model releases themselves.
The past two weeks have delivered major advances across inference engines, speculative
decoding, cloud-native agent runtimes, and data pipelines that will reshape how 
AI workloads are deployed in production.

## vLLM v0.25.0: Model Runner V2 Becomes the Default, PagedAttention Is Retired

The **vLLM project** released version **0.25.0** in mid-July, a milestone release
with 558 commits from 232 contributors, including 64 new contributors. The headline
change is that **Model Runner V2 (MRv2) is now the default execution path for all
dense models**, building on the quantized-model support introduced in the previous
release. This shift represents the formal retirement of the legacy architecture 
and a bet on a more modular, extensible inference engine.

MRv2 brings several new capabilities beyond its predecessor: support for **EVS (
Evolving Video Streaming)**, **realtime embeddings**, **prefix caching for Mamba
hybrid models**, and **multimodal-prefix bidirectional attention**. It also enables**
dynamic speculative decoding** compatible with full CUDA graphs. In parallel, the
legacy **PagedAttention implementation has been deleted entirely** — a decisive 
cleanup that removes one of the oldest attention paths in the codebase.

Version 0.25.0 also expands model support significantly. New architectures include**
LLaVA-OneVision-2**, **Unlimited OCR** with a Triton R-SWA backend, **MOSS-Transcribe-
Diarize**, and **Hy3** with token-suffix and JSON Schema array support. The **GLM-
5** and **DeepSeek-V3.2** families have been added to the model zoo, with GLM-5.2-
specific tuning and pipeline parallelism support for **MiniMax-M3**.

Perhaps most notably, vLLM **v0.25.1** followed quickly with two targeted bug fixes:
a deferred error for TorchCodec when system FFmpeg is missing, and a dtype-match
guard for mixed-precision allreduce RMSNorm quant fusions that had been corrupting
hidden states in models like Gemma and Qwen on NVFP4, producing garbage output such
as repeated exclamation tokens.

## Hugging Face’s Transformers Backend Reaches Native vLLM Speed

In a development that could reshape the relationship between model authors and inference
engines, the **Hugging Face transformers library** has achieved parity — and in 
some cases superiority — with vLLM’s hand-written native implementations. The transformers
vLLM backend now meets or beats native throughput across a range of model architectures
and sizes, from a 4B dense model on a single GPU to a 235B-parameter FP8 MoE running
on data plus expert parallelism across eight H100s.

The technical approach is sophisticated. The backend now uses **torch.fx** to perform
static graph analysis, identifying patterns that can be fused into optimized vLLM
kernels. Then it uses **AST manipulation** to rewrite the model’s source code in
place, applying runtime layer fusions that previously required custom vLLM implementations.

This matters because it collapses a long-standing workflow bottleneck. Previously,
model authors integrated a new architecture once for transformers (focused on readability
and correctness), then again for vLLM (focused on performance). Now, a single transformers
implementation automatically gains native vLLM speed through the `--model-impl transformers`
flag. Compatible architectures benefit from fused operations mapped to vLLM’s optimized
kernels, including expert parallelization for MoE models.

As a side effect, this reduces the maintenance burden on the vLLM team for new model
support. Models that use linear attention are not yet supported but are expected
to follow. Custom models living in private Hub repos may need updates for compliance,
but the broad implication is clear: transformers is becoming the canonical modeling
implementation, with vLLM providing the optimized inference runtime rather than 
duplicating modeling code.

## NVIDIA DFlash Delivers Up to 15x Inference Speedups on Blackwell

**NVIDIA** published detailed benchmarks for **DFlash**, an open-source block-diffusion
speculative decoding drafter that extends conventional speculative decoding by generating
entire token blocks in a single forward pass rather than token-by-token. This block-
parallel approach preserves the target model’s output quality through verification
while dramatically improving GPU utilization.

The results on **NVIDIA Blackwell** are striking. For **gpt-oss-120b**, DFlash increases
inference throughput by **up to 15x** at the same interactivity level. For **Llama
3.1 8B**, it nearly doubles interactivity at the same concurrency compared to the
state-of-the-art EAGLE-3 speculative decoder. The research team has released **20
DFlash checkpoints on Hugging Face** with recipes for both Blackwell and Hopper 
GPUs.

DFlash is also expanding beyond TensorRT-LLM into broader inference stacks. NVIDIA
is working with the **SGLang** and **vLLM** communities to integrate the drafter,
which adds a DFlash option alongside existing speculative decoding approaches like
EAGLE-3 and the new DSpark drafter introduced in vLLM v0.25.0. The convergence of
block-diffusion drafters with high-throughput inference engines points toward a 
near-term future where speculative decoding becomes the default rather than an optimization.

## Mistral Medium 3.5: Cloud Coding Agents and the Rise of Remote Agent Runtimes

**Mistral** released **Mistral Medium 3.5**, a 128B dense model with a 256k context
window that merges instruction-following, reasoning, and coding into a single weight
set. The model scores **77.6% on SWE-Bench Verified** and **91.4 on τ³-Telecom**,
placing it ahead of Devstral 2 and Qwen3.5 397B on real-world coding benchmarks.
It is released as open weights under a modified MIT license and can be self-hosted
on as few as four GPUs.

But the model release was paired with something equally significant: **Mistral Vibe
remote agents**. Coding agents have historically lived locally on developer laptops.
Mistral is moving them to the cloud, where they run asynchronously, in parallel,
and notify the user when complete. Sessions can be spawned from the Vibe CLI or 
directly within Le Chat, and a local CLI session can be “teleported” to the cloud
mid-task.

Le Chat also gained a new **Work mode** — a multi-step agent that calls tools in
parallel until a complex task is finished, powered by Medium 3.5. Reasoning effort
is now configurable per request, so the same model can handle quick chat replies
and deep agentic runs without switching endpoints. This architecture — a single 
powerful model with configurable reasoning tiers, paired with cloud agent runtimes
and human oversight for sensitive actions — represents a template other vendors 
are likely to follow.

## Ollama v0.32.0 Pivots Toward Interactive Agents

**Ollama** shipped version **0.32.0** with a significant UX pivot: running `ollama`
without arguments now launches an **interactive agent experience** for coding, web
search, and task delegation. The default model is **glm-5.2:cloud**, suggesting 
closer integration with cloud-hosted inference rather than purely local execution.

The release also rebrands the Codex App integration as **ChatGPT**, adds deprecation
warnings for older agent models like CodeLlama and Qwen2.5-coder, and simplifies
the integration selection menu to surface only the most popular options. On the 
inference side, v0.31.2 enabled **flash attention on older NVIDIA GPUs** (compute
capability 6.x) and allowed iGPUs to offload vision models with memory-fit padding.

Ollama’s trajectory mirrors the broader shift from local inference tooling to agent
runtimes. What began as a convenient way to run quantized models locally is evolving
into a client for cloud-backed agentic workflows, with the local engine remaining
as a fallback for privacy-sensitive or offline work.

## SkyPilot and Hugging Face Zero-Egress Storage

A practical but significant infrastructure development came from the **SkyPilot**
and **Hugging Face** partnership. Teams can now mount Hugging Face Storage buckets
and Hub repositories into SkyPilot jobs via `hf://` URLs, reading models and datasets
directly from the Hub with **zero egress fees**. This means a SkyPilot task can 
run on GPUs in any of 20+ clouds without paying cross-cloud data transfer costs.

The integration uses Hugging Face’s **Xet-backed deduplication**, so incremental
checkpoints and model variants only transfer changed chunks. The Hugging Face team
upstreamed FUSE fixes for unprivileged containers, making this viable in shared 
Kubernetes environments. For teams running training or fine-tuning across multiple
clouds for spot capacity, this eliminates one of the largest hidden costs in multi-
cloud ML workflows.

## LiteLLM Adds Signed Docker Images and UI Improvements

**LiteLLM** continues its rapid release cadence with **v1.94.0-dev.1**, which introduces**
cosign-signed Docker images** for supply chain security. Every release is now signed
with a pinned commit hash, allowing verification that the running container matches
the published source. The release also includes UI improvements for the complexity
auto-router test connection and a migration to OpenAPI React Query for the management
interface.

LiteLLM’s role as a unified gateway for multi-provider inference is becoming more
critical as organizations mix proprietary APIs with self-hosted endpoints. Signed
artifacts and improved observability are table stakes for production deployments
at scale.

## NVIDIA’s Open Data Push for Agentic AI

Separately, **NVIDIA** published a detailed post on its **open data strategy for
agentic AI**, emphasizing that weights alone are insufficient for reproducible agent
behavior. The company released open datasets including **Nemotron-CC-v2** (synthetic-
enhanced Common Crawl for pretraining), **Nemotron-CC-MATH-v1** (synthetic math 
for reasoning), and broader pretraining collections. The theme: if agents call tools,
execute workflows, and retrieve information, developers need inspectable data, not
just opaque model weights.

Synthetic data is positioned as the key scaling mechanism — preserving organizational
signals without exposing proprietary sources. With nearly 145 ICML 2026 papers citing
Nemotron models and datasets, NVIDIA is clearly investing in open data as a competitive
moat alongside its hardware dominance.

## What This Means for Production AI Infrastructure

Several converging trends emerge from these releases:

 * **Inference engines are consolidating.** vLLM’s MRv2, Hugging Face’s unified 
   transformers backend, and Ollama’s agent pivot all point toward fewer, more capable
   inference runtimes rather than fragmented custom implementations.
 * **Speculative decoding is becoming standard.** DFlash, DSpark, and EAGLE-3 are
   all maturing simultaneously, and block-diffusion drafters may soon be expected
   rather than exotic.
 * **Agent runtimes are moving to the cloud.** Mistral’s remote agents and Ollama’s
   cloud-backed interactive mode signal that local-only agent execution is a transitional
   phase, not an end state.
 * **Data pipelines matter as much as models.** NVIDIA’s open data push and Hugging
   Face’s zero-egress storage integration reflect growing awareness that the infrastructure
   around the model — data access, synthetic generation, deduplication — is where
   competitive advantage increasingly lives.

The next few months will likely see further integration: vLLM’s transformers backend
expanding to more architectures, DFlash landing in SGLang, and cloud agent runtimes
standardizing around configurable reasoning tiers. For operators, the message is
clear: the inference layer is where the action is.

## Sources

 * [vLLM v0.25.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.25.0)
 * [vLLM v0.25.1 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.25.1)
 * [Native-speed vLLM transformers modeling backend — Hugging Face Blog](https://huggingface.co/blog/native-speed-vllm-transformers-backend)
 * [Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash — NVIDIA Blog](https://developer.nvidia.com/blog/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding/)
 * [Remote agents in Vibe. Powered by Mistral Medium 3.5 — Mistral](https://mistral.ai/news/vibe-remote-agents-mistral-medium-3-5/)
 * [Ollama v0.32.0 Release Notes](https://github.com/ollama/ollama/releases/tag/v0.32.0)
 * [Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot — Hugging Face Blog](https://huggingface.co/blog/skypilot-hf-storage)
 * [LiteLLM v1.94.0-dev.1 Release Notes](https://github.com/BerriAI/litellm/releases/tag/v1.94.0-dev.1)
 * [Data for Agents — NVIDIA & Hugging Face Blog](https://huggingface.co/blog/nvidia/open-data-for-agents)