# Inference Is the New Training: vLLM 0.26, Native-Speed Transformers, and the 800M Bet on Open AI Infrastructure

- Link: https://thestackobserver.com/inference-is-the-new-training-vllm-0-26-native-speed-transformers-and-the-800m-bet-on-open-ai-infrastructure/
- Published: 2026-08-05T08:18:08-04:00
- Author: Stackxx

The AI infrastructure landscape is moving at a velocity that would make even the
most caffeinated DevOps engineer blink. In the past few weeks alone, we have seen
a **monster vLLM release** with over 400 commits, a **native-speed transformers 
backend** that eliminates the need for hand-written inference ports, **$800 million
in fresh funding** for open-source inference platforms, and a research breakthrough
that **doubles agentic inference throughput** by treating multi-turn workflows as
schedulable programs. If you are building or operating AI infrastructure, the summer
of 2026 is not quiet.

## vLLM v0.26.0: The Everything Release

vLLM has long been the workhorse of open-source LLM serving, and version **0.26.0**(
released late July) is the kind of drop that justifies why. With **411 commits from
212 contributors** (61 of them new), this is not a point release — it is a platform
expansion.

The headline addition is full support for the **Inkling model family**, the new 
architecture from Thinking Machines that made its public debut earlier in July. 
vLLM now handles Inkling’s base modeling, piecewise CUDA graph support, Hopper FA4
relative attention, MTP=1 speculative decoding, LoRA, and ModelOpt NVFP4 quantization.
That is a full support stack from day one.

For operators running **DeepSeek-V4**, vLLM 0.26.0 delivers a dedicated performance
push: a specialized routing kernel trims 2.94% off end-to-end TPOT, a fused topk_bias
kernel provides 1.5–2x speedup, and redundant repeat/copy removal shaves another
1.8%. ROCm users get a two-stage compressor for HCA prefill, sparse decode/prefill
optimizations, and DSpark speculative decoding on AMD and XPU backends. These are
not marginal gains — they are the difference between a model that fits in production
and one that does not.

Other notable improvements include **fp32 lm_head for generation models** (improving
accuracy for generation heads), **flexible attention backends** selectable per KV-
cache group, and a **maturation of KV offloading and tiered secondary storage** 
with object-store tiers, workload identity, and DP-replica-aware tiering. The Rust
frontend now supports multimodal video and audio, plus a native vllm-bench port.
And Transformers 5.13.0 integration means more models — Olmo/Olmo2, MistralLarge3,
HunyuanVL — now run through the Transformers modeling backend.

In short, if you are not on vLLM 0.26.0 yet, your inference stack is leaving performance
on the table.

## Transformers Backend Now Matches Native vLLM Speed

Here is a development that changes the economics of model deployment: **Hugging 
Face’s transformers modeling backend for vLLM now runs as fast — or faster — than
vLLM’s own hand-written native implementations** for many architectures.

This is significant because, historically, model authors had to write a custom vLLM
port to squeeze maximum inference performance out of their models. The Transformers
backend was a convenience for quick deployment, but it could not match the throughput
of purpose-built code. That gap is now closed.

The trick is a clever runtime optimization pipeline. The backend uses **torch.fx
for static graph analysis**, then manipulates the model’s abstract syntax tree (
AST) to rewrite operations in place. This enables:

 * Fused operations mapped to optimized vLLM kernels, including Expert Parallelization
   for MoE models
 * Inference of tensor-parallel and pipeline-parallel plans automatically
 * Full compatibility with torch.compile and CUDA graphs
 * Models that remain usable for training — the same code works for training, evals,
   and RL rollouts

Hugging Face benchmarked this on three Qwen3 variants: a 4B dense model, a 32B dense
model with tensor parallelism, and a 235B-parameter FP8 MoE running data + expert
parallelism across 8x H100. In every case, the Transformers backend **met or beat**
native throughput. Running any compatible Hugging Face model is now a single flag:`--
model-impl transformers`.

The implication is clear: **model authors no longer need to write a second inference
implementation**. Integrate once to Transformers, and you get vLLM-native speed 
for free. That removes a major bottleneck in the open model ecosystem.

## Together AI Bets $800M on Open Inference

While the engineering is exciting, the money is what signals conviction. Together
AI announced a **$800 million Series C** in early July, backed by an investor roster
that includes NVIDIA, Aramco Ventures, Vista Equity, General Catalyst, and Salesforce
Ventures. On top of the equity, the company secured commitments for **over 500 MW
of compute capacity** to support expected growth.

Together AI’s thesis is straightforward: as AI moves from prototypes to production
infrastructure, the cost structure of closed frontier LLMs becomes unsustainable.
Inference bills compound faster than budgets, forcing companies to ration intelligence.
The alternative is open-weights models running on efficient infrastructure — and
Together AI is positioning itself as the platform for that shift.

The company also announced a **partnership with Y Combinator** to deliver the first
dedicated YC GPU cluster, giving startups on-demand access to B200s without multi-
year contracts. This is compute-as-a-service tailored for the AI-native startup,
not the enterprise datacenter.

## Agentic Inference Gets Its First Real Scheduler

Most inference engines schedule at the **request level** — each LLM call is an independent
unit. That works fine for chat, but it breaks down for agentic workloads where a
single workflow might alternate between GPU-heavy reasoning and GPU-idle tool calls
for dozens of turns.

When an agent pauses for a compiler or search tool, its KV cache gets evicted to
make room for other requests. When the agent resumes, the engine recomputes the 
entire conversation history from scratch — which evicts the next agent’s cache, 
and so on. At high concurrency, this **KV cache thrashing** collapses throughput.

**ThunderAgent**, a research project from Georgia Tech, UIUC, CMU, and Together 
AI (and an ICML 2026 Spotlight paper), solves this by treating each agent workflow
as a **schedulable program**. A lightweight scheduler sits above the inference engine,
tracking which LLM calls belong to which agent workflow. It preserves KV cache across
tool-call boundaries, eliminates thrashing, and achieves **more than 2x single-node
throughput** with roughly 10x lower P50 latency at high concurrency. On an 8-node
cluster, it delivers **2.4x speedup with near-linear scaling** from 16 to 64 GPUs.

Crucially, ThunderAgent is a **drop-in addition**: one `program_id` field, OpenAI-
compatible, and it composes with existing engine optimizations like speculative 
decoding. The authors have released the code on GitHub. For anyone running agents
at scale, this is a must-evaluate system.

## The Autoscaling Problem Nobody Talks About

Together AI also published a deep dive on **autoscaling for LLM inference** — a 
topic that sounds solved but absolutely is not for production serving teams.

The problem is that classic autoscaling assumptions break when applied to LLMs:

 * **GPU utilization is a lie.** A GPU can read 60% utilized while the engine’s 
   request queue is already backing up. Utilization measures arithmetic intensity,
   not pressure.
 * **Cold starts take minutes.** A new replica must be placed, pull tens of gigabytes
   of weights, load into VRAM, and warm up. By the time a spike is visible, it is
   already too late to scale out.

Together AI’s platform exposes **eight inference-native metrics** for autoscaling:
in-flight requests, TTFT (time to first token), GPU utilization, token throughput,
and more. The key insight is an **asymmetric tuning strategy**: scale up eagerly(
false scale-ups cost a few replica-minutes) and scale down patiently (false scale-
downs cost latency and double cold-start fees when you scale back up).

For teams running production inference, this is the most practical autoscaling guidance
currently published.

## Speculative Decoding Goes Block-Diffusion on Blackwell

NVIDIA’s **DFlash** speculative decoding system, built on a block-diffusion drafter
rather than autoregressive token-by-token drafting, is now delivering **up to 15x
inference performance improvement** on gpt-oss-120b running on NVIDIA Blackwell 
hardware.

The innovation is simple in concept, hard in execution: instead of generating candidate
tokens one at a time, DFlash generates an **entire block of candidates in a single
forward pass**, which the target model then verifies in parallel. This shifts sequential
drafting work into block-parallel GPU operations — a perfect match for the memory-
bandwidth-bound decode phase on modern GPUs.

At 500-600 tokens per second per user (a typical interactive agent target), DFlash
delivers **more than 15x throughput over autoregressive decoding** and 1.5x over
the prior state-of-the-art EAGLE-3. The research team has released 20 DFlash checkpoints
on Hugging Face with recipes for both Blackwell and Hopper GPUs, and integration
is spreading to SGLang and vLLM.

## Ollama and LiteLLM: The Edge Keeps Moving

On the local inference front, **Ollama v0.32.6** continues its rapid release cadence.
Highlights include **automatic speculative decoding for Qwen3.5 on Apple GPUs** 
via the MLX engine, full OpenAI wire-format compliance for streaming chat completions,
and support for cloud-only models like kimi-k3. Experimental image generation has
been temporarily removed, but the core engine keeps tightening its compatibility
story.

**LiteLLM** is pushing hard on production trust: v1.97.0-dev.1 introduced Docker
image signing with cosign, using a pinned commit-hash key for cryptographic verification.
For teams running LiteLLM as a gateway between their application and dozens of model
providers, supply-chain integrity is not optional.

## What This Means for Infrastructure Teams

Three trends are converging that every AI infrastructure team should internalize:

**First, the open-source inference stack is closing the gap with proprietary clouds.**
Between vLLM’s massive feature expansion, Hugging Face’s native-speed Transformers
backend, and DFlash speculative decoding on consumer and datacenter GPUs, the performance
argument for closed APIs is eroding.

**Second, agentic workloads are forcing a rethink of scheduling and autoscaling.**
Request-level engines were built for chat. Agents need program-level schedulers 
like ThunderAgent, KV-cache-aware routing, and autoscaling on metrics that actually
describe inference pressure — not GPU utilization theater.

**Third, the economics of inference are becoming the economics of AI.** Together
AI’s $800M bet and 500 MW compute pipeline signal that inference — not training —
is where the money is flowing. As models become commodities, the infrastructure 
that serves them becomes the moat.

If your team is still treating inference as “just an API call,” the summer of 2026
is your wake-up call.

## Sources

 * [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
 * [Hugging Face: Native-speed vLLM transformers modeling backend](https://huggingface.co/blog/native-speed-vllm-transformers-backend)
 * [Together AI: Announcing our $800M Series C](https://www.together.ai/blog/announcing-our-series-c)
 * [Together AI: ThunderAgent — 2x Faster Agentic Inference](https://www.together.ai/blog/thunderagent)
 * [Together AI: Autoscaling endpoints for LLM inference](https://www.together.ai/blog/autoscaling-endpoints-for-llm-inference)
 * [NVIDIA: Boost Inference Performance up to 15x on Blackwell Using DFlash](https://developer.nvidia.com/blog/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding)
 * [Ollama v0.32.6 Release Notes](https://github.com/ollama/ollama/releases/tag/v0.32.6)
 * [LiteLLM v1.97.0-dev.1 Release Notes](https://github.com/BerriAI/litellm/releases/tag/v1.97.0-dev.1)
 * [NVIDIA: ModelExpress — Distributing Model Artifacts](https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light)
 * [NVIDIA: Vera Storage Benchmarks for AI-Native Storage](https://developer.nvidia.com/blog/nvidia-vera-storage-benchmarks)
