Inference Is the New Training: vLLM 0.26, Native-Speed Transformers, and the 800M Bet on Open AI Infrastructure

The AI infrastructure landscape is moving at a velocity that would make even the most caffeinated DevOps engineer blink. In the past few weeks alone, we have seen a monster vLLM release with over 400 commits, a native-speed transformers backend that eliminates the need for hand-written inference ports, $800 million in fresh funding for open-source inference platforms, and a research breakthrough that doubles agentic inference throughput by treating multi-turn workflows as schedulable programs. If you are building or operating AI infrastructure, the summer of 2026 is not quiet.

vLLM v0.26.0: The Everything Release

vLLM has long been the workhorse of open-source LLM serving, and version 0.26.0 (released late July) is the kind of drop that justifies why. With 411 commits from 212 contributors (61 of them new), this is not a point release — it is a platform expansion.

The headline addition is full support for the Inkling model family, the new architecture from Thinking Machines that made its public debut earlier in July. vLLM now handles Inkling’s base modeling, piecewise CUDA graph support, Hopper FA4 relative attention, MTP=1 speculative decoding, LoRA, and ModelOpt NVFP4 quantization. That is a full support stack from day one.

For operators running DeepSeek-V4, vLLM 0.26.0 delivers a dedicated performance push: a specialized routing kernel trims 2.94% off end-to-end TPOT, a fused topk_bias kernel provides 1.5–2x speedup, and redundant repeat/copy removal shaves another 1.8%. ROCm users get a two-stage compressor for HCA prefill, sparse decode/prefill optimizations, and DSpark speculative decoding on AMD and XPU backends. These are not marginal gains — they are the difference between a model that fits in production and one that does not.

Other notable improvements include fp32 lm_head for generation models (improving accuracy for generation heads), flexible attention backends selectable per KV-cache group, and a maturation of KV offloading and tiered secondary storage with object-store tiers, workload identity, and DP-replica-aware tiering. The Rust frontend now supports multimodal video and audio, plus a native vllm-bench port. And Transformers 5.13.0 integration means more models — Olmo/Olmo2, MistralLarge3, HunyuanVL — now run through the Transformers modeling backend.

In short, if you are not on vLLM 0.26.0 yet, your inference stack is leaving performance on the table.

Transformers Backend Now Matches Native vLLM Speed

Here is a development that changes the economics of model deployment: Hugging Face’s transformers modeling backend for vLLM now runs as fast — or faster — than vLLM’s own hand-written native implementations for many architectures.

This is significant because, historically, model authors had to write a custom vLLM port to squeeze maximum inference performance out of their models. The Transformers backend was a convenience for quick deployment, but it could not match the throughput of purpose-built code. That gap is now closed.

The trick is a clever runtime optimization pipeline. The backend uses torch.fx for static graph analysis, then manipulates the model’s abstract syntax tree (AST) to rewrite operations in place. This enables:

Fused operations mapped to optimized vLLM kernels, including Expert Parallelization for MoE models
Inference of tensor-parallel and pipeline-parallel plans automatically
Full compatibility with torch.compile and CUDA graphs
Models that remain usable for training — the same code works for training, evals, and RL rollouts

Hugging Face benchmarked this on three Qwen3 variants: a 4B dense model, a 32B dense model with tensor parallelism, and a 235B-parameter FP8 MoE running data + expert parallelism across 8x H100. In every case, the Transformers backend met or beat native throughput. Running any compatible Hugging Face model is now a single flag: --model-impl transformers.

The implication is clear: model authors no longer need to write a second inference implementation. Integrate once to Transformers, and you get vLLM-native speed for free. That removes a major bottleneck in the open model ecosystem.

Together AI Bets $800M on Open Inference

While the engineering is exciting, the money is what signals conviction. Together AI announced a $800 million Series C in early July, backed by an investor roster that includes NVIDIA, Aramco Ventures, Vista Equity, General Catalyst, and Salesforce Ventures. On top of the equity, the company secured commitments for over 500 MW of compute capacity to support expected growth.

Together AI’s thesis is straightforward: as AI moves from prototypes to production infrastructure, the cost structure of closed frontier LLMs becomes unsustainable. Inference bills compound faster than budgets, forcing companies to ration intelligence. The alternative is open-weights models running on efficient infrastructure — and Together AI is positioning itself as the platform for that shift.

The company also announced a partnership with Y Combinator to deliver the first dedicated YC GPU cluster, giving startups on-demand access to B200s without multi-year contracts. This is compute-as-a-service tailored for the AI-native startup, not the enterprise datacenter.

Agentic Inference Gets Its First Real Scheduler

Most inference engines schedule at the request level — each LLM call is an independent unit. That works fine for chat, but it breaks down for agentic workloads where a single workflow might alternate between GPU-heavy reasoning and GPU-idle tool calls for dozens of turns.

When an agent pauses for a compiler or search tool, its KV cache gets evicted to make room for other requests. When the agent resumes, the engine recomputes the entire conversation history from scratch — which evicts the next agent’s cache, and so on. At high concurrency, this KV cache thrashing collapses throughput.

ThunderAgent, a research project from Georgia Tech, UIUC, CMU, and Together AI (and an ICML 2026 Spotlight paper), solves this by treating each agent workflow as a schedulable program. A lightweight scheduler sits above the inference engine, tracking which LLM calls belong to which agent workflow. It preserves KV cache across tool-call boundaries, eliminates thrashing, and achieves more than 2x single-node throughput with roughly 10x lower P50 latency at high concurrency. On an 8-node cluster, it delivers 2.4x speedup with near-linear scaling from 16 to 64 GPUs.

Crucially, ThunderAgent is a drop-in addition: one program_id field, OpenAI-compatible, and it composes with existing engine optimizations like speculative decoding. The authors have released the code on GitHub. For anyone running agents at scale, this is a must-evaluate system.

The Autoscaling Problem Nobody Talks About

Together AI also published a deep dive on autoscaling for LLM inference — a topic that sounds solved but absolutely is not for production serving teams.

The problem is that classic autoscaling assumptions break when applied to LLMs:

GPU utilization is a lie. A GPU can read 60% utilized while the engine’s request queue is already backing up. Utilization measures arithmetic intensity, not pressure.
Cold starts take minutes. A new replica must be placed, pull tens of gigabytes of weights, load into VRAM, and warm up. By the time a spike is visible, it is already too late to scale out.

Together AI’s platform exposes eight inference-native metrics for autoscaling: in-flight requests, TTFT (time to first token), GPU utilization, token throughput, and more. The key insight is an asymmetric tuning strategy: scale up eagerly (false scale-ups cost a few replica-minutes) and scale down patiently (false scale-downs cost latency and double cold-start fees when you scale back up).

For teams running production inference, this is the most practical autoscaling guidance currently published.

Speculative Decoding Goes Block-Diffusion on Blackwell

NVIDIA’s DFlash speculative decoding system, built on a block-diffusion drafter rather than autoregressive token-by-token drafting, is now delivering up to 15x inference performance improvement on gpt-oss-120b running on NVIDIA Blackwell hardware.

The innovation is simple in concept, hard in execution: instead of generating candidate tokens one at a time, DFlash generates an entire block of candidates in a single forward pass, which the target model then verifies in parallel. This shifts sequential drafting work into block-parallel GPU operations — a perfect match for the memory-bandwidth-bound decode phase on modern GPUs.

At 500-600 tokens per second per user (a typical interactive agent target), DFlash delivers more than 15x throughput over autoregressive decoding and 1.5x over the prior state-of-the-art EAGLE-3. The research team has released 20 DFlash checkpoints on Hugging Face with recipes for both Blackwell and Hopper GPUs, and integration is spreading to SGLang and vLLM.

Ollama and LiteLLM: The Edge Keeps Moving

On the local inference front, Ollama v0.32.6 continues its rapid release cadence. Highlights include automatic speculative decoding for Qwen3.5 on Apple GPUs via the MLX engine, full OpenAI wire-format compliance for streaming chat completions, and support for cloud-only models like kimi-k3. Experimental image generation has been temporarily removed, but the core engine keeps tightening its compatibility story.

LiteLLM is pushing hard on production trust: v1.97.0-dev.1 introduced Docker image signing with cosign, using a pinned commit-hash key for cryptographic verification. For teams running LiteLLM as a gateway between their application and dozens of model providers, supply-chain integrity is not optional.

What This Means for Infrastructure Teams

Three trends are converging that every AI infrastructure team should internalize:

First, the open-source inference stack is closing the gap with proprietary clouds. Between vLLM’s massive feature expansion, Hugging Face’s native-speed Transformers backend, and DFlash speculative decoding on consumer and datacenter GPUs, the performance argument for closed APIs is eroding.

Second, agentic workloads are forcing a rethink of scheduling and autoscaling. Request-level engines were built for chat. Agents need program-level schedulers like ThunderAgent, KV-cache-aware routing, and autoscaling on metrics that actually describe inference pressure — not GPU utilization theater.

Third, the economics of inference are becoming the economics of AI. Together AI’s $800M bet and 500 MW compute pipeline signal that inference — not training — is where the money is flowing. As models become commodities, the infrastructure that serves them becomes the moat.

If your team is still treating inference as “just an API call,” the summer of 2026 is your wake-up call.