AI Infrastructure Heats Up: From 15x Faster Inference to Planetary-Scale Models

The AI infrastructure landscape is shifting from “make it work” to “make it work everywhere, all the time, and affordably.” In the past two weeks alone, we’ve seen major releases from inference engines, cloud providers, and research labs that collectively raise the bar on speed, reliability, and operational sophistication. From speculative decoding breakthroughs that deliver 15x performance gains to planetary-scale geospatial inference platforms, here’s what’s actually happening under the hood.

vLLM v0.26.0: The Open Inference Engine Keeps Accelerating

vLLM released version 0.26.0 on July 27, and it’s one of the project’s most substantial updates in months. The release includes 411 commits from 212 contributors — 61 of them new — and introduces full support for the Inkling model family from Thinking Machines Lab.

What’s notable isn’t just the model support, but the depth of the integration. vLLM now handles Inkling’s piecewise CUDA graph compilation, Hopper FA4 relative attention, MTP=1 speculative decoding, and LoRA adapters — essentially the full stack needed to serve these models in production. The project also landed its native-speed transformers modeling backend earlier in July, which lets vLLM reuse Hugging Face model implementations without the usual performance penalty.

For anyone running open-weight models in production, vLLM remains the default choice, and this release reinforces why. The project’s velocity — both in contributor count and feature breadth — is unmatched in the open inference space. The fact that 61 new contributors joined this single release cycle speaks to how central vLLM has become to the AI infrastructure ecosystem.

NVIDIA DFlash: 15x Faster Inference on Blackwell

NVIDIA’s latest speculative decoding advancement, DFlash, is delivering genuinely impressive numbers. The company published benchmarks showing up to 15x throughput improvement for gpt-oss-120b on Blackwell at the same interactivity level, and nearly double the interactivity for Llama 3.1 8B compared to EAGLE-3 speculative decoding.

DFlash works differently from traditional speculative decoders. Instead of drafting tokens one at a time with a small model, it uses a block-diffusion drafter that generates entire blocks of candidate tokens in a single forward pass. The target model then verifies them in parallel. On Blackwell’s architecture, this parallelism is especially effective because it shifts work from the memory-bound sequential decode path into parallel GPU compute.

The research team has already released 20 DFlash checkpoints on Hugging Face with recipes for both Blackwell and Hopper GPUs, and integration is landing in TensorRT-LLM, SGLang, and vLLM. For latency-sensitive applications — coding assistants, agent workflows, real-time voice — this could meaningfully change the economics of serving large models. NVIDIA also published a companion piece on ModelExpress, a system for distributing model artifacts at network speed — addressing the increasingly painful problem of moving hundred-gigabyte checkpoints across infrastructure.

Together AI: Rethinking Inference Reliability and Routing

Together AI has been unusually prolific in July, publishing several pieces that reveal how mature inference platforms actually operate at scale. Their dedicated model inference architecture introduces a three-layer resource model: endpoints (stable names clients call), deployments (model + hardware + config combinations), and configs (immutable recipes specifying engine, GPU type, and optimization profile).

The clever part is the capacity-aware traffic split. Rather than fixed percentage routing, Together computes each deployment’s effective capacity as weight × ready_replicas and routes proportionally. This means autoscaling composes for free — when a deployment scales from 1 to 3 replicas, it automatically absorbs 3x the traffic without manual reconfiguration. Weight 1 vs weight 2 means “each replica of B works twice as hard,” not “B gets twice the traffic.”

This design enables sophisticated operations that would otherwise require custom tooling: A/B tests are just deployments with cohort assignments. Shadow experiments are deployments at weight zero receiving mirrored traffic. Stopping a deployment is bounding min and max replicas to zero. The entire advanced feature set collapses to a single primitive: “add a deployment, assign it a traffic weight.”

On the reliability front, their uptime engineering post breaks down what 99%, 99.9%, and 99.99% actually require in practice:

99% means your architecture can survive node-level failures: GPU hardware faults, driver crashes, thermal events. Getting there generally takes automated health checking, node draining, and fast replica replacement within a single data center.
99.9% means your architecture can survive a full data center failure. That usually means model weights deployed across two facilities, enough capacity on each side to absorb the full load, and live traffic routing to both — not a cold standby.
99.99% means your architecture can survive a regional outage. That typically calls for multi-region deployment with availability zone redundancy and reserved failover capacity.

The post is refreshingly honest about the gap between published SLAs and actual architecture — something more inference providers should emulate. GPU inference fails differently from conventional services, and the systems are tuned hard for performance. Hitting 1M tokens per minute per GPU at 200 tokens per second, or sub-50ms time-to-first-token on voice models, leaves little headroom for redundancy.

Together also announced a partnership with Y Combinator to deliver the first dedicated GPU cluster for the YC community, offering on-demand B200s without the usual two-year compute contracts. On-demand B200s are now available on Together GPU Clusters, marking one of the first ways to access NVIDIA’s latest hardware without long-term commitments.

Model Routing: Simple in Theory, Hard in Practice

IBM Research published a provocative piece on Hugging Face arguing that model routing — the seemingly straightforward idea of sending simple tasks to cheap models and hard tasks to expensive ones — is far more complex than most implementations acknowledge.

Their research on 417 AppWorld Test Challenge tasks revealed three counterintuitive findings that should reshape how teams think about routing:

Cost isn’t just model pricing. GPT-4.1 has lower per-token pricing than Claude Sonnet 4.6, yet cost nearly double in practice ($155 vs $79 total) because Sonnet’s lower cache-read pricing disproportionately benefited from the high context-reuse patterns typical of agent workloads. A router that only looks at pricing sheets is optimizing against the wrong numbers.

Complexity isn’t just task difficulty. A “simple” request like “summarize this contract” might trigger retrieval, compliance checks, tool use, and multiple refinement rounds. Meanwhile, a highly technical prompt might be handled efficiently by a smaller specialized model. You often don’t know how hard a task actually is until execution is underway.

Latency isn’t just model speed. Infrastructure factors — which hardware a model is running on, whether the cache is warm, how busy the endpoint is — often dominate end-to-end response times more than model size does. A theoretically faster model can still produce a slower experience if the serving conditions aren’t right.

IBM’s proposed solution is to treat routing as an optimization problem rather than a classification problem, optimizing across cost, quality, and latency simultaneously. Their router achieves 84% accuracy at $93 per task — a 21% cost reduction and 9% latency reduction compared to running Claude Opus alone, with only a 4% accuracy drop. Configuration 2 pushes cost even lower. The key insight: the router gives you a range of operating points to choose from depending on whether you want to prioritize cost, latency, or accuracy.

Ollama and LiteLLM: The Local and Unified Layers

On the local inference front, Ollama shipped v0.32.4 with MLX engine support for Laguna on Apple GPUs, speculative decoding draft model quantization, and faster Qwen3 MoE decoding. The v0.32.5 patch fixed an MLX Metal bug affecting NVFP4 model output quality. These incremental improvements matter because Ollama is increasingly the default way developers run models locally — the Apple Silicon optimizations in particular keep Mac users competitive with GPU-based workflows.

LiteLLM released v1.94.0 with a notable security enhancement: all Docker images are now signed with cosign, allowing teams to cryptographically verify image integrity before deployment. In an era where supply chain attacks on AI infrastructure are a real concern, this kind of verifiable artifact publishing should become standard. The project also added support for additional providers and models, continuing its role as the universal proxy that unifies dozens of LLM APIs behind a single OpenAI-compatible interface.

Planetary-Scale Inference: OlmoEarth and the Georgia Datacenter

At the extreme end of the infrastructure spectrum, two projects show just how wide the AI infrastructure space has become.

The OlmoEarth Platform from the Allen Institute for AI (AI2) tackles geospatial inference at planetary scale. Satellite imagery inference operates on an entirely different magnitude than text models — a single job can move terabytes of data and run for hours. Inputs may span multiple spectral bands, sensor types, and time steps across a large geographic area, coming from several providers each using different projections and resolutions.

AI2’s solution divides work into three hardware-matched stages: CPU for data acquisition and preprocessing, GPU for model inference, and CPU again for postprocessing and geographic stitching. The platform can process continent-scale areas in roughly a day at fractions of a penny per square kilometer. Developing it meant confronting challenges that others working on large-scale geospatial systems are likely to encounter: efficient data pipelines that handle high-volume I/O, multiprocess data loaders that keep GPUs fed, and infrastructure that recovers gracefully from the routine failures of distributed computing.

Meanwhile, OpenAI announced Project Camellia — a 3.2 gigawatt datacenter in Effingham County, Georgia, with power delivery phased between 2028 and 2032. The project includes $80 million in community benefits and up to $71 million in Codex credits for Georgia students. OpenAI committed that electricity rates will not go up for residents, the project will use a closed-loop water system, and it will prioritize local contractors and businesses. At 3.2 GW, this is among the largest datacenter power commitments ever announced, underscoring just how much compute infrastructure the AI transition requires.

What This Means for Practitioners

Taken together, these developments paint a clear picture: AI infrastructure is maturing along three axes simultaneously.

Speed: Speculative decoding (DFlash, vLLM’s MTP support), native transformers backends, and optimized CUDA kernels are squeezing more throughput from the same hardware. The days of “just add more GPUs” are giving way to “make each GPU work smarter.” A 15x improvement on existing hardware is the kind of leap that changes business models.

Reliability: Capacity-aware routing, multi-DC failover, and honest uptime engineering are raising the operational bar. 99.9% uptime now requires surviving full datacenter failures — a standard that would have seemed excessive a year ago. For products whose user experience depends on inference availability, this isn’t optional.

Intelligence: Model routing, multi-model orchestration, and agent harnesses are pushing complexity up the stack. The infrastructure layer is no longer just “serve a model” — it’s “serve the right model, at the right time, for the right cost.” IBM’s optimization-based approach points toward a future where routing decisions are as sophisticated as the models being routed.

For teams building AI products, the practical implication is that inference is becoming a genuine engineering discipline. The winners won’t be the ones with the biggest GPU clusters — they’ll be the ones with the most sophisticated understanding of how to route, optimize, and reliably serve models at scale. The infrastructure race is on, and the bar keeps rising.