The Stack Observer
Cloud Native

The Open-Weight AI Infrastructure Stack Is Growing Up

•Stackxx•Cloud Native

The open-weight AI ecosystem is having its infrastructure moment. In the span of just two weeks, Together AI shipped a full production inference platform, announced a dedicated GPU cluster partnership with Y Combinator, and closed an $800 million Series C — while vLLM pushed its most consequential release in months and Hugging Face proved that its Transformers backend can now match custom vLLM implementations on throughput. The message is unambiguous: the tooling that runs open models is catching up to the models themselves.

Together AI’s Production Inference Platform

On July 23, Together AI unveiled a major overhaul of its Dedicated Model Inference platform. The update treats inference as more than hosting a model behind an API — it builds in the deployment primitives that enterprises actually need.

The headline features are production rollout controls: canary deployments, blue-green updates, rolling updates with automatic rollback, A/B testing, and shadow traffic. Teams can route a percentage of live traffic to a new model version or configuration, measure real performance, and automatically roll back if key metrics regress. Multiple deployments can sit behind a single stable endpoint, so the application layer never needs to change even as the model serving underneath evolves.

Autoscaling is similarly configurable, supporting not just request volume but GPU utilization, time-to-first-token, latency, decoding speed, or throughput. Deployment profiles package Together’s tested configurations for different hardware and optimization targets, so teams can start with proven defaults rather than assembling a serving stack from scratch.

Startup time — historically dominated by moving hundreds of gigabytes of weights into place — has been reduced roughly 4× through a rebuilt model caching and distribution layer. Frontier-scale MoE models like MiniMax M3 and GLM 5.2 now deploy in roughly 7–14 minutes, down from what used to be far longer waits.

Together also opened a closed beta for custom training, including full-weight and LoRA reinforcement learning and supervised fine-tuning, with checkpoints deployable straight to production inference endpoints. The idea is a unified lifecycle: train, evaluate, and serve on the same platform without stitching together separate systems.

From Compute to Control

The inference platform launch was not an isolated announcement. It followed a July 20 partnership with Y Combinator to deliver the first dedicated YC GPU cluster, giving portfolio startups access to compute without two-year contracts. Startups can spin up GPUs for short-term sprints while benefiting from long-term rates — a direct response to the reality that compute procurement has become the biggest bottleneck for early-stage AI companies.

On the cluster reliability side, Together shipped passive health checks and auto node repair for its GPU clusters. Passive checks observe live workloads for GPU failures, thermal throttling, and Xid errors without synthetic overhead. When degradation is detected, the system recommends remediation — reboot, reprovision, failover, or removal — for operator approval before execution. This “automation with a human touch” approach targets the failure modes that actually take down multi-day training jobs.

Together also rebuilt its Slurm-on-Kubernetes stack, based on its Slinky fork, to eliminate zombie processes, move job accounting to durable storage, reliably clean up GPU-holding child processes at job end, and rebuild GPU state after pod reschedules. The new stack also exposes DCGM metrics in Grafana dashboards by default.

vLLM Removes PagedAttention, Goes All-In on Model Runner V2

While Together was shipping platform features, the open inference engine vLLM released version 0.25.0 on July 11 — a release that marks a structural transition. Model Runner V2 is now the default execution path for all dense models, and the legacy PagedAttention implementation has been removed entirely.

MRv2 adds support for EVS (extreme value sampling), real-time embeddings, prefix caching for Mamba hybrid models, multimodal-prefix bidirectional attention, and dynamic speculative decoding compatible with full CUDA graphs. The Transformers modeling backend, which lets any Hugging Face model run inside vLLM without a custom port, is now officially as fast as native vLLM implementations for many architectures — a claim Hugging Face demonstrated with benchmarks across Qwen3 4B, 32B, and 235B FP8 MoE models.

The significance is practical: model authors no longer need to write custom vLLM kernels to get native-speed inference. A single --model-impl transformers flag composes with standard parallelism options and achieves parity through runtime graph analysis with torch.fx and AST-based code rewriting for fused operations. This collapses the gap between “works in Transformers” and “production-ready in vLLM.”

What It Means

Taken together, these announcements describe an open-weight infrastructure stack that is graduating from research tooling to production platform. Together AI is building the managed layer — deployment controls, autoscaling, observability, training-to-serving handoff — that enterprises need to run open models at scale. vLLM and Hugging Face are collapsing the friction between model authoring and optimized serving. The result is that teams can now adopt open-weight models not just for cost savings, but for the operational control that closed APIs cannot provide.

The $800 million Series C Together announced on July 1 suggests investors agree: the economics of closed models do not scale, and the infrastructure to run open alternatives is now a platform business worth betting on.

Sources

Next signal