The Stack Observer
AI

NVIDIA Rubin, Together AI’s $800M Bet, and Hugging Face’s Native vLLM Speed: The Infrastructure Convergence Reshaping AI

•Stackxx•AI

The AI infrastructure landscape is undergoing its most significant transformation since the first GPU clusters were dedicated to model training. We are not merely adding more GPUs — we are redesigning the data center itself as a single unit of compute purpose-built for agentic AI. This week, three major threads converged: NVIDIA unveiled the Rubin GPU architecture and set a world record for MoE pre-training; Together AI raised $800 million to scale open-source inference infrastructure; and Hugging Face made it possible for any model on its Hub to run at native vLLM speeds without a custom port.

The Rubin Architecture: A GPU Purpose-Built for Agentic AI

NVIDIA published its deepest technical dive yet into the NVIDIA Rubin GPU, the silicon at the heart of the Vera Rubin platform. The headline is striking: up to 10x more agentic throughput per unit of energy than the Blackwell generation. For workloads where a single user request can trigger dozens of model calls, tool invocations, and memory lookups across multiple reasoning steps, this is not an incremental improvement — it is a generational reset.

Rubin is a 336-billion-transistor dual-die design unified by the NVIDIA High-Bandwidth Interface (NV-HBI). It packs 224 streaming multiprocessors, 896 Tensor Cores, and a third-generation Transformer Engine that delivers up to 50 petaflops of NVFP4 inference performance. The memory subsystem is equally ambitious: up to 288 GB of HBM4 driven by 12-Hi stacks, delivering 22 TB/s of peak bandwidth.

Why Agentic Workloads Need a Different GPU

Traditional inference is defined by a prompt and a response. Agentic workloads are defined by sustained reasoning across many steps, each with dependencies on the last. The GPU must therefore excel at:

  • Low per-step latency for interleaved tool calls and reasoning loops
  • High decode throughput across long context windows
  • Efficient KV cache capacity for multi-turn agent sessions
  • Scale-out model parallelism across tightly coupled GPU domains

Rubin addresses these directly. Its Tensor Memory Accelerator now supports inline descriptor updates, which reduces metadata-management overhead for Mixture-of-Experts (MoE) models as expert counts grow. The Tensor Core throughput per clock is doubled along the K dimension, meaning fewer loop iterations and better utilization at high tensor-parallel scale. For long-context attention — often the dominant cost in agentic workflows — Rubin combines activation sparsity with adaptive compression and improves softmax exponential throughput by 4x for FP16/BF16, unblocking the critical path.

Perhaps most importantly for production systems, Rubin introduces tile-level dependent kernel triggering. Instead of waiting for an entire producer kernel to finish before a consumer can start, consumer work begins as soon as the required activation data is available. This fills GPU timeline bubbles and is exactly the kind of fine-grained scheduling optimization that matters when agents are chaining dozens of dependent operations per request.

GB300 NVL72: Proving the Platform at Scale

While Rubin looks to the future, NVIDIA also published results showing the GB300 NVL72 already breaking records today. The system achieved 1,648 TFLOPs per GPU while pre-training DeepSeek-V3 671B — a world record for MoE pre-training throughput.

The significance is not the headline number alone. MoE models fundamentally shift the training bottleneck from compute to communication: each token is routed to a subset of experts, and the all-to-all dispatch sits in the critical path of every layer. GB300 NVL72 addresses this through fifth-generation NVLink, giving each GPU 1.8 TB/s of bandwidth with 130 TB/s of non-blocking all-to-all bandwidth across the rack. NVLink is memory-semantic, meaning a GPU reads and writes a peer’s HBM as native load/store operations over a lossless fabric — no software send path, no added latency.

For infrastructure operators, the implication is clear: rack-scale co-design is now the default. The era of assembling GPUs from different vendors and hoping the network keeps up is ending. Pre-training efficiency is measured in delivered FLOPs, not peak FLOPs, and delivered FLOPs require silicon, interconnect, networking, and software to be engineered as one system.

Together AI: $800 Million on the Economics of Open-Source Inference

While NVIDIA pushes the hardware frontier, Together AI is making a parallel bet on the economics of open-source model serving. The company announced $800 million in Series C funding from investors including NVIDIA, Aramco Ventures, and Salesforce Ventures, alongside commitments for over 500 MW of compute capacity.

Together’s thesis is that closed frontier APIs are becoming economically unsustainable at production scale. Companies like Decagon have reported 6x cost reductions after moving from proprietary APIs to Together’s open-model endpoints. Token volume through Together’s APIs has grown from 30 billion to more than 400 trillion tokens per month in nine months — a substantial share of which represents workloads migrated from closed providers.

Provisioned Throughput: A New Tier for Production

Together’s most significant recent product launch is Provisioned Throughput: reserved inference capacity for frontier open models, priced per token with a 99% uptime SLA. It sits between best-effort serverless and fully managed dedicated GPUs, offering the simplicity of a token-based API with the predictability of guaranteed capacity.

For MiniMax M3, one Provisioned Throughput Unit (PTU) delivers 138,840 input tokens per minute or 23,140 output tokens per minute, at roughly $0.36 per million input tokens — compared to $5 on Claude Opus 4.8. The savings reach up to 90% below proprietary APIs at list price. This is not a minor pricing arbitrage; it is a structural advantage that will reshape how enterprises budget for AI.

GPU Clusters That Fail Gracefully

Together also shipped meaningful reliability improvements to its GPU Clusters product: passive health checks that observe real workloads for GPU bus failures, thermal throttling, and Xid errors; auto node repair with human-in-the-loop remediation recommendations; and Slurm-on-Kubernetes 2.0 with self-healing worker daemons, zombie process reaping, and durable PVC-backed job accounting. These are the operational capabilities that separate a research cluster from a production platform.

Hugging Face + vLLM: Zero-Port Native Speed

On the framework side, Hugging Face announced a milestone that quietly changes how models reach production. The transformers modeling backend for vLLM now achieves native speed — or faster — compared to hand-written vLLM implementations for many architectures.

The key innovation is runtime graph optimization using torch.fx and Python AST manipulation. When a model is loaded with --model-impl transformers, vLLM performs static analysis on the model graph, identifies optimizable patterns, and rewrites operations in place. This enables fused operations mapped to vLLM’s optimized kernels (including Expert Parallelization for MoE models), parallel plan inference for tensor and pipeline parallelism, and full compatibility with torch.compile and CUDA Graphs.

The practical impact is enormous. Hugging Face supports 450+ architectures. Previously, getting maximum inference performance required a custom vLLM port for each new model — a bottleneck that slowed adoption. Now, any compliant transformers model can run at native vLLM speed with a single flag. Model authors write once for transformers; inference users get optimal performance automatically. This is the kind of composability that defines mature infrastructure.

What This Convergence Means

Taken together, these developments point to a coherent trend: AI infrastructure is maturing from research tooling to production substrate.

NVIDIA’s Rubin and GB300 NVL72 establish that the hardware layer must be co-designed for agentic workloads — not just bigger GPUs, but GPUs with attention-optimized sparsity, tile-level kernel scheduling, and rack-scale NVLink fabrics. Together AI proves that open-source model serving can be both higher-performance and dramatically cheaper than closed alternatives, with the operational reliability required for enterprise production. Hugging Face eliminates the porting bottleneck between research and optimized inference, letting the open ecosystem move at the speed of model releases.

The implications for platform engineers are immediate. Inference costs are no longer a fixed tax determined by a single provider; they are a variable cost that can be optimized across hardware, serving stack, and model selection. The choice between open and closed APIs is increasingly a choice between 6-20x cost multiples for equivalent capability. And the path from “model released on Hugging Face” to “serving at native vLLM speed on production hardware” no longer requires a custom engineering project.

The next phase of AI infrastructure will be defined not by who has the biggest cluster, but by who can deliver agentic intelligence at the lowest marginal cost per reasoning step. The pieces are falling into place.

Sources

Next signal