# NVIDIA Rubin, Together AI’s $800M Bet, and Hugging Face’s Native vLLM Speed: The Infrastructure Convergence Reshaping AI

- Link: https://thestackobserver.com/nvidia-rubin-together-ais-800m-bet-and-hugging-faces-native-vllm-speed-the-infrastructure-convergence-reshaping-ai/
- Published: 2026-07-22T08:33:13-04:00
- Author: Stackxx

The AI infrastructure landscape is undergoing its most significant transformation
since the first GPU clusters were dedicated to model training. We are not merely
adding more GPUs — we are redesigning the data center itself as a single unit of
compute purpose-built for **agentic AI**. This week, three major threads converged:
NVIDIA unveiled the Rubin GPU architecture and set a world record for MoE pre-training;
Together AI raised $800 million to scale open-source inference infrastructure; and
Hugging Face made it possible for any model on its Hub to run at native vLLM speeds
without a custom port.

## The Rubin Architecture: A GPU Purpose-Built for Agentic AI

NVIDIA published its deepest technical dive yet into the **NVIDIA Rubin GPU**, the
silicon at the heart of the Vera Rubin platform. The headline is striking: up to**
10x more agentic throughput per unit of energy** than the Blackwell generation. 
For workloads where a single user request can trigger dozens of model calls, tool
invocations, and memory lookups across multiple reasoning steps, this is not an 
incremental improvement — it is a generational reset.

Rubin is a 336-billion-transistor dual-die design unified by the NVIDIA High-Bandwidth
Interface (NV-HBI). It packs 224 streaming multiprocessors, 896 Tensor Cores, and
a third-generation Transformer Engine that delivers up to **50 petaflops of NVFP4
inference performance**. The memory subsystem is equally ambitious: up to 288 GB
of HBM4 driven by 12-Hi stacks, delivering **22 TB/s** of peak bandwidth.

### Why Agentic Workloads Need a Different GPU

Traditional inference is defined by a prompt and a response. Agentic workloads are
defined by sustained reasoning across many steps, each with dependencies on the 
last. The GPU must therefore excel at:

 * Low per-step latency for interleaved tool calls and reasoning loops
 * High decode throughput across long context windows
 * Efficient KV cache capacity for multi-turn agent sessions
 * Scale-out model parallelism across tightly coupled GPU domains

Rubin addresses these directly. Its Tensor Memory Accelerator now supports **inline
descriptor updates**, which reduces metadata-management overhead for Mixture-of-
Experts (MoE) models as expert counts grow. The Tensor Core throughput per clock
is doubled along the K dimension, meaning fewer loop iterations and better utilization
at high tensor-parallel scale. For long-context attention — often the dominant cost
in agentic workflows — Rubin combines **activation sparsity with adaptive compression**
and improves softmax exponential throughput by 4x for FP16/BF16, unblocking the 
critical path.

Perhaps most importantly for production systems, Rubin introduces **tile-level dependent
kernel triggering**. Instead of waiting for an entire producer kernel to finish 
before a consumer can start, consumer work begins as soon as the required activation
data is available. This fills GPU timeline bubbles and is exactly the kind of fine-
grained scheduling optimization that matters when agents are chaining dozens of 
dependent operations per request.

## GB300 NVL72: Proving the Platform at Scale

While Rubin looks to the future, NVIDIA also published results showing the **GB300
NVL72** already breaking records today. The system achieved **1,648 TFLOPs per GPU**
while pre-training DeepSeek-V3 671B — a world record for MoE pre-training throughput.

The significance is not the headline number alone. MoE models fundamentally shift
the training bottleneck from compute to communication: each token is routed to a
subset of experts, and the all-to-all dispatch sits in the critical path of every
layer. GB300 NVL72 addresses this through fifth-generation NVLink, giving each GPU**
1.8 TB/s** of bandwidth with **130 TB/s of non-blocking all-to-all bandwidth** across
the rack. NVLink is memory-semantic, meaning a GPU reads and writes a peer’s HBM
as native load/store operations over a lossless fabric — no software send path, 
no added latency.

For infrastructure operators, the implication is clear: rack-scale co-design is 
now the default. The era of assembling GPUs from different vendors and hoping the
network keeps up is ending. Pre-training efficiency is measured in delivered FLOPs,
not peak FLOPs, and delivered FLOPs require silicon, interconnect, networking, and
software to be engineered as one system.

## Together AI: $800 Million on the Economics of Open-Source Inference

While NVIDIA pushes the hardware frontier, **Together AI** is making a parallel 
bet on the economics of open-source model serving. The company announced **$800 
million in Series C funding** from investors including NVIDIA, Aramco Ventures, 
and Salesforce Ventures, alongside commitments for over **500 MW of compute capacity**.

Together’s thesis is that closed frontier APIs are becoming economically unsustainable
at production scale. Companies like Decagon have reported **6x cost reductions**
after moving from proprietary APIs to Together’s open-model endpoints. Token volume
through Together’s APIs has grown from 30 billion to more than **400 trillion tokens
per month** in nine months — a substantial share of which represents workloads migrated
from closed providers.

### Provisioned Throughput: A New Tier for Production

Together’s most significant recent product launch is **Provisioned Throughput**:
reserved inference capacity for frontier open models, priced per token with a **
99% uptime SLA**. It sits between best-effort serverless and fully managed dedicated
GPUs, offering the simplicity of a token-based API with the predictability of guaranteed
capacity.

For MiniMax M3, one Provisioned Throughput Unit (PTU) delivers 138,840 input tokens
per minute or 23,140 output tokens per minute, at roughly **$0.36 per million input
tokens** — compared to $5 on Claude Opus 4.8. The savings reach **up to 90% below
proprietary APIs** at list price. This is not a minor pricing arbitrage; it is a
structural advantage that will reshape how enterprises budget for AI.

### GPU Clusters That Fail Gracefully

Together also shipped meaningful reliability improvements to its GPU Clusters product:**
passive health checks** that observe real workloads for GPU bus failures, thermal
throttling, and Xid errors; **auto node repair** with human-in-the-loop remediation
recommendations; and **Slurm-on-Kubernetes 2.0** with self-healing worker daemons,
zombie process reaping, and durable PVC-backed job accounting. These are the operational
capabilities that separate a research cluster from a production platform.

## Hugging Face + vLLM: Zero-Port Native Speed

On the framework side, Hugging Face announced a milestone that quietly changes how
models reach production. The **transformers modeling backend for vLLM** now achieves
native speed — or faster — compared to hand-written vLLM implementations for many
architectures.

The key innovation is runtime graph optimization using `torch.fx` and Python AST
manipulation. When a model is loaded with `--model-impl transformers`, vLLM performs
static analysis on the model graph, identifies optimizable patterns, and rewrites
operations in place. This enables fused operations mapped to vLLM’s optimized kernels(
including Expert Parallelization for MoE models), parallel plan inference for tensor
and pipeline parallelism, and full compatibility with `torch.compile` and CUDA Graphs.

The practical impact is enormous. Hugging Face supports **450+ architectures**. 
Previously, getting maximum inference performance required a custom vLLM port for
each new model — a bottleneck that slowed adoption. Now, any compliant transformers
model can run at native vLLM speed with a single flag. Model authors write once 
for transformers; inference users get optimal performance automatically. This is
the kind of composability that defines mature infrastructure.

## What This Convergence Means

Taken together, these developments point to a coherent trend: **AI infrastructure
is maturing from research tooling to production substrate**.

NVIDIA’s Rubin and GB300 NVL72 establish that the hardware layer must be co-designed
for agentic workloads — not just bigger GPUs, but GPUs with attention-optimized 
sparsity, tile-level kernel scheduling, and rack-scale NVLink fabrics. Together 
AI proves that open-source model serving can be both higher-performance and dramatically
cheaper than closed alternatives, with the operational reliability required for 
enterprise production. Hugging Face eliminates the porting bottleneck between research
and optimized inference, letting the open ecosystem move at the speed of model releases.

The implications for platform engineers are immediate. Inference costs are no longer
a fixed tax determined by a single provider; they are a variable cost that can be
optimized across hardware, serving stack, and model selection. The choice between
open and closed APIs is increasingly a choice between 6-20x cost multiples for equivalent
capability. And the path from “model released on Hugging Face” to “serving at native
vLLM speed on production hardware” no longer requires a custom engineering project.

The next phase of AI infrastructure will be defined not by who has the biggest cluster,
but by who can deliver agentic intelligence at the lowest marginal cost per reasoning
step. The pieces are falling into place.

## Sources

 * [Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI](https://developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/)—
   NVIDIA Developer Blog, July 21, 2026
 * [Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72](https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/)—
   NVIDIA Developer Blog, July 20, 2026
 * [Announcing our $800M Series C to accelerate the shift to open-source AI](https://www.together.ai/blog/announcing-our-series-c)—
   Together AI Blog, July 1, 2026
 * [Open, convenient and predictable: Introducing Provisioned Throughput](https://www.together.ai/blog/provisioned-throughput)—
   Together AI Blog, July 8, 2026
 * [New in Together GPU Clusters: Reliability and control for production GPU clusters](https://www.together.ai/blog/new-in-together-gpu-clusters-reliability-and-control-for-production-gpu-clusters)—
   Together AI Blog, July 15, 2026
 * [Native-speed vLLM transformers modeling backend](https://huggingface.co/blog/native-speed-vllm-transformers-backend)—
   Hugging Face Blog, July 8, 2026