AI Infrastructure Roundup: Together AI Raises $800M, vLLM Hits Native Speed, and NVIDIA BlueField Targets Agentic Factories

The AI infrastructure landscape is shifting faster than ever. Between a blockbuster $800 million Series C for open-source inference, new DPU architectures purpose-built for agentic AI, and inference engines achieving native-speed performance without custom implementations, the ground beneath production AI systems is being re-poured in real time.

Here is what matters right now in AI infrastructure, inference engines, and the plumbing that makes modern AI agents possible.

The Economics of Open-Source AI Just Got Real

On July 1, 2026, Together AI announced an $800 million Series C, bringing its total funding past $1 billion. The company, which operates a full-stack platform for open-source AI inference, fine-tuning, and GPU clusters, is making a blunt bet: the economics of closed models do not scale, and open weights will win on cost, transparency, and flexibility.

Together AI claims 40+ models chosen for production by customers including Cursor, Eleven Labs, Cognition, and Decagon. Its inference engine has been benchmarked at 31% higher throughput than the next-fastest open-source engine on coding agent workloads, with 2× better time-to-first-token (TTFT) at saturation and 76% lower cost than Claude Opus 4.6 on comparable tasks. These numbers matter because coding agents are rapidly becoming the defining workload for inference infrastructure: multi-step, tool-using, context-heavy, and latency-sensitive.

The funding also coincides with Together AI making on-demand NVIDIA B200 GPUs available through its GPU Clusters product, alongside reliability improvements including passive health checks, node repair, and stronger Slurm integration. For enterprises running their own training or fine-tuning pipelines, access to Blackwell-class GPUs without long-term contracts is a meaningful unlock.

NVIDIA BlueField: Infrastructure Silicon for the Agentic Era

While Together AI is building at the platform layer, NVIDIA is re-architecting the silicon underneath. A recent NVIDIA technical blog detailed how BlueField DPUs and Vera BlueField-4 STX storage processors are being positioned as the infrastructure backbone for “agentic AI factories.”

The argument is straightforward but significant: agentic AI changes the infrastructure pattern. A single user request can trigger dozens of model calls, tool calls, memory lookups, policy checks, storage accesses, and network transfers before a final answer is produced. That means data movement, not just compute, becomes the bottleneck.

BlueField offloads networking, storage, security, telemetry, and control-plane services from host CPUs, while DOCA provides the programmable software layer. The company claims this co-design delivers higher GPU utilization, more predictable latency, stronger tenant isolation, lower cost per token, and more tokens per watt. In the upcoming Vera Rubin platform, BlueField sits alongside the GPU compute as a dedicated infrastructure processing layer.

For anyone building multi-tenant AI platforms or running agents at scale, this is a signal that infrastructure offload is moving from optimization to requirement.

vLLM Achieves Native Speed Without Custom Code

On the inference engine front, vLLM’s transformers modeling backend just crossed an important threshold. As of vLLM 0.25.x, the backend is now as fast—or faster—than vLLM’s hand-written native implementations for many LLM architectures.

Hugging Face and the vLLM team benchmarked the backend across three Qwen3 variants: a 4B dense model, a 32B dense model on tensor parallelism, and a 235B-parameter FP8 Mixture-of-Experts on data + expert parallelism across eight H100s. The transformers backend met or exceeded native throughput on every configuration.

The practical implication is enormous. Model authors no longer need to port their architectures to vLLM manually. A single flag, --model-impl transformers, composes with existing parallelism options and unlocks vLLM’s continuous batching and custom attention kernels. With support for 450+ architectures in the transformers library, this removes a major friction point in the “release model → get fast inference” pipeline.

vLLM itself is on a rapid release cadence—v0.25.1 shipped July 14 with bug fixes around TorchCodec and model loading. Ollama v0.32.1 also shipped recently with improved Gemma 4 tool calling, MLX cache leak fixes, and better multi-turn reasoning. Meanwhile, llama.cpp continues its near-daily build releases (b10057 at time of writing), and LiteLLM hit v1.94.0-dev.3 with Docker image signing via cosign and expanded model routing capabilities.

Embedding Models Go Agentic: NVIDIA Nemotron 3 Embed

Retrieval is the quiet bottleneck in multi-step agentic workflows. A bad retrieval step causes agents to fetch irrelevant context, re-query, waste token budget, and pollute downstream reasoning. NVIDIA’s answer is Nemotron 3 Embed, a family of embedding models that currently ranks #1 on the RTEB (Retrieval Evaluation Benchmark) leaderboard.

The family includes three variants: an 8B flagship for precision-critical RAG, a 1B high-efficiency model for production latency-sensitive serving, and a 1B NVFP4 Blackwell-optimized variant for ultra-high-throughput deployments. All support query understanding, multilingual retrieval, and cross-encoder reranking integration.

What makes this infrastructure-relevant is that retrieval is no longer a preprocessing step—it is a real-time, latency-bound service that agents invoke repeatedly during execution. Having embedding models that can run on a spectrum from single-GPU edge deployments to multi-node high-throughput clusters gives architects more flexibility in how they design agent memory and context systems.

GPT-5.6 and the Infrastructure Pressure Test

OpenAI launched the GPT-5.6 family on July 9, introducing three tiers: Sol (frontier), Terra (balanced), and Luna (cost-efficient). The models are already rolling into Microsoft 365 Copilot across Word, Excel, PowerPoint, and the new Cowork application.

From an infrastructure perspective, the key claim is efficiency: GPT-5.6 Sol achieves better results on the Artificial Analysis Coding Agent Index with 54% fewer output tokens and 57% less time per task than predecessor models. OpenAI also introduced “ultra” mode, which coordinates multiple agents across parallel workstreams for the most demanding tasks.

For inference providers, this sets a competitive bar. If frontier models can deliver more useful work per token, then infrastructure must optimize for end-to-end task completion time—not just raw tokens per second. That shifts the benchmarking conversation from throughput to task-level efficiency, which is harder to game and more relevant to users.

Mistral Medium 3.5 and Cloud-Native Coding Agents

Mistral is also pushing agent infrastructure forward with Mistral Medium 3.5, a 128B open-weight model with a 256k context window that merges instruction-following, reasoning, and coding into a single dense model. Released under a modified MIT license, it is priced at $1.5 per million input tokens and $7.5 per million output tokens.

More interesting than the model itself is the delivery mechanism: remote agents in Mistral Vibe. Coding agents now run in the cloud asynchronously, can be spawned from the CLI or Le Chat, and notify users when complete. A local CLI session can even be “teleported” up to the cloud mid-session. This is a fundamentally different operational model from local agent loops, and it places new demands on session management, state persistence, and remote execution infrastructure.

Mistral also introduced Work mode in Le Chat, a multi-step agent for research and cross-tool workflows powered by Medium 3.5. For infrastructure teams, the pattern is clear: agents are becoming long-running, stateful, remote services—not stateless API calls.

Apache Spark 4.2 and the Data Layer for AI

Infrastructure is not just about GPUs and inference. The data layer matters too. Apache Spark 4.2 shipped with features explicitly designed for AI-native workloads: metric views for governed business semantics, vector functions and NEAREST BY for retrieval, Arrow-first Python paths for agent invocation, and change data capture for keeping context fresh.

Databricks, which co-led the Spark 4.2 release, is also pushing a “unified context” narrative—positioning the lakehouse as the missing layer for enterprise AI coworkers. The company introduced a context engineer certification and training programs, acknowledging a widening skills gap in the agentic era.

What This Means for Infrastructure Teams

Several clear takeaways emerge from this month’s news cycle:

Inference efficiency is now the primary battleground. Open-source engines (vLLM, Together, TensorRT-LLM) are competing on real-world task throughput, not synthetic benchmarks. Providers that optimize for agent workloads—multi-turn, tool-heavy, long-context—will win.
Agentic AI requires rethinking the full stack. From DPUs that offload infrastructure work to embedding models that power real-time retrieval to remote execution runtimes that maintain session state, the infrastructure surface area is expanding.
Open weights are getting serious capital. Together AI’s $800M round is a vote of confidence that open-source models, served on efficient infrastructure, can compete with closed APIs on both performance and economics.
Context and memory are moving to the center. Whether through vector functions in Spark, BlueField-accelerated storage, or 256k-context models, the ability to hold, search, and update large working memory is becoming a core infrastructure capability.

The next six months will likely see this infrastructure stack converge: inference engines, agent runtimes, embedding services, and data platforms merging into unified platforms that can deploy, scale, and observe agentic workloads end-to-end. The pieces are already on the table. Someone just needs to assemble them.