On August 11, NVIDIA shipped two releases that signal a shift in how AI agents are built. Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts model with only 3 billion active parameters per forward pass, was released as the latest entry in NVIDIA’s open model family. Alongside it came NeMo Switchyard, an open-source model routing library designed to intelligently route agent workloads across a pool of models instead of relying on a single frontier model for every call.
Together, the pair mark a maturing phase in agent infrastructure: the tools needed to build cost-effective, production-ready agent systems are starting to arrive in usable form, and the default architecture of “one big model handles everything” is beginning to look like an expensive mistake.
The Problem: One Model Is the Wrong Model for Most Tasks
Long-running AI agents make a lot of LLM calls. A single multi-step task can easily involve dozens of tool calls, file reads, reasoning steps, and subagent delegations. Historically, most teams have sent every single one of those calls to the same frontier model — usually the most capable one they can afford.
The result is predictable: high costs, high latency, and a lot of wasted capacity. A git-pull or file-read turn does not need the same reasoning horsepower as debugging a failing test or planning a multi-file refactor. Yet both are charged at the same per-token rate because both are hitting the same endpoint.
LangChain’s internal Deep Agents evaluation suite quantifies this bluntly. In their 145-task benchmark covering customer support, incident investigation, and multi-step workflow automation, the gap in accuracy between a frontier model like Claude Opus 4.8 and a smaller model like Nemotron 3.5 Lightning was only about eight percentage points. The gap in cost was closer to an order of magnitude. That is a lot of room for a router to work with.
Nemotron 3.5 Lightning: The Execution-Layer Model
Nemotron 3.5 Lightning is built for the execution layer of always-on agents. Its 30B MoE architecture activates just 3B parameters per token, giving it the capacity of a larger model at the compute cost of a small one. NVIDIA positions it as the model that handles git pulls, tool output validation, result formatting, and the routine calls that dominate any agent’s token budget.
Performance claims are measured against real agent tasks, not generic benchmarks. On PinchBench, which evaluates personal assistant–style task completion, Nemotron 3.5 Lightning hits 86% accuracy while completing 10,000 tasks 30% faster than Qwen3.6 35B at similar accuracy. NVIDIA also reports it sits on the accuracy-speed Pareto frontier on the Artificial Analysis Intelligence Index, a composite across agentic, coding, scientific reasoning, and general intelligence evaluations.
The model is not just fast — it is designed to be deployed anywhere. It runs on NVIDIA DGX Spark, GeForce RTX 5090, and Jetson devices, as well as standard data center GPUs. It ships with NVFP4 quantization alongside BF16, using the same low-precision kernels that power NVIDIA’s larger Nemotron models. NVIDIA also provides two speculative decoding draft models: DSpark (for lower-concurrency and edge workloads) and DFlash (for higher-concurrency data center scenarios), which can boost inference throughput further.
Crucially, Nemotron 3.5 Lightning is fully open. Weights, training data, and fine-tuning recipes are released under the OpenMDW-1.1 license, meaning teams can fine-tune, distill, or deploy without commercial restrictions. NVIDIA includes a specialized reinforcement learning dataset, Nemotron-RL Agentic Terminal Pivot, alongside LoRA and full SFT recipes using NeMo Automodel and NeMo Megatron Bridge.
NeMo Switchyard: The Router That Makes Pairing Practical
Having a small, fast model is only half the equation. The other half is deciding which model gets which call, and doing it in a way that does not require teams to rewrite their applications for every new model or provider.
NeMo Switchyard is NVIDIA’s answer: an open-source, provider-agnostic routing library that sits between an agent and its model endpoints. It accepts requests in OpenAI, Anthropic, or Responses API formats, evaluates each turn, and forwards it to the most appropriate model in a configured pool. Routing decisions can be stateful across an agent’s session, so affinity policies and escalation history carry forward without the application needing to track them.
Tuning-Free Routing
NeMo Switchyard ships with three routing strategies that require no workload-specific training:
- LLM Classifier: Uses a small judge model to classify each request and route it to the best candidate. It supports capability-based routing, escalation mode, and custom rules. Escalation is particularly useful for agents: every task starts on the cheaper model, and the judge monitors each turn. After a configurable number of consecutive bad turns, the session is promoted to the frontier model for the remainder of the task.
- Stage Router: Examines recent tool activity and error patterns to decide how much model capability is needed. A turn showing steady file writes and passing tests gets routed to the efficient model. Repeated errors or prolonged exploration trigger a promotion. Unlike the LLM classifier, it adds no extra model call and near-zero latency overhead.
- Escalation Router: An adaptive extension of the LLM classifier that monitors progress turn-by-turn. It is designed for multi-turn agent workloads where a smaller model handles routine work but needs support after repeated failures or loops.
Tunable Routing
For teams that want more precision, Switchyard also includes a tunable prefill router. During training, it extracts residual stream signals from the model’s prefill phase and maps them to predicted accuracy labels for each model in the pool. At inference time, the router scores each candidate model on expected accuracy versus cost and latency, then applies a configurable policy to select the best tradeoff. NVIDIA’s published benchmarks show that this approach can move accuracy-cost curves meaningfully for learned workloads.
Real Benchmarks: 74% Cost Reduction on Production Agent Tasks
LangChain’s benchmark is the most credible external validation published so far. They ran their Deep Agents evaluation suite — 145 multi-step tasks averaging 6.3 model calls each — through Switchyard’s escalation router, pairing Nemotron 3.5 Lightning as the weak target with Claude Opus 4.8 as the strong target.
The results are instructive. Running Opus 4.8 alone cost $11.45 per run with 86.0% accuracy. Routing between the two models cost $3.00 per run with 80.0% accuracy. That is a 74% cost reduction at the expense of six percentage points of accuracy. Nemotron 3.5 Lightning alone cost just $0.72 per run with 77.7% accuracy.
The distribution of calls is what matters. The lightweight model handled 93% of model calls. Opus received only 7%. Yet because of the price gap — roughly 87x per call between the two models — the frontier model still consumed 68.4% of spend in the routed setup. A small judge model (Gemini 3.1 Flash Lite) ate another 21.2%.
LangChain’s analysis notes an important caveat: this workload only had eight points of variance between models, which is a relatively narrow gap. The narrower the gap, the harder it is for routing to prove its value. In workloads where the cheap model is further behind, the savings story may look different — though the formula for whether routing is worth it is clear: the price gap between your models must be large enough to pay for the judge’s overhead.
Cognition also reported results on its FrontierCode benchmark. Using Switchyard’s stage router between Opus 5 and Kimi K2.7, their Devin Desktop implementation achieved 50.6% accuracy at $3.11 mean cost — within 2.8 points of the frontier-only baseline at approximately 28% lower cost.
Ecosystem Integration: Does It Fit Your Stack?
A routing library is only useful if it integrates cleanly with what developers already use. NeMo Switchyard’s architecture is deliberately provider-agnostic: model targets have semantic names, and the client behind each target maps those names to real endpoints and model IDs. That means swapping a model, moving it to a different provider, or adding a new region deployment changes only the client configuration, not the routing logic.
NVIDIA has announced integrations with LiteLLM (as a plug-in), Kong (for AI gateway and governance), and LangChain (via a Switchyard middleware). Partners including Cognition, Ramp, Nous Research, and Cadence (via its ChipStack AI Super Agent) are also integrating routing into their agent workflows. Cloud providers like Together AI, Baseten, Fireworks AI, and CoreWeave already host Nemotron 3.5 Lightning as a routing target.
For teams that want to self-host, Switchyard is fully open-source and runs as a standalone server that proxies common LLM APIs, or as in-process middleware via the switchyard-lib SDK. It also records routing decisions, token usage, latency, and outcomes, giving operators visibility into how traffic is actually flowing.
When Model Routing Makes Sense — and When It Does Not
Model routing is not free. The judge model adds latency and cost on every turn until escalation, and the accuracy tradeoff is real. LangChain’s analysis suggests three conditions where it is worth the complexity:
- Wide price gaps between models. If your cheap model costs nearly as much as your frontier model, the judge overhead may eat the savings.
- Multi-turn tasks. Escalation routing needs a trajectory to read. Single-turn workloads get no benefit from escalation logic.
- Known traffic composition. If most of your agent’s work is routine (tool calls, file reads, formatting) with occasional complex reasoning, the distribution favors routing. If every call is hard, there is nothing to save.
A practical formula LangChain proposes: routing pays off when the share of offloaded turns exceeds judge cost / (expensive cost − cheap cost). For their pairing, that bar was 5.9% of turns; they offloaded 93%, clearing it by a wide margin.
For teams who want minimum cost above all else and can accept the accuracy floor of the smaller model, running the cheap model alone is the simpler and cheaper option. Routing is for teams who need frontier capability on the hard requests but cannot tell in advance which ones those are.
The Bigger Picture: From Model Wars to System-of-Models Infrastructure
The significance of Nemotron 3.5 Lightning and NeMo Switchyard is not just the performance numbers. It is the architectural implication: the era of treating a single model as your entire inference layer is ending, and the era of model orchestration is beginning.
Frontier models will not disappear. They will remain essential for the hardest tasks, for safety-critical decisions, and for the edge cases where accuracy matters more than cost. But most agent workloads are not edge cases. They are composed of thousands of routine calls punctuated by occasional hard ones. Building infrastructure that recognizes that distinction and routes accordingly is the next logical step in AI platform maturity.
NVIDIA is positioning this as a system-of-models approach: frontier models handle orchestration and complex planning, while specialized efficient models handle the high-volume execution layer. It is a sensible split, particularly for teams running 24/7 agents or coding copilots that generate thousands of LLM calls per day.
The tools are now available in open source, with real benchmarks, real partner integrations, and a clear adoption path. Whether the cost savings hold up under your specific workload is a question only your own data can answer. But the direction is clear: if you are running AI agents in production and still sending every call to the same expensive API, you are probably paying too much.
