The Open-Source Agent Stack: What Meta, NVIDIA, and vLLM Shipped This Week for Local AI Agents

Agentic AI is moving off the cloud and onto local hardware faster than most observers expected. The week of August 10, 2026 delivered a coordinated wave of open-weight models, inference engine updates, and edge tooling that makes running a 24/7 agent fleet on your own infrastructure practical — not theoretical. Here is what shipped and why it matters for operators.

Muse Glimmer: The First Model Built for Local Agents

On August 10, Meta released Muse Glimmer, a 30B parameter dense model distilled from its larger Muse family and released under the Apache 2.0 license. Unlike previous open-weight releases that happened to support agent use cases, Glimmer was designed for them from the ground up.

The specs tell the story: a 120K+ context window, a 2B ViT-style perception encoder paired with a 28B text decoder, and hybrid attention that alternates between local and global patterns to keep memory use manageable at long context lengths. An optional DFlash speculative decoding drafter trades a small memory overhead for significantly faster structured generation — the JSON schemas and function-call payloads that agents emit constantly.

Benchmarks confirm the design choices. SWE-Bench Pro at 51.2 puts Glimmer in the same league as much larger closed models. TerminalBench 2.1 (51.7) and SciCode (43.6) show it handles the tool-use and code-generation loops that define agentic work. Multimodal scores — OmniDocBench v1.5 at 75.8 and ScreenSpot Pro at 75.4 — mean agents can read screens and documents without calling out to a separate vision API.

The Day-Zero Ecosystem Effect

What makes Glimmer an infrastructure event is not the model weights but the immediate toolchain support. Hugging Face shipped native transformers integration at release. vLLM, llama.cpp, and Ollama all added support within hours. Ollama’s MLX backend delivers state-of-the-art Apple Silicon performance with DFlash and image input enabled. This is the first time a major model release has been simultaneously available across every significant local inference stack.

For operators, this means a single model choice can power agents on MacBooks (Ollama/MLX), Linux servers (vLLM), mobile and edge devices (llama.cpp), and cloud endpoints (Hugging Face Inference Endpoints) without re-architecting the application layer.

NVIDIA’s Edge-to-Cloud Agent Tooling

Nemotron 3.5 Lightning: A 3B Active MoE for Always-On Agents

NVIDIA’s Nemotron 3.5 Lightning takes a different approach: a 30B mixture-of-experts model with only 3B active parameters, optimized for the high-frequency, low-latency work that agents do repeatedly — tool validation, result parsing, and subagent dispatch. It is designed for harnesses like OpenClaw and Hermes Agent and ships with NemoClaw, an open-source security and lifecycle management stack for long-running agents.

Ollama added Nemotron 3.5 Lightning in v0.32.9 (August 11), making it available for local execution alongside Muse Glimmer. The pairing — Glimmer for complex reasoning and planning, Nemotron Lightning for fast execution — is already being discussed in deployment forums as a practical split for resource-constrained environments.

NeMo Switchyard: Model Routing as Infrastructure

NVIDIA’s NeMo Switchyard addresses a problem every multi-agent system eventually hits: no single model is optimal for every step. Switchyard is a routing layer that dispatches agent workloads across available models based on task type, cost targets, and real-time latency. It treats model selection as an automated infrastructure decision rather than a hardcoded config value.

JetPack 7.2.1 and Vera Storage for Edge Fleets

For robotics and industrial deployments, JetPack 7.2.1 adds agentic video skills and T3000 GPU emulation to Jetson. The accompanying Vera storage benchmarks show faster encryption, compression, and recovery for AI-native storage — critical for edge agents that cache model weights and locally store interaction logs.

vLLM v0.27.0: The Inference Engine Goes Massive

The vLLM v0.27.0 release on August 10 is the project’s largest to date: 561 commits from 242 contributors, 64 of them new. The headline feature is full Kimi K3 support — the first open 3T-class parameter model — delivered across core kernels, Python and Rust frontends, and AttnRes attention implementations in a single coordinated drop.

vLLM remains the default inference engine for production LLM serving. The speed at which a model of Kimi K3’s scale and architectural novelty achieved native vLLM support is a signal: the open-source inference ecosystem is now mature enough to absorb frontier-class models without months of porting work.

Together AI: Scheduling and Scaling for Agent Workloads

ThunderAgent: Program-Aware Inference Scheduling

Together AI’s ThunderAgent, published in late July, tackles a scheduling problem that traditional inference engines ignore. Standard batching treats every LLM request as independent. Agent workflows violate that assumption: an agent emits a tool call, waits for a result, receives it, and continues — generating intermediate KV cache state that is discarded if the scheduler does not understand the program boundary.

ThunderAgent introduces a program abstraction for agentic LLM requests. By treating each agent workflow as a schedulable program rather than a sequence of unrelated jobs, it eliminates KV cache thrashing. The measured results are substantial: more than 2x single-node throughput, roughly 10x lower P50 latency at high concurrency, and 2.4x speedup on an 8-node cluster with near-linear scaling behavior.

Autoscaling on Metrics That Matter

Together AI’s separate autoscaling work addresses the replica-warmup problem. GPU utilization can read healthy while queues back up and new replicas take minutes to become productive. Their dedicated inference platform now scales on engine-native metrics: in-flight requests, time-to-first-token (TTFT), GPU utilization, and token throughput.

Users set replica bounds, choose a target metric, and tune scale-up and scale-down windows. In Together’s replay experiments, queue-depth-based scaling maintained stable latency where GPU-utilization-based scaling spiked. For agentic workloads with bursty, multi-step traffic, this distinction is the difference between a responsive fleet and a degraded one.

IBM Research: Shrinking the Agent Memory Bill

While hardware and scheduling layers advanced, IBM Research published ALTK-Evolve, a system that lets agents learn from their own trajectories using fewer tokens than the comparable ACE (Agentic Context Engineering) approach.

Both systems convert an agent’s failure history into reusable lessons without fine-tuning or human labels. The architectural difference is in delivery: ACE consolidates everything into a single evolving playbook, while ALTK-Evolve clusters near-duplicate lessons and merges them support-conservatively. The result is a smaller memory store that preserves the record of how much experience backs each guideline. For agents burning tokens on every context window refresh, this is a direct cost reduction.

The Stack in Practice: What to Deploy Now

The convergence of these releases creates a coherent local-first agent stack for the first time:

Model layer: Muse Glimmer (30B dense, multimodal) or Nemotron 3.5 Lightning (30B MoE, 3B active) depending on whether reasoning depth or execution speed matters more.
Inference layer: vLLM for server-scale serving, Ollama for local and Apple Silicon deployment, llama.cpp for mobile and edge.
Scheduling layer: ThunderAgent or custom program-aware schedulers for KV-cache-efficient agent pipelines.
Routing layer: NeMo Switchyard or equivalent multi-model routers for cost-aware dispatch.
Memory layer: ALTK-Evolve-style trajectory learning to keep context windows small and token bills manageable.
Edge layer: JetPack 7.2.1 and Vera storage for robotics and industrial agents that need local video processing and secure caching.

This is not a vendor roadmap. It is a set of independently shipped, interoperable tools that happen to fit together. The open-source inference engines — vLLM, Ollama, llama.cpp — are the glue. The fact that Muse Glimmer and Nemotron 3.5 Lightning both achieved day-zero support across all three is the signal that the local agent stack has crossed from experimental to operational.