# AI Infrastructure Update: vLLM 0.23, Ollama MLX, and the Rise of Sovereign Models

- Link: https://thestackobserver.com/ai-infrastructure-update-vllm-0-23-ollama-mlx-and-the-rise-of-sovereign-models/
- Published: 2026-06-19T09:06:30-04:00
- Author: Stackxx

The AI infrastructure layer is where the rubber meets the road. Model weights mean
nothing if you cannot serve them efficiently, route traffic intelligently, or keep
a KV cache warm across thousands of concurrent users. June 2026 has delivered a 
barrage of releases across the stack — from the inference engine itself to the gateways,
local runners, and hardware benchmarks that determine what is actually possible 
in production.

## vLLM 0.23.0: The Inference Engine Keeps Accelerating

The vLLM project shipped **v0.23.0** this week with 408 commits from 200 contributors,
63 of them first-time contributors. That scale of contribution alone signals how
central vLLM has become to the open-source serving ecosystem. But the headline changes
are what matter for operators.

### DeepSeek-V4 Hardens Across Backends

DeepSeek-V4, introduced in v0.22.0, received a massive optimization and hardening
pass. Its sparse MLA metadata is now decoupled from DeepSeek-V3.2, it gained a TRTLLM-
gen attention kernel, EPLB support for the Mega-MoE architecture, selective prefix-
cache retention for sliding-window KV cache, and an index-share feature for DSA 
MTP. The model was also detached from `torch.compile`, with its attention and RoPE
paths refactored for cleaner execution. An XPU attention decode path was added for
Intel accelerator support. If you are running DeepSeek-V4 in production, this release
is a mandatory upgrade.

### Model Runner V2 Expands to Llama and Mistral

MRv2 is now selected by default for **Llama and Mistral dense models** in addition
to Qwen3, bringing its FlashInfer sampler, breakable CUDA graphs, pipeline-parallel
bubble elimination, and kernel block-size support for hybrid models to a broader
set of deployments. Gemma 4 MTP support also landed in MRv2. The unified parser 
unifies reasoning and tool-call parsing behind a single `Parser.parse()` interface,
with the Responses parser migrated to match.

### Multi-Tier KV Cache Offloading

The offloading framework gained an **object-store secondary tier**, enabling KV 
cache pages to spill to S3-compatible storage when GPU and host memory are exhausted.
HMA is now enabled by default for capable connectors, and per-request offloading
policy is available via the `on_new_request` lifecycle hook. For high-throughput
serving with long contexts, this changes the cost calculus dramatically.

### New Model Support

v0.23.0 adds Step-3.7-Flash, Cosmos3 Reasoner, Gemma 4 Unified (encoder-free), JetBrains
Mellum v2, Granite Speech Plus, and Cohere Mini Code. Transformers v5 compatibility
is now targeted, with vendored MiniCPM-V/O processors and fixes for Sarvam and Voxtral.

## Ollama 0.30.10: Local Inference Gets Smarter

While vLLM dominates datacenter serving, **Ollama** continues to own the local and
edge inference story. Version 0.30.10 brings **Command A and North family models
to Apple Silicon via the MLX engine**, updated the underlying llama.cpp engine to
build 9672, and added support for the Cohere2Moe architecture. Context shifting 
now allows shiftable prompts, fixing cases where `ollama launch claude` and other
coding-agent workflows would emit only a single token.

The MLX integration for Command A is particularly notable. Cohere’s Command A family
is a 111-billion-parameter model that activates only 24 billion per prompt, making
it practical to run on as few as two H100s — or now, on a high-end Mac Studio. Ollama’s
ability to bridge datacenter-grade models to local hardware is narrowing the gap
between “enterprise model” and “laptop model” faster than most expected.

## LiteLLM 1.89.2: The Gateway Layer Matures

**LiteLLM** shipped v1.89.2 with a significant operational improvement: **all Docker
images are now signed with cosign**. Every release is cryptographically verifiable
against a pinned public key, with both commit-hash and tag-based verification workflows
documented. For teams running LiteLLM as their production LLM gateway — routing 
requests across OpenAI, Anthropic, Azure, and self-hosted vLLM backends — this is
a baseline security requirement that was previously missing.

The project continues its aggressive release cadence, with backported fixes landing
on both stable/1.89.x and stable/1.88.x branches. As the de facto standard for unified
LLM API access, LiteLLM’s stability and security posture directly impact how enterprises
think about multi-provider inference.

## Cohere Command A+: Sovereign AI for Regulated Industries

Cohere released **Command A+**, an open-source enterprise model built for sovereign
deployments. With **111 billion parameters and 24 billion active per prompt**, it
is designed for complex RAG pipelines, multi-step SQL generation, and high-stakes
financial document analysis — all while fitting on minimal hardware. The model is
fully multimodal, supporting charts, PDFs, slides, and complex reports alongside
text.

Released under **Apache 2.0** with fully open weights, Command A+ supports 48 languages
including all official EU languages. Cohere is explicitly positioning it for regulated
industries and governments that need data sovereignty: on-premises deployment, no
external data transmission, and zero vendor lock-in. The company also announced 
strategic MOUs with Indra Group and Multiverse Computing to advance sovereign AI
across Spain, Canada, and Europe, and acquired Reliant AI to expand into biopharma
and healthcare verticals.

For infrastructure operators, Command A+ represents a new category: _frontier-capable
models that do not require frontier hardware or cloud dependencies_. That changes
deployment topology decisions for banks, defense agencies, and healthcare systems.

## Google I/O 2026: The Agentic Gemini Era

Google I/O 2026 was a firehose of AI announcements, but the infrastructure-relevant
ones are worth isolating. **Gemini 3.5** was released as a series combining frontier
intelligence with action capabilities — read, it is designed for agentic workflows,
not just chat. **Gemini Omni** adds native multimodal reasoning across text, image,
audio, and video in a single model call.

On the serving side, Google announced a **$100 AI Ultra subscription plan** and 
expanded AI Mode in Search, which has now shifted user behavior from keyword queries
to natural language across the U.S. For developers, Google AI Studio now supports“
vibe coding” experiments — rapid prototyping with Gemini-powered assistance. The
infrastructure implication is clear: Google is betting that agentic AI will drive
compute demand far beyond current chatbot workloads, and it is pricing its stack
accordingly.

## Hugging Face: Open Models, Open Benchmarks, Open Agents

The Hugging Face ecosystem delivered several infrastructure-relevant pieces this
month:

 * **“Beyond LoRA”** — A research investigation into whether alternative fine-tuning
   techniques can outperform the de facto standard, with implications for how adapters
   are served at scale.
 * **“Is it agentic enough?”** — A benchmarking framework for evaluating open models
   on real tooling and agentic tasks, filling a critical gap between academic benchmarks
   and production behavior.
 * **GLM-5.2** — Zhipu AI’s latest model, built for long-horizon tasks with extended
   context and reasoning capabilities.
 * **Agentic Resource Discovery** — A new Hugging Face feature letting agents search
   the Hub autonomously for models, datasets, and spaces.
 * **Nemotron 3.5 Content Safety** — NVIDIA’s multimodal safety model for enterprise
   AI, hosted on the Hub and ready for integration into serving pipelines.

The common thread: the open ecosystem is not just releasing models, it is building
the _tooling to evaluate, serve, and secure them_ — which is where infrastructure
value actually lives.

## NVIDIA Blackwell Tops MLPerf Training 6.0

On the hardware front, **NVIDIA Blackwell** topped the MLPerf Training 6.0 benchmarks
with industry-leading scale and performance. Blackwell is now the dominant training
accelerator, and its inference variants are beginning to ship into cloud and on-
premises deployments. For vLLM, Ollama, and Triton Inference Server users, Blackwell’s
arrival means higher throughput per watt and new kernel optimization targets. The
hardware-software co-evolution cycle continues.

## OpenClaw: Agent Tooling Infrastructure

OpenClaw shipped several infrastructure-facing updates in June. The **Skill Workshop**
now allows agents to review, revise, and apply proposed skills before they change
runtime behavior — a governance layer for agentic systems. A collaboration with **
NVIDIA** brought SkillSpector scanning to every ClawHub skill, checking for hidden
instructions and agentic risks. **Auto Mode for exec approvals** adds opt-in automated
guardrails for low-risk operations, with human review for anything that misses policy.

These are not user-facing features. They are _infrastructure for running autonomous
agents safely_ — and as agentic AI moves from demo to production, that infrastructure
is becoming as critical as the inference engine itself.

## What This Means for Operators

June 2026 is a snapshot of an infrastructure layer in rapid motion:

 * **Inference engines** are adding model support faster than ever, with vLLM’s 
   408-commit release and Ollama’s MLX bridging showing two paths: datacenter scale
   and local edge.
 * **Gateway layers** are maturing on security (LiteLLM cosign) and capability (
   unified routing across providers).
 * **Models themselves** are becoming deployable on smaller footprints (Command 
   A+ on 2x H100 or Apple Silicon), collapsing the old “frontier vs. practical” 
   divide.
 * **Hardware** (Blackwell) and **software** (vLLM MRv2, Ollama MLX) are co-evolving,
   with each release targeting the other.
 * **Agent infrastructure** (OpenClaw skills, Hugging Face agentic discovery) is
   emerging as a distinct layer above raw inference.

The practical takeaway: if you are building AI infrastructure today, you are not
just choosing a model and a GPU. You are choosing a serving stack, a gateway, a 
security posture, a hardware generation, and increasingly, an agent runtime. The
winners will be the teams that integrate across all five — not the ones that optimize
any single layer in isolation.

## Sources

 * [vLLM v0.23.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.23.0)
 * [Ollama v0.30.10 Release Notes](https://github.com/ollama/ollama/releases/tag/v0.30.10)
 * [LiteLLM v1.89.2 Release Notes](https://github.com/BerriAI/litellm/releases/tag/v1.89.2)
 * [Cohere Command A+ Announcement](https://cohere.com/blog/cohere-releases-command-a-plus)
 * [Google AI Updates May 2026](https://blog.google/innovation-and-ai/technology/ai/google-ai-updates-may-2026/)
 * [Gemini 3.5 Announcement](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/)
 * [Hugging Face: Beyond LoRA](https://huggingface.co/blog/peft-beyond-lora)
 * [Hugging Face: Is it agentic enough?](https://huggingface.co/blog/is-it-agentic-enough)
 * [Hugging Face: GLM-5.2](https://huggingface.co/blog/zai-org/glm-52-blog)
 * [NVIDIA Blackwell MLPerf Training 6.0](https://developer.nvidia.com/blog/nvidia-blackwell-tops-mlperf-training-6-0-with-industry-leading-scale-and-performance/)
 * [OpenClaw Skill Workshop](https://openclaw.ai/blog/openclaw-agent-skill-workshop/)
 * [OpenClaw NVIDIA Security Collaboration](https://openclaw.ai/blog/openclaw-nvidia-skill-security/)
 * [OpenClaw Auto Mode for Exec Approvals](https://openclaw.ai/blog/safer-than-yolo-auto-mode-for-exec-approvals/)
