# OpenCost 1.121.0 Brings Token-Level GPU Cost Visibility to Kubernetes AI Workloads

- Link: https://thestackobserver.com/opencost-1-121-0-brings-token-level-gpu-cost-visibility-to-kubernetes-ai-workloads/
- Published: 2026-08-19T08:47:18-04:00
- Author: Stackxx

Platform teams running large language model inference on Kubernetes face a problem
that is deceptively simple: they know their GPU bill is rising, and they know their
models are serving tokens, but they have no way to connect those two numbers. Without
per-model, per-token costs derived from actual cluster resource consumption, every
infrastructure decision becomes guesswork.

The question — _is self-hosting cheaper than a managed API?_ — has historically 
been unanswerable with data. Teams see total infrastructure spend and track token
throughput, but the causal chain between them is invisible. When executives ask 
about AI return on investment, the answer is fuzzy at best. Recognizing this gap,
the maintainers of **OpenCost**, a CNCF incubating project, have shipped a feature
in version 1.121.0 that measures inference costs at the token level. The integration
works through CNCF sandbox project **llm-d**, an open-source, Kubernetes-native 
stack for distributed LLM inference. The result is a set of metrics that reconcile
what a model costs to keep warm against what it costs to actually run — and expose
both through standard Prometheus and REST APIs.

## The Inference Cost Black Box

Cost visibility in Kubernetes has always lagged behind operational visibility. Teams
built dashboards for CPU throttling, memory pressure, and pod restarts long before
they could answer basic questions about which workload was driving their cloud bill.
GPU workloads make this worse by an order of magnitude. A GPU actively serving a
loaded LLM is expensive. A GPU sitting idle with model weights loaded in VRAM but
no requests arriving is _equally_ expensive, because the reservation cost does not
disappear when the queue empties. A low-traffic model may spend ninety percent or
more of its time in this warm but unproductive state.

OpenCost previously handled GPU allocation at the device level: it knew which namespace
claimed which GPU and attributed the hourly price accordingly. What it could not
do was connect that allocation to the actual inference layer. The token stream flowing
through vLLM was invisible to cost accounting. A model burning through thousands
of dollars per month might be doing so because of heavy traffic, poor batching, 
or simply sitting mostly idle with oversized weights. Without inference-aware metrics,
all three cases looked identical on a cost report.

The 1.121.0 release closes this gap by ingesting vLLM telemetry — specifically `
vllm:prompt_tokens_total` and `vllm:generation_tokens_total` — and correlating it
with OpenCost’s existing allocation engine. It also accounts for KV cache hits, 
which reduce the compute required to process repeated prompts. The result is not
a single number, but two complementary views of cost that answer two fundamentally
different operational questions.

## Allocation Cost vs. Usage Cost

The first metric, **allocation-based cost per model**, captures every dollar attributed
to having that model available: the GPU memory reserved for its weights, compute
consumed during active inference, and a share of shared infrastructure components
including the llm-d gateway, inference scheduler, and KV cache storage. This figure
reconciles with the infrastructure bill. When finance asks what a model is costing
the organization, this number is the answer.

The second metric, **usage-based cost per model**, narrows the scope to only the
resources consumed during active inference. It attributes costs to the tokens actually
processed and adjusts for KV cache savings. When engineering asks what the model’s
_actual work_ cost, this is the relevant figure.

The gap between the two represents the cost of keeping the model warm and ready.
A large gap suggests the deployment is underutilized — paying for idle capacity —
while a small gap indicates the GPU is being kept productively busy. Both metrics
can be expressed as cost per million tokens, and both are published with labels 
for model name, version, namespace, and cost basis. They are available via Prometheus
scrape endpoints and OpenCost’s REST API, making them easy to pull into Grafana 
or FinOps dashboards.

## The Build-vs-Buy Trap That Everyone Falls Into

One of the most common mistakes in AI infrastructure planning is using usage-based
cost per token to justify self-hosting. The reasoning follows a familiar pattern:
compute-only cost for a million tokens runs at one dollar on the cluster, while 
a SaaS API charges two. Self-hosting appears cheaper by half.

The error is that usage-based cost excludes GPU reservation, idle time, model loading
overhead, and the shared infrastructure pieces that surround inference. If utilization
sits at twenty-five percent, the real allocation-based cost is closer to four dollars
per million tokens — double the SaaS price. At fifty percent utilization, self-hosting
becomes competitive. Without the allocation-based figure, teams do not know where
they stand until the quarterly bill arrives.

OpenCost 1.121.0 resolves this by making the distinction explicit. Platform teams
can now compare allocation-based cost against external API pricing and know exactly
where the crossover point lies. The same data also serves as an optimization target:
increasing utilization through smarter routing, traffic consolidation, or model 
sharing directly reduces the allocation-based cost per token and can make self-hosting
economical.

## The Shared Infrastructure Problem

A model deployed on llm-d does not run in isolation. The surrounding infrastructure
includes a CPU-based inference scheduler, a gateway proxy, KV cache storage that
can reach terabyte scale, and a workload autoscaler. These costs are real, and they
need to be attributed fairly across the models they serve.

The OpenCost integration handles this through a shared-labels mechanism rather than
tight coupling to llm-d internals. Each llm-d component is labeled at deployment
time, and OpenCost’s existing distribution logic spreads those costs proportionally.
This keeps the two projects decoupled: OpenCost does not need to understand llm-
d architecture, and llm-d does not need to emit cost-specific metadata. Shared infrastructure
costs are included in allocation-based figures but excluded from usage-based ones,
preserving the semantic difference between the two metrics.

## How Teams Should Use These Metrics

The four possible combinations of high and low allocation and usage costs each tell
a clear operational story.

 * **High allocation cost, low usage cost:** The model is expensive to keep available
   but efficient when active. The problem is underutilization, and the fix is traffic
   consolidation or model sharing.
 * **High allocation cost, high usage cost:** Both keeping the model warm and running
   it are expensive. This is a signal to question whether the model architecture
   or hardware choice is appropriate for the workload.
 * **Low allocation cost, low usage cost:** The deployment is well-sized and productive.
   This is the target state.
 * **Low allocation cost, high usage cost:** The model is cheap to host but expensive
   per token during inference. This points to questions about model size, quantization,
   and hardware fit.

For platform teams operating chargeback or showback programs, querying costs by 
namespace and team label allows monthly billing reports with per-model granularity.
FinOps teams can identify underutilized deployments and quantify the savings from
right-sizing or decommissioning. Looking ahead, llm-d is developing a smart router
that can query the OpenCost API to factor per-token cost into routing decisions 
alongside latency and throughput, creating a feedback loop where cost optimization
happens automatically at the request level.

## What It Took to Build

The feature was validated on a cluster running one hundred nine GPUs across thirty
deployed AI models. The generated metrics were cross-checked against known cluster
costs and token volumes. Core metrics come directly from vLLM, which means that 
vLLM users who do not deploy through llm-d can also benefit from cost attribution,
since the integration reads standard Prometheus metrics rather than llm-d-specific
ones.

On the OpenCost side, upcoming work focuses on measuring wasted GPU capacity, improving
idle-GPU detection for LLM-specific workloads, integrating the new metrics into 
the OpenCost UI, and estimating optimization savings. On the llm-d side, the maintainers
are expanding workload and tenant capture and tightening deployment automation. 
Both projects are open source and accept contributions upstream.

## Why This Matters Now

AI inference workloads are becoming a dominant line item in Kubernetes infrastructure
budgets. Yet the tooling for measuring, attributing, and optimizing those costs 
has been missing. OpenCost 1.121.0 is not a dashboard feature or a billing export—
it is a measurement primitive. By distinguishing what a model costs to keep warm
from what it costs to run, it gives platform teams the data they need to answer 
the build-vs-buy question, to defend budget decisions to leadership, and to optimize
utilization without guessing.

For organizations already running vLLM or llm-d on Kubernetes, the upgrade path 
is straightforward: version 1.121.0 is available now and publishes the new metrics
to existing Prometheus and REST endpoints. For teams still evaluating whether self-
hosted inference makes financial sense, these metrics provide the missing justification
framework. The era of flying blind on AI infrastructure costs is ending. Platform
teams now have a clear signal.

## Sources

 * [OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking — CNCF Blog](https://www.cncf.io/blog/2026/08/05/opencost-1-121-0-first-of-a-kind-kubernetes-inference-cost-tracking/)(
   Sima Nadler, Alex Meijer, August 2026)
 * [OpenCost 1.121.0: First-of-a-Kind Kubernetes Inference Cost Tracking — OpenCost Blog](https://opencost.io/blog/opencost-llmd-inference-cost/)(
   August 2026)
 * [llm-d: Kubernetes-native distributed LLM inference](https://llm-d.ai/)