Platform teams running large language model inference on Kubernetes face a problem that is deceptively simple: they know their GPU bill is rising, and they know their models are serving tokens, but they have no way to connect those two numbers. Without per-model, per-token costs derived from actual cluster resource consumption, every infrastructure decision becomes guesswork.
The question — is self-hosting cheaper than a managed API? — has historically been unanswerable with data. Teams see total infrastructure spend and track token throughput, but the causal chain between them is invisible. When executives ask about AI return on investment, the answer is fuzzy at best. Recognizing this gap, the maintainers of OpenCost, a CNCF incubating project, have shipped a feature in version 1.121.0 that measures inference costs at the token level. The integration works through CNCF sandbox project llm-d, an open-source, Kubernetes-native stack for distributed LLM inference. The result is a set of metrics that reconcile what a model costs to keep warm against what it costs to actually run — and expose both through standard Prometheus and REST APIs.
The Inference Cost Black Box
Cost visibility in Kubernetes has always lagged behind operational visibility. Teams built dashboards for CPU throttling, memory pressure, and pod restarts long before they could answer basic questions about which workload was driving their cloud bill. GPU workloads make this worse by an order of magnitude. A GPU actively serving a loaded LLM is expensive. A GPU sitting idle with model weights loaded in VRAM but no requests arriving is equally expensive, because the reservation cost does not disappear when the queue empties. A low-traffic model may spend ninety percent or more of its time in this warm but unproductive state.
OpenCost previously handled GPU allocation at the device level: it knew which namespace claimed which GPU and attributed the hourly price accordingly. What it could not do was connect that allocation to the actual inference layer. The token stream flowing through vLLM was invisible to cost accounting. A model burning through thousands of dollars per month might be doing so because of heavy traffic, poor batching, or simply sitting mostly idle with oversized weights. Without inference-aware metrics, all three cases looked identical on a cost report.
The 1.121.0 release closes this gap by ingesting vLLM telemetry — specifically vllm:prompt_tokens_total and vllm:generation_tokens_total — and correlating it with OpenCost’s existing allocation engine. It also accounts for KV cache hits, which reduce the compute required to process repeated prompts. The result is not a single number, but two complementary views of cost that answer two fundamentally different operational questions.
Allocation Cost vs. Usage Cost
The first metric, allocation-based cost per model, captures every dollar attributed to having that model available: the GPU memory reserved for its weights, compute consumed during active inference, and a share of shared infrastructure components including the llm-d gateway, inference scheduler, and KV cache storage. This figure reconciles with the infrastructure bill. When finance asks what a model is costing the organization, this number is the answer.
The second metric, usage-based cost per model, narrows the scope to only the resources consumed during active inference. It attributes costs to the tokens actually processed and adjusts for KV cache savings. When engineering asks what the model’s actual work cost, this is the relevant figure.
The gap between the two represents the cost of keeping the model warm and ready. A large gap suggests the deployment is underutilized — paying for idle capacity — while a small gap indicates the GPU is being kept productively busy. Both metrics can be expressed as cost per million tokens, and both are published with labels for model name, version, namespace, and cost basis. They are available via Prometheus scrape endpoints and OpenCost’s REST API, making them easy to pull into Grafana or FinOps dashboards.
The Build-vs-Buy Trap That Everyone Falls Into
One of the most common mistakes in AI infrastructure planning is using usage-based cost per token to justify self-hosting. The reasoning follows a familiar pattern: compute-only cost for a million tokens runs at one dollar on the cluster, while a SaaS API charges two. Self-hosting appears cheaper by half.
The error is that usage-based cost excludes GPU reservation, idle time, model loading overhead, and the shared infrastructure pieces that surround inference. If utilization sits at twenty-five percent, the real allocation-based cost is closer to four dollars per million tokens — double the SaaS price. At fifty percent utilization, self-hosting becomes competitive. Without the allocation-based figure, teams do not know where they stand until the quarterly bill arrives.
OpenCost 1.121.0 resolves this by making the distinction explicit. Platform teams can now compare allocation-based cost against external API pricing and know exactly where the crossover point lies. The same data also serves as an optimization target: increasing utilization through smarter routing, traffic consolidation, or model sharing directly reduces the allocation-based cost per token and can make self-hosting economical.
The Shared Infrastructure Problem
A model deployed on llm-d does not run in isolation. The surrounding infrastructure includes a CPU-based inference scheduler, a gateway proxy, KV cache storage that can reach terabyte scale, and a workload autoscaler. These costs are real, and they need to be attributed fairly across the models they serve.
The OpenCost integration handles this through a shared-labels mechanism rather than tight coupling to llm-d internals. Each llm-d component is labeled at deployment time, and OpenCost’s existing distribution logic spreads those costs proportionally. This keeps the two projects decoupled: OpenCost does not need to understand llm-d architecture, and llm-d does not need to emit cost-specific metadata. Shared infrastructure costs are included in allocation-based figures but excluded from usage-based ones, preserving the semantic difference between the two metrics.
How Teams Should Use These Metrics
The four possible combinations of high and low allocation and usage costs each tell a clear operational story.
- High allocation cost, low usage cost: The model is expensive to keep available but efficient when active. The problem is underutilization, and the fix is traffic consolidation or model sharing.
- High allocation cost, high usage cost: Both keeping the model warm and running it are expensive. This is a signal to question whether the model architecture or hardware choice is appropriate for the workload.
- Low allocation cost, low usage cost: The deployment is well-sized and productive. This is the target state.
- Low allocation cost, high usage cost: The model is cheap to host but expensive per token during inference. This points to questions about model size, quantization, and hardware fit.
For platform teams operating chargeback or showback programs, querying costs by namespace and team label allows monthly billing reports with per-model granularity. FinOps teams can identify underutilized deployments and quantify the savings from right-sizing or decommissioning. Looking ahead, llm-d is developing a smart router that can query the OpenCost API to factor per-token cost into routing decisions alongside latency and throughput, creating a feedback loop where cost optimization happens automatically at the request level.
What It Took to Build
The feature was validated on a cluster running one hundred nine GPUs across thirty deployed AI models. The generated metrics were cross-checked against known cluster costs and token volumes. Core metrics come directly from vLLM, which means that vLLM users who do not deploy through llm-d can also benefit from cost attribution, since the integration reads standard Prometheus metrics rather than llm-d-specific ones.
On the OpenCost side, upcoming work focuses on measuring wasted GPU capacity, improving idle-GPU detection for LLM-specific workloads, integrating the new metrics into the OpenCost UI, and estimating optimization savings. On the llm-d side, the maintainers are expanding workload and tenant capture and tightening deployment automation. Both projects are open source and accept contributions upstream.
Why This Matters Now
AI inference workloads are becoming a dominant line item in Kubernetes infrastructure budgets. Yet the tooling for measuring, attributing, and optimizing those costs has been missing. OpenCost 1.121.0 is not a dashboard feature or a billing export — it is a measurement primitive. By distinguishing what a model costs to keep warm from what it costs to run, it gives platform teams the data they need to answer the build-vs-buy question, to defend budget decisions to leadership, and to optimize utilization without guessing.
For organizations already running vLLM or llm-d on Kubernetes, the upgrade path is straightforward: version 1.121.0 is available now and publishes the new metrics to existing Prometheus and REST endpoints. For teams still evaluating whether self-hosted inference makes financial sense, these metrics provide the missing justification framework. The era of flying blind on AI infrastructure costs is ending. Platform teams now have a clear signal.
Sources
- OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking — CNCF Blog (Sima Nadler, Alex Meijer, August 2026)
- OpenCost 1.121.0: First-of-a-Kind Kubernetes Inference Cost Tracking — OpenCost Blog (August 2026)
- llm-d: Kubernetes-native distributed LLM inference
