Grafana has released the OpenLIT Operator, a Kubernetes-native solution for monitoring AI workloads without requiring code changes. The integration with Grafana Clouds AI Observability suite promises automatic instrumentation of LLMs, vector databases, and agent frameworks—addressing a critical gap as AI infrastructure becomes standard in production environments. For organizations struggling to gain visibility into distributed AI systems, the zero-code approach removes the primary barrier to comprehensive observability.
The Zero-Code Problem
Traditional observability requires developers to instrument their applications manually, adding SDKs, configuring exporters, and maintaining instrumentation code alongside business logic. For AI workloads, this burden multiplies: teams may use multiple model providers (OpenAI, Anthropic, Google, AWS Bedrock), agent frameworks (LangChain, CrewAI, LlamaIndex), vector databases, and custom tools across distributed microservices. Maintaining consistent instrumentation across this heterogeneous stack is unsustainable.
The OpenLIT Operator addresses this by automatically injecting OpenTelemetry instrumentation into pods based on Kubernetes label selectors. No code changes required. No image rebuilds needed. Teams can instrument existing AI workloads in minutes rather than weeks.
How It Works
The operator uses Kubernetes mutating webhooks to inject init containers that configure OpenTelemetry auto-instrumentation at pod startup. When a pod matching your configured labels starts, the operator transparently adds the instrumentation libraries and configuration before the application containers begin execution. The application itself requires no awareness that its being instrumented.
Telemetry flows to Grafana Cloud via OTLP (OpenTelemetry Protocol), where its stored in Tempo for traces and Prometheus for metrics. Pre-built dashboards then visualize the data without additional configuration:
- Request throughput and latency percentiles (p50, p95, p99)
- Token usage counts and cost estimation derived from token volumes
- Agent workflow step sequences and tool invocation patterns
- Vector database query latency and cache performance
- MCP (Model Context Protocol) health and latency metrics
Supported Frameworks and Providers
The OpenLIT Operator covers the major components of modern AI infrastructure:
LLM Providers: OpenAI, Anthropic, Google, AWS Bedrock, and Mistral are all instrumented. Token counts, model selection, and latency are captured for every call.
Agent Frameworks: LangChain, LlamaIndex, CrewAI, Haystack, DSPy, and the OpenAI Agents SDK are all supported. The instrumentation captures agent step sequences, tool invocations, and intermediate reasoning.
Vector Databases: Chroma, Pinecone, Weaviate, and other popular vector stores are instrumented for query latency, cache hit rates, and embedding generation times.
The plugin architecture allows extending support to additional providers without requiring application code changes—just an operator configuration update.
Setup and Configuration
Implementation requires four steps:
- Enable the AI Observability integration in Grafana Cloud, which provisions dashboards and the OTLP gateway automatically.
- Deploy the OpenLIT Operator via Helm: helm repo add openlit and helm install openlit-operator.
- Create an AutoInstrumentation custom resource defining label selectors and OTLP endpoints.
- Rolling restart pods matching the selector to trigger instrumentation injection.
Once running, new pods matching the label selectors automatically receive instrumentation without further manual intervention. Existing workloads gain observability with a single restart.
Why AI Observability Requires Specialization
AI observability differs fundamentally from traditional application performance monitoring. Standard APM focuses on request latency, error rates, and throughput. AI workloads add critical dimensions: token consumption directly translates to cost; model selection impacts both quality and price; agent decision sequences are themselves valuable debugging information; and vector database performance can bottleneck entire RAG pipelines.
By removing the instrumentation barrier entirely, Grafana and OpenLIT enable teams to adopt AI infrastructure without sacrificing visibility or dedicating engineering resources to building custom monitoring solutions. The platform understands AI-specific concerns natively rather than treating LLM calls as opaque HTTP requests.
Vendor neutrality via OpenTelemetry means teams can migrate between observability backends without re-instrumenting, avoiding lock-in while benefiting from Grafana Clouds pre-built AI dashboards and alerting. For teams already invested in OpenTelemetry, the integration is seamless. For those new to observability, the zero-code approach eliminates the steepest part of the learning curve.
Sources
- Grafana Blog: Instrument zero-code observability for LLMs and agents on Kubernetes (March 20, 2026)
- Grafana Cloud AI Observability Documentation
- OpenLIT Project Documentation