# Kyverno’s CEL Fix Removes the Per-Policy Admission Tax

- Link: https://thestackobserver.com/kyvernos-cel-fix-removes-the-per-policy-admission-tax/
- Published: 2026-11-20T11:58:46-05:00
- Author: Stackxx

Kyverno has removed a deceptively expensive step from its Common Expression Language
admission path: converting the same Kubernetes object once for every matching policy.
The optimization, described by the project this week and shipped in Kyverno 1.19.2,
changes request preparation from work that grew with policy count to work performed
once per admission request.

That sounds like a narrow implementation fix. It is more important than that. Admission
control sits directly on the Kubernetes API write path, where small inefficiencies
can become cluster-wide latency and CPU costs. Kyverno’s profiling found that 86.8%
of allocated bytes in its validating CEL engine came from converting the admitted
object, while policy evaluation itself barely registered. The result is a useful
reminder for platform teams: the expensive part of policy-as-code may be preparing
and dispatching the input, not evaluating the policy language.

## The hidden loop inside admission control

When a workload is created or changed, the Kubernetes API server sends an admission
request containing the object as raw JSON. Kyverno’s CEL engine needs that object
in an unstructured map that CEL expressions can inspect. Before the fix, Kyverno
performed that conversion inside the loop over matching policies.

The input did not change from one policy to the next, but the engine repeatedly 
parsed and converted it. A typical Pod evaluated against a Pod Security Standards
policy set can match about 16 policies, so one admission operation could pay for
the same object preparation 16 times. Larger objects and larger policy portfolios
amplified the waste.

This created the wrong scaling relationship. The necessary work—evaluating each 
policy—should increase with the number of policies. The fixed cost of preparing 
a request should not. Yet object conversion had become an `O(policies)` operation.
In practice, every additional policy added both legitimate evaluation work and an
avoidable serialization tax.

The fix builds the CEL request value once before policy evaluation begins and reuses
it for every matched policy. Kyverno temporarily blanks the expensive object field
while converting the rest of the admission request, then restores the converted 
object. This preserves the existing representation exposed to policy authors without
maintaining a second, potentially divergent conversion implementation.

## The benchmark shows why the slope matters

Kyverno’s benchmark drove the validating engine directly with a roughly 50 KB Pod.
At one matched policy, allocations fell from 2,327 to 854 and engine time dropped
from 0.31 milliseconds to 0.10 milliseconds. At 16 policies, allocations fell from
37,055 to 2,718, while engine time dropped from 4.91 milliseconds to 0.16 milliseconds.
At 64 policies, allocations declined from 148,131 to 8,672 and engine time moved
from 20.01 milliseconds to 0.36 milliseconds.

The largest headline number is tempting, but the curve is the more durable finding.
Moving from one policy to 64 previously increased allocations by roughly 63 times.
After the change, allocations grew by about 10 times, reflecting more of the evaluation
work users actually requested. Object preparation is now constant per request instead
of proportional to the number of deployed policies.

The mutating CEL engine showed the same pattern, though with different totals. At
64 policies, allocations dropped from 258,943 to 52,689 and engine time from 10.59
milliseconds to 2.97 milliseconds. Image verification received the same object-conversion
fix. Its benefit is expected to appear more in memory use than latency because remote
signature and attestation retrieval often dominates that path.

These are engine benchmarks, not promises about end-to-end API latency. They exclude
the webhook network hop, API server processing, and etcd. Still, they isolate the
corrected behavior precisely: Kyverno no longer converts the same admitted object
once per policy. Operators should treat the results as evidence about scaling characteristics,
then validate absolute improvements under their own request mix and topology.

## Compatibility made the optimization harder

Hoisting a conversion out of a loop is straightforward only if every consumer interprets
the result identically. Kyverno’s validating, generating, and mutating engines had
developed slightly different contracts for `request.object`. Validation and generation
saw the object as supplied by the API server; mutation saw a normalized form. The
engines also represented an absent object differently for operations such as DELETE.

A single universal request builder would have been cleaner internally but could 
have changed the result of policies already running in production. Kyverno instead
retained the existing behavior of each engine while sharing the optimized implementation
underneath. Equivalence tests compare the new and old outputs field by field across
request types.

That choice is operationally significant. Admission policy changes can reject deployments,
block automation, or alter mutation results. A performance patch that also changes
input semantics creates a risky migration disguised as a routine upgrade. Preserving
the policy-visible contract lets teams adopt the optimization without rewriting 
expressions or rediscovering edge cases in production.

## What it changes in the native-versus-webhook debate

Kubernetes ValidatingAdmissionPolicy has been stable since version 1.30 and provides
an in-process, CEL-based alternative to validating webhooks. Its architecture avoids
a webhook network call, making native admission attractive for straightforward validation.
Kyverno adds a broader policy system, including generation, mutation, image verification,
reporting, and policy distribution, but it must receive requests over the admission
webhook path.

The Kyverno analysis compared the same 16 Pod Security Standards policies in both
engines. In its single-host setup, native admission led at one policy, at about 
9.2 milliseconds versus Kyverno’s 12.8 milliseconds. At all 16 policies, Kyverno
measured about 20 milliseconds versus 29 milliseconds for native admission. Those
Kyverno figures came from version 1.19.0, before the new optimization, and the project
has not rerun the comparison with the fix.

The study suggests different cost models. Native admission dispatches policies inside
the API server, so its work grows with each policy and consumes control-plane CPU.
Kyverno pays a webhook cost once per request and evaluates the matching set in a
separately scalable controller. The project measured roughly 3.2 CPU cores for Kyverno
against 8.2 for native under its matched test load, with much of the API server 
CPU attributed to policy dispatch rather than CEL bytecode execution.

Those results should not be treated as a universal victory for either approach. 
The comparison used a loopback webhook on one machine, removing the real network
latency a multi-node cluster introduces. It also used generated equivalent policies,
dry-run operations, and disabled side controllers. Native admission remains compelling
when a small set of validation rules fits its scope. An external engine becomes 
more attractive when teams need a larger policy portfolio, features beyond validation,
or the ability to scale policy processing separately from the control plane.

## What platform teams should do now

Teams running Kyverno should first inventory the engine and version they use. The
CEL optimization and a separate legacy-engine context-copy fix ship in 1.19.2. Kyverno
1.20.0 is also available, so upgrade planning should follow the project’s supported
upgrade path rather than target a patch release in isolation.

Before and after an upgrade, operators should measure admission behavior under realistic
policy density. Useful signals include:

 * **Admission latency by percentile:** compare median and tail latency for create
   and update operations, especially during deployment bursts.
 * **Controller CPU and allocation pressure:** watch for reduced CPU consumption,
   garbage collection, and memory churn as the number of matching policies rises.
 * **API server impact:** separate webhook duration from overall API request latency
   so improvements are not hidden by etcd, authentication, or other admission plugins.
 * **Policy-count sensitivity:** test one policy, a typical workload set, and a 
   high-density case. The optimization’s main value is the improved scaling slope.
 * **Semantic equivalence:** run representative CREATE, UPDATE, and DELETE requests
   in audit or test environments to confirm existing validation and mutation outcomes.

Policy authors should not interpret better engine efficiency as permission to ignore
expression cost. Complex authorization checks, external data calls, image verification,
and mutation chains still have their own latency profiles. The fix removes redundant
preparation; it does not make every policy operation constant-time.

## The remaining performance boundaries

Kyverno still performs one JSON round trip for the admitted object per request to
preserve the raw representation historically exposed as `request.object`. Replacing
that with a typed or cheaper representation could save more work, but it would require
an explicit compatibility design. Mutating policies also pass full objects between
policies rather than propagating only patch differences, which leaves another potential
but much larger optimization.

The project also cautions that the change does not address tail-latency variation
under sustained load on a single replica. Benchmarking the engine in isolation cannot
answer how network placement, controller saturation, failure policy, API server 
concurrency, and object size interact in a production cluster. Teams should scale
admission controller replicas, set disruption budgets, and monitor webhook availability
as reliability concerns independent of this CPU improvement.

The broader lesson extends beyond Kyverno. In extensible control planes, input construction
and dispatch often outweigh the apparent core computation. CEL evaluation was the
obvious suspect, but profiling found repeated object conversion instead. Cloud native
teams evaluating policy engines should therefore ask not only how fast an expression
executes, but how often inputs are parsed, where dispatch runs, and whether fixed
work is accidentally repeated for every rule.

## A better foundation for larger policy portfolios

Kyverno’s optimization does not eliminate the admission webhook tradeoff, and it
does not establish one policy architecture as universally fastest. It does make 
Kyverno’s CEL path scale more honestly: adding policies now adds mainly the evaluation
work those policies require, not repeated conversion of an unchanged Kubernetes 
object.

For platform teams expanding policy-as-code, that is the meaningful outcome. Policy
count can grow as organizations standardize security, supply-chain, and operational
controls. Removing hidden per-policy overhead gives those teams more predictable
capacity planning and keeps policy processing from consuming resources for work 
that should have happened only once.

## Sources

 * [CNCF: Optimizing Kubernetes admission control—where Kyverno’s CEL admission cost actually goes](https://www.cncf.io/blog/2026/11/18/optimizing-kubernetes-admission-control-where-kyvernos-cel-admission-cost-actually-goes/)
 * [Kyverno releases on GitHub](https://github.com/kyverno/kyverno/releases)
 * [Kubernetes documentation: Validating Admission Policy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/)
