Every container image you pull from registry.k8s.io passes through kpromo, the Kubernetes image promoter. It’s the unsung hero of Kubernetes releases—and it just underwent a complete rewrite that nobody noticed. That was exactly the point.
The Invisible Infrastructure
The image promoter started in 2018 as an internal Google tool to replace manual, Googler-gated processes for pushing images to k8s.gcr.io. Over seven years, it grew into a monolithic codebase carrying the weight of incremental additions from multiple SIGs and 42 contributors across 3,500 commits.
By 2025, production promotion jobs regularly exceeded 30 minutes and frequently failed with rate limit errors. The core logic had become hard to extend and painful to test.
The Eight-Phase Rewrite
Between February and March 2026, SIG Release executed a complete pipeline rewrite across eight tracked phases:
- Phase 1: Adaptive rate limiting with proper backoff
- Phase 2: Clean interfaces for registry and auth operations
- Phase 3: Pipeline engine replacing monolithic promotion
- Phase 4: SLSA provenance verification for staging images
- Phase 5: Vulnerability scanning and SBOM support
- Phase 6: Separated signing from replication to eliminate rate contention
- Phase 7: Legacy pipeline deletion
- Phase 8: Legacy dependency cleanup (audit subsystem, deprecated tools)
Real-World Impact
The results are significant:
- 20% of the codebase deleted
- Dramatically faster promotion times
- Rate limit failures eliminated through proper throttling
- Testable, extensible architecture
- SLSA provenance attestations for supply chain security
Why This Matters
If the image promoter breaks, no Kubernetes release ships. The rewrite demonstrates how critical infrastructure can be modernized incrementally without disrupting production—proving that invisible rewrites, when done right, are the best kind.
Leave a Reply