Prometheus 3.15 adds a small command-line option with unusually large operational consequences: --auto-gomemlimit.refresh-interval. The flag lets a running Prometheus process periodically rediscover its container or host memory limit and update Go’s GOMEMLIMIT without a restart. For Kubernetes operators using vertical autoscaling or changing pod resources through deployment automation, that closes a gap between the memory Kubernetes grants and the memory the Go runtime believes it can use.
The central lesson is not that Prometheus has become self-sizing. It has not. The release instead makes one layer of its memory behavior responsive to a changing cgroup boundary. Teams still need defensible requests and limits, headroom for non-Go-heap memory, and alerts for pressure. But they no longer have to accept a stale garbage-collector target simply because the pod started under a different limit.
The stale-limit problem inside a long-running pod
Prometheus already enables automatic GOMEMLIMIT detection by default. At startup it detects the Linux container or system memory limit and, by default, sets the Go runtime limit to 90% of that maximum. That behavior is useful because the runtime can begin garbage collection with knowledge of the boundary imposed by the cgroup, rather than behaving as though the host’s full memory were available.
Before the new refresh option, however, that discovery was effectively a startup snapshot. Kubernetes could later change the pod’s memory allocation, but the process would continue using the value derived from the original boundary. A higher container limit would leave the runtime operating under an unnecessarily conservative target. A lower limit was more dangerous: the process could retain a heap policy calibrated for memory it no longer had.
Prometheus 3.15 makes rediscovery periodic. The documented default remains 0s, which means detection still happens only once at startup unless an operator opts in. Setting a duration such as 1m asks Prometheus to check again on that cadence. The feature only applies while automatic memory-limit handling is enabled.
This distinction matters in Kubernetes because resource configuration is increasingly dynamic. A Vertical Pod Autoscaler can recommend or apply new memory settings, and newer Kubernetes resource-resize paths can change some running containers without recreating the pod when the platform, workload, and policy support it. Even where a resize still causes replacement, operators often run long-lived Prometheus instances between rollouts. A runtime limit that follows the cgroup provides a cleaner contract for the in-place case and a safer default for future automation.
What GOMEMLIMIT controls—and what it does not
GOMEMLIMIT is a soft memory limit used by the Go garbage collector. It guides how aggressively the runtime should collect to keep runtime-managed memory near a target. It is not a hard cap on the process’s resident set size, and it does not make the kernel’s out-of-memory enforcement disappear.
That boundary is especially important for Prometheus. The process uses memory for the Go heap, but its total footprint can also include memory-mapped TSDB data, stacks, executable mappings, buffers, and other runtime or operating-system allocations. Kubernetes enforces the container limit against the container’s memory accounting, not merely against the Go heap. A 90% ratio therefore preserves some headroom, but the right ratio depends on workload characteristics and observed non-heap consumption.
The new option changes the runtime’s target when the detected outer limit changes. It does not:
- choose the correct Kubernetes memory request or limit;
- predict cardinality growth, query bursts, or remote-write backpressure;
- guarantee that resident memory immediately falls after a downward resize;
- replace query sample limits, retention controls, or capacity planning; or
- make every live downward resize safe.
Prometheus documentation explicitly warns that lowering the detected limit can temporarily increase garbage-collection activity. That is expected: the runtime has to work harder to approach a smaller target. If the new cgroup ceiling leaves too little room for current live data and non-heap allocations, additional GC cycles cannot solve the underlying capacity mismatch.
Why downward resizing deserves a separate policy
Increasing a memory limit is operationally forgiving. After the next refresh, the Go runtime can adopt a larger target and reduce some collection pressure. The pod receives more room, while the application remains functional during convergence.
Decreasing a limit reverses the risk. Kubernetes enforcement may change faster than the application can shed memory. Even after Prometheus notices the new ceiling, the garbage collector can only reclaim unreachable Go objects; it cannot erase live series, accelerate retention expiry on demand, or instantly shrink every mapped region. A lower setting may therefore produce a period of high CPU consumption, latency, or an OOM kill.
Platform teams should treat upward and downward actions differently. Upward changes can usually be automated within tested bounds. Downward changes should pass a safety gate based on recent working-set behavior and should preserve a rollback path. A conservative policy is to permit automated increases, collect recommendations for decreases, and apply reductions during controlled windows until the team has evidence that live resizing is safe for its ingestion and query profile.
The refresh interval also deserves tuning. A very long interval extends the stale-target window. An extremely short one adds little value because cgroup limits do not normally change every few seconds, and it can make incident timelines harder to interpret. A one- to five-minute interval is a reasonable evaluation point for many environments, but it should be aligned with the autoscaler’s update cadence and the organization’s response objectives rather than copied blindly.
A rollout plan for Kubernetes operators
1. Establish the current memory envelope
Before enabling refresh, record each Prometheus shard’s memory request, memory limit, ingestion rate, active-series count, query concurrency, and recent peak working set. Compare normal operation with compaction, rule evaluation, and high-cardinality query periods. The point is to identify how much of the limit is genuinely spare and how much is already consumed outside the Go heap.
Do not infer safety from the absence of OOM kills alone. A pod can avoid termination while spending excessive CPU in garbage collection or slowing queries under pressure. Include process CPU, container memory, restart count, scrape health, rule-evaluation duration, and query latency in the baseline.
2. Upgrade before changing autoscaling behavior
Prometheus 3.15 is the feature-bearing release, while 3.15’s release notes also contain TSDB, PromQL, histogram, discovery, and remote-write changes. Treat the version upgrade as its own controlled change. Validate rules, dashboards, sidecars, direct TSDB readers, and storage integrations in a canary before coupling the rollout to vertical resizing.
This separation makes failures legible. If the binary upgrade and the autoscaling policy change land together, an operator may struggle to distinguish a query regression from an aggressive memory recommendation or a resize-induced GC spike.
3. Enable refresh on a canary
Add --auto-gomemlimit.refresh-interval to one representative instance while leaving --auto-gomemlimit enabled. Keep the existing ratio initially. Verify through logs and runtime telemetry that the instance remains stable, then perform a modest, controlled upward change. If the cluster supports the intended in-place path, confirm that the container limit changes without a restart and that Prometheus adopts the new runtime target after the configured interval.
For a downward test, use a noncritical canary with substantial headroom. Observe CPU, GC activity, resident memory, scrape gaps, query latency, and pod events throughout the transition. Abort if pressure rises toward the cgroup limit or service-level objectives degrade.
4. Put guardrails around recommendations
Vertical autoscaling recommendations are inputs, not truth. Prometheus memory varies with active series, label cardinality, scrape volume, query load, rule evaluation, and TSDB activity. A recommendation window dominated by a quiet weekend can be unsafe on Monday morning.
- Set a minimum memory floor derived from peak production behavior.
- Limit the percentage reduction allowed in a single change.
- Require longer evidence windows for decreases than for increases.
- Preserve headroom for non-heap allocations and transient work.
- Alert on repeated resizing, OOM events, and sustained memory pressure.
- Document whether a resize is expected to be in place or restart-based in each cluster version and policy.
5. Test the failure mode, not only the happy path
A useful exercise is to simulate a recommendation that is too small. Confirm that alerts fire before termination, the autoscaler or deployment controller can reverse the change, and the on-call engineer can identify the resize in pod events and configuration history. Also test what happens if the limit changes while compaction or a large query is active.
These tests reveal whether the organization has built an adaptive system or merely enabled an adaptive flag. The flag changes one runtime variable. The surrounding controls determine whether that change improves resilience.
The broader signal in Prometheus 3.15
The release contains other substantial work, including OpenMetrics 2.0 scraping, Unix domain socket targets, TSDB performance improvements, a stabilized XOR2 float encoding configuration, and more tracing around scrapes, queries, and rule evaluation. But runtime memory refresh is the clearest cloud-native signal because it acknowledges that a container’s resource boundary may be mutable during the life of the process.
That is a meaningful shift in operational assumptions. Many applications read environment and cgroup state once and then depend on a restart to reconcile reality. As Kubernetes gains more mature mechanisms for changing resources, workloads need to decide which settings can safely follow those changes and which still require replacement. Prometheus now gives operators an explicit control for the Go memory target rather than forcing a restart solely to refresh it.
The practical payoff is modest but real: better alignment between the runtime and its scheduler-imposed envelope, particularly after an in-place memory adjustment. The operational obligation remains larger. Teams must preserve headroom, distinguish heap targets from total container memory, stage the 3.15 upgrade, and make downward resizing conservative. Used that way, periodic GOMEMLIMIT refresh can remove one source of stale configuration without pretending that observability capacity has become automatic.
