Kubernetes 1.37 Memory QoS: Why cgroups v2 Do Not Automatically Protect SaaS from Memory Pressure

This article explains what actually changes after upgrading to Kubernetes 1.37, how memory.high, memory.min, and memory.low affect latency and OOM behavior, and how to safely test Memory QoS under real workloads.

After upgrading a cluster to Kubernetes 1.37, you may see the MemoryQoS feature gate enabled and draw a dangerous conclusion: kubelet now automatically protects critical services from memory pressure.

It does not. In the default Kubernetes 1.37 configuration, kubelet does not write memory.high, memory.min, or memory.low for containers. A regular memory limit still maps to memory.max, and the new mechanisms affect node behavior only after you explicitly configure memoryThrottlingFactor or memoryReservationPolicy: TieredReservation.

Even once enabled, Memory QoS does not eliminate memory-related failures. It changes their form:

• instead of a fast OOM, an application may spend a long time in direct reclaim and lose its P99 latency;

• protecting a Guaranteed Pod with memory.min can make its file cache unavailable for reclaim;

• protecting a critical service can reduce the elasticity available to neighboring workloads;

• fewer OOMKilled events do not necessarily mean better SLOs.

Memory QoS is therefore not a safe switch. It is a policy for allocating a scarce resource. It must be evaluated at the node-pool level, against the workload profile and user-facing latency.

What Actually Changes in Kubernetes 1.37

Kubernetes 1.37 was released on August 26, 2026, and in September the project announced that Memory QoS had moved to Beta. The MemoryQoS feature gate is enabled by default in kubelet 1.37, as confirmed by the release announcement and the dedicated Memory QoS post.

However, feature availability and an active policy are different things. The defaults are:

• memoryThrottlingFactor: null;

• memoryReservationPolicy: None.

With this configuration, the upgrade alone does not enable memory.high throttling or create protection through memory.min or memory.low. Container memory behavior remains largely unchanged.

There is one important exception. If memoryThrottlingFactor was explicitly set before the upgrade, that value is retained and memory.high throttling continues to operate. As a result, two otherwise identical clusters can behave differently after upgrading to 1.37 because of their configuration history.

A version rollback does not necessarily restore the previous state immediately, either. A memory.high value already written to a container may persist until the container restarts, its resources change, or the runtime applies resource configuration. This is why you need to inspect not only declarative kubelet configuration, but also the actual cgroup files of running containers. KEP-2570 describes this behavior in detail.

The Basic Model: Four Distinct cgroup v2 Mechanisms

Confusion around Memory QoS often starts with treating every cgroup parameter as a type of limit. In practice, they solve different problems.

The detailed semantics of these parameters are documented in the cgroup v2 memory controller documentation.

A Limit Remains a Hard Limit

A standard Kubernetes memory limit maps to memory.max. Memory QoS does not replace it. If a container reaches memory.max and the kernel cannot reclaim enough memory, the kernel invokes the OOM killer within that cgroup.

This fundamentally distinguishes a limit from memory.high. The former defines how much memory a container may retain. The latter makes further memory allocations costly before the container reaches the hard limit.

Requests Become Protection Only with TieredReservation

Without a corresponding policy, a memory request primarily participates in scheduling and Pod eviction. It does not mean that the Linux kernel has reserved an unreclaimable area of RAM for the container.

With memoryReservationPolicy: TieredReservation, kubelet begins mapping requests to cgroup v2 mechanisms:

• Guaranteed receives memory.min;

• Burstable receives memory.low;

• BestEffort receives no protection.

Kubelet must also configure ancestor QoS cgroups; otherwise, protection on a child container will not work effectively through the hierarchy. The mapping between QoS classes and memory parameters is described in the Kubernetes documentation.

memoryThrottlingFactor: The Space Between Normal Operation and OOM

Once memoryThrottlingFactor is explicitly configured, kubelet sets memory.high for Burstable and BestEffort containers. This mechanism does not apply to Guaranteed containers.

For Burstable workloads, the threshold is calculated as follows:

memory.high = request + factor × (limit − request)

The result is rounded to the memory page size. If no limit is set, the node's allocatable memory available for Pod placement is used in its place.

For example, with a request of 256 MiB, a limit of 512 MiB, and a factor of 0.9, the threshold is about 486 MiB. The container can continue toward memory.max, but once it crosses memory.high, new allocations become substantially more expensive.

Crossing memory.high Is Not an OOM

The kernel does not terminate a process immediately after it crosses memory.high. Instead, it puts cgroup processes into intensive direct reclaim: the thread requesting memory participates in freeing it. This slows memory growth and can give an operator or an external controller time to:

• reduce incoming load;

• raise the limit;

• terminate the task in a controlled manner;

• move the workload to another node.

For a batch or retryable workload, this can be better than an immediate OOM. For a latency-sensitive API, however, it can be worse: the process remains alive and formally ready, while requests start stalling on memory allocation.

Readiness and liveness probes are not guaranteed to detect this degradation. They may remain successful while user-facing P99 has already exceeded the SLO.

What a Synthetic Test Shows

In an experiment published in a repository, a Burstable Pod with a 256 MiB memory request, a 512 MiB limit, and a 0.9 factor received a memory.high value of roughly 486 MiB. Before being OOMKilled about 71 seconds later, the cgroup accumulated 3,596 high events.

The measured operation took about 0.5 ms before memory pressure, then increased to:

• 27 ms at 490 MiB;

• 119 ms at 495 MiB;

• 448 ms at 500 MiB.

This test is useful as a demonstration of the mechanism: memory.high can indeed replace an immediate failure with a period of severe degradation. But it cannot be used to infer the expected performance loss for a JVM, a Go service, a database, or a file-heavy workload. Public data does not support transferring these figures to a typical SaaS environment.

The Kernel Requirement Is Not a Formality

The earlier promotion of Memory QoS to Beta for Kubernetes 1.28 was canceled because of the risk of livelock when using memory.high on Linux kernels older than 5.9. On newer kernels, the fix should ensure that processes can make forward progress even under memory pressure.

Kubelet 1.36 and later warns about kernels that are too old, but a warning is not a substitute for inventorying node images. This is especially important in managed Kubernetes, where the Kubernetes version, node image, kernel, container runtime, and availability of kubelet configuration can be updated independently.

TieredReservation: Real Protection at the Cost of Elasticity

memory.min and memory.low do not reserve physical memory in advance. If a cgroup uses 200 MiB with memory.min set to 2 GiB, the remaining 1.8 GiB is not subtracted from free RAM.

Protection applies only to memory that is actually in use. As a result, enabling TieredReservation does not necessarily reduce placement density immediately: the scheduler already accounted for requests rather than fully preallocating physical memory.

The problem emerges later, once the protected working set or file cache is populated. Used memory within the effective memory.min cannot be reclaimed for neighboring workloads. With memory.low, protection is soft: the kernel can violate it when there is no better option. With memory.min, protection is hard.

Why a Guaranteed Pod Can Unexpectedly Retain Cache

A cgroup accounts for more than a process's anonymous memory; it also accounts for file cache. For a Guaranteed Pod, memory requests equal limits. With TieredReservation, this results in:

memory.min = memory.max

Therefore, memory already charged to that cgroup, including file cache, receives hard protection up to the limit. If the application needs room for new anonymous allocations, reclaiming its own cache may be constrained by that protection, with a cgroup OOM as the result.

This matters especially for services with intensive file reads, tmpfs, shared files, or large caches. The actual effect depends on how the kernel charges pages, which workload accesses files first, and how I/O is structured. Assessing risk from RSS alone is therefore insufficient: at a minimum, you need the anon and file fields from memory.stat.

The Conflict with Dense Mixed Placement

Some teams assign the Guaranteed QoS class not because an application needs unreclaimable memory, but to obtain predictable eviction semantics: they set the memory request equal to the limit while keeping actual consumption substantially lower.

On a mixed node, TieredReservation changes the meaning of that setup. An unused memory request still does not consume RAM, but accumulated, otherwise reclaimable file cache in a Guaranteed Pod becomes hard-protected. Neighboring elastic workloads lose the ability to use that memory during a burst.

This is a known architectural limitation: TieredReservation is a node-wide policy. You cannot enable memory.min only for a selected set of Pods. An accepted SIG Node issue documents the conflict between hard Guaranteed protection and dense mixed placement; the discussed future directions include soft protection and Pod-level configuration.

At this stage, a separate node pool remains a more manageable boundary for workloads that genuinely require hard protection.

What Happens When Memory Runs Short

Memory QoS does not create a single, sequential failure path. Several mechanisms operate on a node at the same time.

1. Reclaim and cgroup Throttling

The kernel attempts to reclaim memory. memory.low affects the order of reclamation, memory.min prevents reclaiming the protected portion, and memory.high makes processes in a specific cgroup perform direct reclaim.

2. Kubelet Eviction

Under MemoryPressure, kubelet may evict Pods. Memory QoS does not disable this mechanism. Pod ranking considers:

1. whether actual usage exceeds the request;

2. Pod Priority;

3. the size of the excess over the request.

The detailed ordering is described in the documentation on node-pressure eviction.

This means a request simultaneously affects scheduling, protection under TieredReservation, and eviction probability. A mistaken request is no longer only a question of placement density.

3. Kernel OOM

The kernel may invoke the OOM killer before kubelet has time to react. Process selection depends in part on oom_score_adj, which is associated with the QoS class. Both cgroup OOM at memory.max and node-level failure are possible.

Therefore, the absence of eviction does not prove that the policy is working correctly, and the absence of OOM does not prove the absence of user impact. An application may remain alive while spending a substantial portion of its time in reclaim.

What Data You Need for Diagnosis

Watching only kubectl top and the OOMKilled count is insufficient. For every tested container, you need data from at least four groups.

cgroup State

• memory.current;

• memory.high;

• memory.min;

• memory.low;

• memory.max.

These show not the intended policy, but the values actually applied by the kernel.

Memory Controller Events

In memory.events, monitor:

• high;

• low;

• max;

• oom;

• oom_kill.

Growth in high without OOM may not be a success. It can indicate that the application is being silently throttled.

Memory Composition and Pressure

You need:

• memory.stat, especially the relationship between file and anon;

• memory PSI;

• I/O metrics when the workload actively uses files;

• kubelet protected-memory metrics, if they are available in your particular distribution.

Application and Kubernetes Metrics

The same time series should include:

• P50, P95, and P99 for user-facing operations;

• throughput and RPS;

• error and timeout rates;

• container_oom_events_total;

• OOMKilled events and restarts;

• evictions;

• MemoryPressure state;

• readiness and liveness probe results.

Otherwise, a team may declare success after reducing OOM events that were merely traded for prolonged latency.

How to Test the Policy Before Deployment

Run the experiment on a separate node pool with the same kernel, runtime, and OS image planned for production. It is useful to compare four modes:

1. the baseline configuration, with no active Memory QoS policies;

2. memoryThrottlingFactor only;

3. TieredReservation only;

4. both policies together.

Each mode requires the same reproducible workload. Synthetic memory allocation verifies that values are written to the cgroup and that the expected events occur. The decision to deploy should be based on reproducing the service's real profile: requests, background jobs, cache, storage access, and characteristic bursts.

A minimal test scenario should include:

• sustained normal load;

• gradual growth in memory consumption;

• a short burst above the request;

• pressure from a neighboring container;

• a profile with a large anonymous working set;

• a profile with a substantial file cache;

• reaching memory.high and memory.max;

• verifying kubelet behavior under MemoryPressure.

Define success and stop criteria before testing. They cannot be reduced to “fewer OOMs.” A policy is acceptable only if latency, throughput, errors, eviction behavior, and recovery time remain within the team's SLOs.

A Practical Rollout Strategy

Before enabling the policy, work through four stages.

1. Build an Inventory

For every node pool, verify:

• whether cgroup v2 is actually in use;

• the kernel version and compliance with the 5.9-or-newer requirement;

• the node image and container runtime;

• the kubelet configuration actually loaded;

• current cgroup values for running containers;

• whether memoryThrottlingFactor was configured previously;

• whether these parameters can be controlled through your Kubernetes provider.

Kubernetes 1.37 alone answers none of these questions.

2. Separate Workloads by Their Desired Failure Mode

memory.high is sensible where slowing down is preferable to immediate termination: for example, retryable background jobs or workloads that an external controller can throttle.

For APIs with strict SLOs, decide separately which outcome is worse: a rapid failure followed by restart, or a sustained increase in P99. There is no universal answer.

TieredReservation is better suited to a dedicated pool with well-measured memory requests and critical Guaranteed services. On a node mixing APIs, batch jobs, and file caches, one uniform hard policy is materially riskier.

3. Introduce Changes One at a Time

If throttling and reservation are enabled simultaneously, it will be difficult to distinguish the impact of direct reclaim from the impact of insufficient reclaimable memory during a degradation. Test each policy separately first, then test the combination.

Rollback must account for the fact that removing a parameter from configuration does not necessarily clear memory.high immediately on already running containers. The rollback plan should include cgroup verification and a controlled rollout of affected Pods.

4. Build Protection as a System, Not a Single Parameter

Meta's experience with cgroup v2 demonstrates the value of soft protection through memory.low, but also its limits. In the fbtax2 model, protection for critical workloads was combined with PSI, oomd, swap, and I/O control. Even with that design, temporary RPS dips and incomplete-isolation effects remained. The results are described in the fbtax2 documentation.

This example is not a ready-made Kubernetes policy, but it demonstrates the right level of thinking: cgroups set priorities under scarcity; they do not eliminate scarcity itself. For elastic SaaS, soft protection, PSI monitoring, and controlled pressure response may be more useful than hard memory.min for every Guaranteed Pod.

The practical position for Kubernetes 1.37 is straightforward: do not treat Memory QoS as an outcome of the upgrade. First, verify the actual cgroup settings. Then choose the desired failure mode for each workload class. Only after measuring latency, reclaim, file cache, evictions, and OOM behavior should you decide whether the policy belongs across an entire node pool.