Why Kubernetes 1.35 resizePolicy: RestartContainer Isn't Required for ZGC/Shenandoah

A lot of current advice says to set resizePolicy: RestartContainer for memory on every JVM container. For growing the heap ceiling, that advice is right. For shrinking and regrowing within a ceiling you chose at launch, I tested ZGC and Shenandoah in a Linux container and found a restart isn't needed, as long as the JVM releases memory before the limit drops.

The short version

• Kubernetes 1.35 can change a running container's memory in place (the pods/resize subresource is GA).
• A JVM fixes its maximum heap at startup. Raising the limit in place does not raise the heap. Two popular articles are right about this.
• That does not mean every memory change needs a restart. Three different operations hide inside "resize memory", and only one of them requires it.
• On ZGC and Shenandoah you can lower and raise the heap's soft ceiling at runtime. In my test, releasing memory first let a container's limit drop from 900 MB (or 700 MB) to 300 MB with no restart. Lowering the limit first got the JVM OOM-killed every time.

What the advice says, and where it's right

Rack2Cloud's VPA in production article says: "-Xmx is a startup parameter. The JVM reads it once at initialization and sets the maximum heap ceiling for the lifetime of the process." It concludes that resizePolicy: RestartContainer for memory "is the correct setting for JVM services, not NotRequired". Its Kubernetes 1.35 article lists RestartContainer as appropriate "for memory decreases on workloads where the allocator won't release memory without a restart".

Shubham Chepe's post on the JVM and in-place resize gives the same prescription: "CPU NotRequired, memory RestartContainer". He explains why: "If you raise the memory limit in place, the heap ceiling doesn't grow." On shrinking he says it is "riskier still, since the JVM may already be using more than the new limit allows."

Both are correct that:

• the heap ceiling comes from -Xmx or -XX:MaxRAMPercentage at launch, and
• a bigger cgroup limit does not make a running JVM use it.

If you need the heap ceiling itself to move, you need a restart. This article doesn't dispute that.

Three operations, not one

The advice treats all three as the first one. That is the right call if you size -Xmx to today's limit. It is the wrong one if you set -Xmx for the peak and treat memory below that as adjustable.

ZGC and Shenandoah make this possible because of SoftMaxHeapSize, a manageable flag. You can change it on a running JVM:

jcmd <pid> VM.set_flag SoftMaxHeapSize <bytes> jcmd <pid> GC.run 

The collector tries to stay under the soft limit and uncommits what it frees. -Xmx remains the hard ceiling.

The test

I wanted to see what actually happens, so I ran it. The setup:

• Eclipse Temurin 21.0.13 in a Linux container on Docker Desktop, on cgroup v2 (/sys/fs/cgroup/memory.current was present).
• A small program that allocates and touches 300 MB, holds it for 10 seconds, then drops all references and idles.
• docker update --memory=300m --memory-swap=300m to lower the cgroup limit on the running container. This changes memory.max in place, which is the same kernel mechanism a pod resize uses, but it is not the kubelet.
• Two runs per collector. Naive: lower the limit straight away. Ordered: set SoftMaxHeapSize=100MB, run GC.run, wait for memory.current to fall, and only then lower the limit.
• Uncommit delays were shortened (ZUncommitDelay=1, ShenandoahUncommitDelay=1000) so the test finishes quickly. Defaults are longer.

Results:

With the ordered sequence, memory.max read back as 314572800 (300 MB) and the container never restarted.

What the kubelet adds

My test used Docker, so it had no kubelet. In a real cluster, the kubelet tries to avoid an OOM kill when a memory limit goes down: according to the Kubernetes docs, if current usage is above the requested limit, the resize is skipped and stays "In Progress". The docs call this best-effort, with no guarantee, and note a race where usage can spike right after the check.

For a JVM, that has a practical consequence. Committed heap counts as memory in use even when most of it is free, so the resize usually sits pending until the JVM gives memory back. Releasing first is what gets usage under the new limit, so the kubelet applies the resize. The ordered sequence works with the kubelet's check, not against it.

When RestartContainer is still the right answer

• You need the heap ceiling to change. -Xmx is fixed per process.
• You run G1 and haven't enabled periodic GC. G1 has no soft ceiling and returns heap mainly at a full GC or concurrent cycle, so release is slower and less predictable. I didn't test it here.
• You can accept the restart. A restart discards compiled code, loaded classes, pools and caches, and a service with few replicas loses capacity while it warms up. If that is fine for you, RestartContainer is simpler and needs no extra tooling.
• Your limit is already close to -Xmx plus non-heap memory. Shrinking safely needs room: the smallest limit must exceed -Xmx plus metaspace, code cache, thread stacks and direct buffers.

What this test does not show

• It is a synthetic program, not a production service, and I tested ZGC and Shenandoah on one JDK build.
• ZGC and Shenandoah used slightly different heap and limit sizes (600/900 MB and 500/700 MB). My first ZGC run at 500 MB ran out of heap while filling 300 MB, so I gave it more room.
• It was not run through a kubelet. The kubelet's check will change what you see on a real cluster.
• I haven't measured how long a release takes on a large heap under real load.

Reproduce it

You need Docker. Save the two files below in an empty folder.

Hold.java allocates and touches 300 MB, holds it for 10 seconds, then drops it:

import java.util.*; public class Hold { public static void main(String[] a) throws Exception { List<byte[]> l = new ArrayList<>(); for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); } System.out.println("allocated 300MB"); System.out.flush(); Thread.sleep(10000); l.clear(); l = null; System.out.println("released refs"); System.out.flush(); Thread.sleep(600000); } } 

run.sh starts a container, optionally releases memory first, then lowers the cgroup limit in place:

#!/bin/bash # usage: ./run.sh <naive|ordered> <jvm flags...> (run from the folder containing Hold.java) # Example: LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1 set -u mode=$1; shift; n=warden-test docker rm -f $n >/dev/null 2>&1 docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java docker run -d --name $n --memory=${LIM:-700}m --memory-swap=${LIM:-700}m -v "$PWD":/w -w /w \ eclipse-temurin:21-jdk java "$@" -Xmx${XMX:-500}m Hold >/dev/null for i in $(seq 1 60); do docker logs $n 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2 mem() { echo $(( $(docker exec $n cat /sys/fs/cgroup/memory.current) / 1048576 )); } echo "memory.current after the app dropped its data: $(mem) MB" if [ "$mode" = ordered ]; then docker exec $n jcmd 1 VM.set_flag SoftMaxHeapSize 104857600 # 100 MB soft ceiling docker exec $n jcmd 1 GC.run for i in $(seq 1 30); do [ "$(mem)" -lt 250 ] && break; sleep 1; done echo "memory.current after SoftMaxHeapSize + GC.run: $(mem) MB" fi docker update --memory=300m --memory-swap=300m $n # in-place cgroup change, no restart sleep 8 docker inspect -f 'running={{.State.Running}} OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}' $n docker rm -f $n >/dev/null 2>&1 

Then run the four cases:

chmod +x run.sh LIM=900 XMX=600 ./run.sh naive -XX:+UseZGC -XX:ZUncommitDelay=1 LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1 LIM=700 XMX=500 ./run.sh naive -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000 LIM=700 XMX=500 ./run.sh ordered -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000 

Expect OOMKilled=true exit=137 for the naive runs and running=true OOMKilled=false for the ordered ones. The exact megabyte numbers will differ a little on your machine.

FAQ

Do I need resizePolicy: RestartContainer for JVM memory?

Only if the heap ceiling has to move. A JVM fixes its maximum heap at startup, so growing beyond the -Xmx it launched with needs a restart. Shrinking, and growing back up to the launch ceiling, did not need one on ZGC or Shenandoah in my test, as long as the JVM released memory before the limit dropped.

Why doesn't raising the container's memory limit give the JVM more heap?

The JVM computes its maximum heap once, from -Xmx or from -XX:MaxRAMPercentage applied to the limit it saw at startup. A later in-place change to the cgroup limit does not change that number.

Can I change -Xmx on a running JVM?

No. What you can change on ZGC and Shenandoah is the soft ceiling, SoftMaxHeapSize, which sits at or below -Xmx.

What happens if I lower the memory limit without releasing memory first?

In my test the JVM was OOM-killed (exit code 137) every time, on both ZGC and Shenandoah. On a real cluster the kubelet makes a best-effort attempt to avoid this and may keep the resize pending while usage is above the new limit. The Kubernetes docs say this check gives no guarantees, because usage can spike right after it.

How do I make a JVM give memory back without restarting it?

On ZGC and Shenandoah, lower the soft ceiling and force a collection:

jcmd <pid> VM.set_flag SoftMaxHeapSize <bytes> jcmd <pid> GC.run 

Then check that the process's resident memory dropped before you lower the limit. The collector's uncommit delay applies: I shortened it for the test, and the defaults are longer.

Does this work with G1?

I didn't test it. G1 has no soft heap ceiling, so you cannot use the same flag. Plan for a slower, less predictable release, or use RestartContainer.

What is the default resizePolicy?

NotRequired for both CPU and memory, which means the change is applied without restarting the container. Setting RestartContainer for memory restarts the container, not the whole pod.

How small can the memory limit go?

The smallest limit must stay above -Xmx plus non-heap memory (metaspace, code cache, thread stacks, direct buffers) plus working-set headroom. For a typical Spring Boot service the non-heap part is commonly 100 to 150 MiB.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that runs this order for you: lower the soft ceiling, force a GC, verify the process's resident memory dropped, and only then shrink the pod in place. To grow, it reverses the order. It does not change -Xmx, so the same sizing rule applies. There is a guide with the full reasoning, and a walkthrough on a kind cluster.

The point of this article doesn't depend on any tool. The jcmd commands above work without it.