In short. A container memory limit covers everything the JVM uses, and the heap is only part of it. Metaspace, thread stacks, the code cache and direct buffers live outside the heap, so a pod with -Xmx close to its limit is killed while the heap still has room. Size the limit as heap plus non-heap, check which GC and how many CPUs the JVM really sees, and add a startup probe. jvm-fit does that arithmetic from the manifest before you deploy.

The pod restarts with exit code 137. The dashboard shows the heap at 50 or 60 percent. Nothing in the application log, because the kernel killed the process and the JVM never got to write an OutOfMemoryError. This is one of the most common ways Java breaks on Kubernetes, and the cause is almost always visible in the manifest.

Why is the pod killed when the heap isn't full?

Because the limit is on the container, and the heap is one of several things in it. The JVM also needs memory for:

  • Metaspace: class metadata. Grows with the number of loaded classes, which in a Spring Boot app is a lot.
  • Thread stacks: each thread reserves its own stack.
  • Code cache: machine code produced by the JIT compiler.
  • Direct buffers: off-heap memory used by NIO and Netty-based clients and servers.
  • The GC's own structures and other native memory.

None of that is counted by -Xmx. If the heap is allowed to take 900Mi of a 1Gi container, everything else has to fit in what is left, and it doesn't.

Memory budget container limit 1024Mi heap max (-Xmx) 900Mi non-heap estimate ~350Mi ----------------------------- needed ~1250Mi over limit by ~226Mi

The JVM does not know it is over budget. It grows into memory it believes it has, the container crosses its limit, and the kernel's OOM killer ends it.

A real run: 51% heap, exit code 137

I reproduced it on a kind cluster with a small Spring Boot service made for the purpose. It keeps a fixed 260Mi in the heap and 140Mi in direct buffers, which is what NIO clients do.

Before After the fix
Memory limit 512Mi 832Mi
Heap setting -Xmx450m sized to leave room for non-heap
Result under the same load OOMKilled, heap at 51% in the last log line 0 restarts, peak 670Mi

"Stable" here means no restarts for a minute after full load, not a soak test. The point is the first column: a heap that was half empty, and a pod that died anyway.

Three container defaults that are easy to get wrong

I wanted the rules in jvm-fit to match what the JVM actually does, so a test starts Temurin 11, 17 and 21 in Docker with different limits and reads back the heap size, CPU count and GC. Three results are easy to get wrong.

Container What you might expect What the JVM does
2 CPUs, 1Gi G1, because there are two cores SerialGC. G1 also needs about 1792Mi of memory
256Mi, no heap flags A 64M heap, 25% of the limit A 126M heap. Small limits have a floor
1Gi with -XX:-UseContainerSupport Roughly the same as without it A heap sized from the host's memory, not the container's

The first one matters most in practice. A service with a 1 CPU limit or a small memory limit runs on a single-threaded collector, and nobody chose that. The app is slow, CPU usage looks low, and the cause is one line in the pod spec.

The other default worth knowing: with no -Xmx and no MaxRAMPercentage, HotSpot gives the heap 25% of the limit. In a 1Gi container that is 256M, and the rest of the memory you pay for sits idle. OpenJ9 behaves differently: the same 1Gi container got a 512M heap.

How much memory should a Java container get?

Work it out as a sum, not a percentage:

  • Start from the heap the app needs, taken from real usage if you have it.
  • Add non-heap. About 350Mi is a typical figure for a Spring Boot service, more if it uses many threads or large direct buffers. This is an estimate, so measure your own app when you can.
  • Set the limit to the sum, with some headroom.
  • Then express the heap as MaxRAMPercentage so it follows the limit when someone changes it. Values above 80 rarely leave enough room.

There is no single correct MaxRAMPercentage. Non-heap memory does not shrink when the limit does, so 75% is comfortable in a 4Gi container and too much in a 512Mi one.

What else to check in the manifest

  • A memory limit exists. Without one the JVM takes a percentage of the whole node.
  • The CPU limit. Below 1 core means throttling; below 2 means SerialGC unless you set a collector yourself.
  • ActiveProcessorCount is not higher than the CPU limit.
  • A startup probe. A liveness probe with a short initial delay kills a JVM that is still starting.
  • Memory request equals the limit, so the scheduler places the pod where the memory really is.

Checking it before deploy

To stop doing this arithmetic by hand, I wrote jvm-fit. It reads the pod spec and the JVM flags together, from plain manifests, Helm charts or Kustomize overlays, and prints the budget and a fix for each finding.

go install github.com/andronaft/jvm-fit/cmd/jvm-fit@latest jvm-fit scan ./k8s/ # files and directories jvm-fit scan ./charts/payments -f values.yaml # Helm chart jvm-fit explain JVM007 # why it matters, how to fix it

It collects flags in the order the JVM applies them (JAVA_TOOL_OPTIONS, then JDK_JAVA_OPTIONS, then the command line), has 12 rules, and outputs text, JSON or SARIF for CI.

What it doesn't do

It is a static check. It does not see how much memory your app really uses, and the non-heap figure is an estimate you can override. Tools like KRR and Goldilocks answer the other half of the question from real usage: they tell you how much the app needs, and jvm-fit checks that the JVM flags fit the limit you chose. Reading live clusters and Prometheus metrics is planned for the next version.

Source and docs: github.com/andronaft/jvm-fit (Apache 2.0) · the measurements: results on real JDKs · the demo: OOMKilled with a half-empty heap

Originally published at zhukovskyi.space.