After upgrading from Java 8, G1 stopped returning memory

When you move from Java 8 to any of Java 11 through 25, the GC the JVM picks when you don't specify one (the default GC) changes from Parallel to G1. A GC is the mechanism that collects the memory of objects no longer in use. Parallel and G1 are names of two kinds of GC.

With this change, container memory (cgroup memory.current) no longer drops after a load peak passes.

I ran the same program under the same load on each version and compared how much container memory changed after the peak¹.

In this article, the committed heap is the amount of heap the JVM has taken from the OS, and shrinking means reducing the committed heap and returning that memory to the OS. Memory stays high not because of a leak, but because G1 does not shrink the committed heap while load continues.

You can tell whether memory that stays high after a peak is a leak or the default GC's behavior from the JVM metrics and GC logs you already collect, without touching production. If you still want it to drop, add a setting to G1 or switch the GC back to Parallel.

How GC works

Whether memory drops after a peak depends not on whether the GC ran, but on whether it shrank the committed heap.

Collecting vs. shrinking

The GC collects objects that are no longer in use and creates free space inside the heap. Collecting reduces the used heap, meaning the part of the committed heap that objects occupy. Whether the freed space is returned to the OS to shrink the committed heap is decided separately from collecting.

When a GC happens

The JVM splits the heap into a young generation and an old generation. Most new objects are placed in the young generation, and most of them become unused there. When the young generation fills up, a GC that collects only the young generation (young GC) runs. While the program keeps allocating, the young generation fills up again and again, so young GCs keep coming.

Objects that survive several young GCs are moved to the old generation. When the old generation fills up, the whole heap has to be collected. A GC that stops the program and collects the whole heap is called a full GC².

Two ways to pause

While the GC collects, the program is stopped, and requests that arrive in the meantime wait. GC designs differ in how they pause.

The name Parallel comes from using multiple threads in parallel to collect while the program is stopped. It does not mean Parallel runs concurrently with the program³.

To keep pauses short, G1 has a phase that scans live objects across the whole heap without stopping the program (concurrent marking). Concurrent marking ends with a short pause (Remark). It starts at the end of a young GC when old generation occupancy exceeds a threshold⁴.

After concurrent marking, G1 collects part of the old generation together with the young generation in the following young GCs (mixed GCs). G1 does a full GC only when this collection can't keep up and memory runs short⁵.

What changed

Upgrading from Java 8 changes two things: the default GC, and what triggers shrinking the committed heap.

The default GC

When no GC is specified, Java 8 picks Parallel, and Java 9 and later pick G1⁶. OpenJDK's stated reason for switching to G1 was that "limiting GC pause times is, in general, more important than maximizing throughput" for many users⁷.

Shrink triggers

Java 8's Parallel resizes the young generation on every young GC, and shrinks it⁸. Young GCs keep coming while load continues, so the trigger for shrinking keeps coming too.

Java 11's G1 shrinks the committed heap only after a full GC⁹. G1 does a full GC only when collection can't keep up, so this trigger almost never comes.

From Java 12, G1 also shrinks the committed heap at the end of concurrent marking¹⁰. But concurrent marking starts only when the old generation is filling up, so under a load where newly allocated objects quickly become unused, this trigger doesn't come either¹¹.

Periodic GC

According to the Java 12 change (JEP 346), G1 avoids full GCs as much as possible and starts concurrent marking only based on heap occupancy and allocation rate, so it often doesn't return heap memory to the OS unless prompted from outside.

This is a particular disadvantage in container environments where you pay for what you use, so Java 12 added a mechanism in which G1 starts concurrent marking on its own and shrinks the committed heap when no GC has happened for a while¹². This is called periodic GC.

At a set interval, G1 checks the time since the last GC. If at least one interval has passed, it starts concurrent marking and shrinks at the Remark that ends it. If not, it skips¹³. Despite the name, it does not necessarily run a GC every interval.

For example, with a 10-second interval, if objects keep being allocated and young GCs happen less than a second apart, 10 seconds never pass since the last GC. During that time, periodic GC never runs.

The interval is set in milliseconds with G1PeriodicGCInterval. Setting it to 0 disables periodic GC. The default is 0, so periodic GC doesn't run unless you set it¹⁴.

With periodic GC enabled

With periodic GC enabled, it is skipped while load continues, and runs after load stops to shrink the committed heap.

I simulated two situations after a traffic surge subsides and compared them¹⁵.

Both ran Java 25 with -XX:G1PeriodicGCInterval=10000, a 10-second interval.

While load continues

When load continues after the peak, the check at 10 seconds after startup writes the following line to the GC log. This line appears when -Xlog:gc+periodic=debug is set.

[10.017s][debug][gc,periodic ] Last GC occurred 505ms before which is below threshold 10000ms. Skipping. 

The last GC was 505ms earlier, short of the 10000ms interval, so periodic GC was skipped¹³.

In the 45 seconds after the peak, the check ran 5 times, was skipped all 5 times, and periodic GC never ran. The committed heap didn't change from the end of the peak to the end of the run.

After load stops

When nothing was allocated after the peak, periodic GC ran at the check 20 seconds after startup. At the 10-second check, 10 seconds had not yet passed since the GC at the end of the peak. First, a line for the young GC that starts concurrent marking appears.

[20.016s][info ][gc ] GC(46) Pause Young (Concurrent Start) (G1 Periodic Collection) 195M->37M(377M) 1.104ms 

(G1 Periodic Collection) marks this as a GC started by periodic GC. 195M->37M is the used heap before and after, and the 377M in parentheses is the committed heap¹⁶. This young GC does not change the committed heap.

On the Remark line that follows, the committed heap in parentheses is 27M¹⁰.

[20.019s][info ][gc ] GC(47) Pause Remark 37M->7M(27M) 0.478ms 

The committed heap seen in JVM metrics (the sum of committed across the heap memory pools, read via MemoryPoolMXBean) dropped by the same amount¹⁷.

The container memory compared at the start (memory.current, sampled every second) dropped too. It went from 393.9 to 421.4MB at the sample just before periodic GC to 53.8 to 59.5MB at the last sample.

Diagnosing your own environment

Use the JVM metrics and GC logs you already collect. For GC logs, the lines printed with -Xlog:gc are enough.

Which GC was selected

In JVM metrics, the heap shows up as several areas, and their names depend on the GC. These areas are called memory pools¹⁸.

The pool name is in the id tag of jvm.memory.committed in Micrometer, and in the pool label of jvm_memory_pool_committed_bytes in the Prometheus Java client¹⁹. The names tell you which GC was selected in production.

If you see the Parallel pools, the behavior described here doesn't apply. If you see neither, see the "Scope" section.

Is it a leak?

When memory doesn't drop, a leak is the first suspect. With a leak, the amount of live objects keeps growing over time even under constant load²⁰. With the behavior described here, live objects stay few and only the committed heap is high.

To tell them apart, compare the committed heap with the used heap after GC. Read the committed heap as the sum of committed across the heap pools. Read the used heap after GC from the after-GC value on each GC line in the GC log. A pool's used value is not suitable for this comparison, because the Eden part goes up and down with every GC.

On Java 11 with continued load, the first young GC line after the peak was²¹:

[6.877s][info ][gc ] GC(27) Pause Young (Normal) (G1 Evacuation Pause) 679M->89M(989M) 1.773ms 

On this line, the 89M after the arrow is the used heap after GC, and the 989M in parentheses is the committed heap. Most of the committed heap is free, and collection is keeping up. The freed space just isn't being returned to the OS.

Live objects can't exceed the used heap after GC, so if the used heap after GC isn't growing, you don't need to suspect a leak.

However, a young GC collects only the young generation, so garbage piled up in the old generation stays in the used heap after a young GC. This value can creep up even without a leak.

If it is growing, the behavior described here doesn't explain it on its own. If you have values after a GC that also collects the old generation (a full GC, or a mixed GC after concurrent marking), look at those. If they keep growing too, or if no such GC has happened, investigate it as a leak²⁰.

GCs that trigger shrinking

Next, look in the GC log after the peak for lines from GCs that trigger shrinking the committed heap. The line names are the same in Java 11 and Java 25²². Java 11 also prints Pause Remark, but it doesn't shrink the committed heap there.

When concurrent marking starts, a Pause Young (Concurrent Start) line also appears before Pause Remark. If none of these appear, no shrink trigger has come. On Java 11, that settles it. On Java 12 and later, the remaining trigger is periodic GC.

Is periodic GC running?

On Java 12 and later, if there is no Pause Remark, periodic GC isn't running either. Periodic GC starts concurrent marking, so when it runs, Pause Remark appears.

If it isn't running, either the interval isn't set, or it is set but the check is skipped every time because load continues (see "With periodic GC enabled"). Besides the startup option, the interval can also be set on a running JVM with jcmd¹⁴.

If all of the following hold, memory isn't dropping because G1 isn't getting a trigger to shrink the committed heap, not because of a leak.

• The pools are G1's
• The used heap after GC isn't growing
• There is no Pause Full after the peak, and on Java 12 and later, no Pause Remark either

Fixes

If it isn't a leak but you still want memory to drop, either keep G1 and add a trigger to shrink when load stops, or go back to a GC that shrinks on every young GC.

Which to choose depends on what you found in "Diagnosing your own environment" and on whether there are periods when load stops.

Check for periods when load stops using the timestamp at the start of each GC log line. If there are periods with long gaps between GCs, periodic GC can run there.

Periodic GC runs when, at an interval check, at least one interval has passed since the last GC. Because the check comes only once per interval, it can take up to twice the interval from when load stops until it runs. Set the interval to less than half the length of the periods when load stops¹³.

Going back to Parallel makes each pause longer. How much longer depends on heap size and the amount of live objects. Before production, run the same load in a test environment with -XX:+UseParallelGC and compare the pause times at the end of each GC log line with those under G1.

Scope

The tests in this article ran in a 2GB, 2-CPU container with no GC specified. The committed heap's upper limit is set by -Xmx, and its value at startup by -Xms. The tests did not set -Xms. The load was one where newly allocated objects quickly became unused. The conclusions don't apply in the following cases.

You can tell Serial was selected if the memory pool names are Eden Space, Survivor Space, and Tenured Gen³¹, or if the following line appears in the GC log at startup³².

[0.003s][info][gc ] Using Serial 

Java 15 and later, 8u372 and later, and 11.0.16 and later can read cgroup v2 container limits. Earlier versions pick the GC based on the host's CPU and memory³³.

1. Docker on a GitHub Actions runner, with a 2GB, 2-CPU container and -Xmx1500m. A 5-second peak (simulating a sudden surge in traffic by allocating 1000MB of objects per second) was followed by 45 seconds of normal load, allocating 200MB of objects per second. Ranges are over 3 runs per version. Scripts and results are in show-your-work and the GitHub Actions run. ↩

2. Oracle's HotSpot Virtual Machine Garbage Collection Tuning Guide, Generations (Java 25). ↩

3. The same Oracle guide (Java 25): The Parallel Collector, Garbage-First (G1) Garbage Collector, and Selecting a Collector in Available Collectors. ↩

4. In Java 25, the code that decides at the end of a young GC whether to start concurrent marking. ↩

5. The same Oracle guide (Java 25), Garbage-First (G1) Garbage Collector: the Space-Reclamation Phase and the description of Full GC that follows. ↩

6. When Java 8 is started with -XX:+PrintCommandLineFlags, the flags line the JVM prints includes -XX:+UseParallelGC. Java 11 and Java 25 print a Using G1 line at the top of the GC log (in Java 11, [0.004s][info ][gc ] Using G1). ↩

7. The Motivation section of JEP 248: Make G1 the Default Garbage Collector. ↩

8. Java 8's Parallel resizes the young generation on every young GC. ↩

9. Java 11's G1 resizes the committed heap as part of a full GC. ↩

10. The change that made Remark shrink the heap is JDK-6490394 (Java 12). In Java 25, the committed heap is resized inside Remark. ↩

11. With 200MB of objects allocated per second after the peak, concurrent marking did not start after the peak on any of Java 11, 17, 21, or 25. ↩

12. The Motivation section of JEP 346: Promptly Return Unused Committed Memory from G1. ↩

13. Java 25's periodic GC check and the flag that makes it start concurrent marking by default (G1PeriodicGCInvokesConcurrent). ↩

14. The definition of G1PeriodicGCInterval. It is a MANAGEABLE flag, so it can be changed on a running JVM. ↩

15. 2GB, 2-CPU container, -Xmx1500m, no GC specified. In the load-continues runs, objects were allocated for 45 seconds after the peak. GC logs were captured with -Xlog:gc* and -Xlog:gc+periodic=debug. The committed heap table shows the sample just before periodic GC and the first sample after it. ↩

16. How to read GC log lines ("used before GC" -> "used after GC" ("heap size")) in Generations in Oracle's GC tuning guide (Java 25). ↩

17. The same test was run 3 times. Values shown as "X to Y" are the range across the 3 runs. ↩

18. The pool names are defined in the source of G1 and Parallel. ↩

19. Both read values from MemoryPoolMXBean. JvmMemoryMetrics in Micrometer 1.13.0, JvmMemoryMetrics in the Prometheus Java client 1.3.1. ↩

20. Oracle's Troubleshooting Guide, Troubleshoot Memory Leaks (Java 25). ↩

21. The first young GC line after the peak, under the condition where 200MB of objects per second kept being allocated after the peak. ↩

22. Java 11's Pause Full, Pause Young (Concurrent Start), and Pause Remark. Java 25's Pause Full. The other two for Java 25 appear in the logs in "With periodic GC enabled". ↩

23. Java 11's Parallel also resizes the young generation on every young GC. With -XX:+UseParallelGC on Java 11 and 200MB of objects per second allocated after the peak, the committed heap went from 369.0 to 380.0MB at the end of the peak to 103.5 to 120.5MB at the last sample, and container memory (memory.current) dropped by 141.4 to 145.5MB. ↩

24. Parallel resizes during young GC on both Java 8 and Java 11. ↩

25. On Java 11, setting it makes the JVM print Unrecognized VM option 'G1PeriodicGCInterval=10000' and fail to start. Periodic GC is available from Java 12. ↩

26. Only when there are 2 or more CPUs and memory is at least 2GB minus 256MB does the JVM select G1 on Java 9 and later or Parallel on Java 8. Java 8's check is the same. ↩

27. -Xms sets both the committed heap at startup and its lower bound (MinHeapSize). G1 does not shrink below MinHeapSize. ↩

28. G1 checks old generation occupancy at the end of a young GC to start concurrent marking, and shrinks at Remark. ↩

29. Java 26's G1 (JDK-8238687) and Parallel (JDK-8338977). ↩

30. JEP 523. ↩

31. Serial's memory pools. ↩

32. On Java 25, this line appeared both when only the container memory was set to 1GB and when only the CPU count was set to 1. ↩

33. The change to read cgroup v2 limits landed in Java 15 and was backported to 11.0.16 and 8u372. ↩