Our quote pricing service is a Spring Boot application on Kubernetes. In August we gave it a horizontal pod autoscaler with two targets: seventy percent CPU and seventy percent memory, between four and twenty replicas. Memory was there because the service had once run out of it, and scaling on it felt like a sensible safety net.

On the first busy Monday it scaled out to twenty, which was right. It was still at twenty on Tuesday, and on Saturday, and three weeks later, with CPU around fifteen percent and traffic at overnight levels. The cluster autoscaler had kept the nodes to match.

The autoscaler computes the replicas each metric wants as the current replicas times the current utilisation divided by the target, rounded up, and when there are several metrics it takes the largest answer. Memory utilisation is the pod's working set divided by its memory request. CPU was asking for four replicas. Memory was asking for twenty or more, every time it was asked.

The JVM sizes its heap for the work it has seen. Under Monday's load each pod's heap grew towards its maximum, and the garbage collector keeps that memory committed afterwards rather than handing it back to the operating system. It rarely gives committed heap back without settings we had never touched. So from the container's side, a pod handling two requests a second used almost the same memory as one handling two hundred. The autoscaler read that as load, and memory alone was enough to hold every replica in place.

Nothing had malfunctioned. The autoscaler was scaling on a number that only ever went up, and doing it correctly.

We removed the memory target. The service now scales on CPU and on requests per second per pod, which it exports and which falls when the traffic does. We set the initial and maximum heap to the same value, so memory use is flat from start to finish and nobody is tempted to read meaning into it. A fixed heap that never changes is a capacity setting, not a load signal. Memory pressure is now covered by an alert on heap use after garbage collection, which is the number that actually predicts running out.

The rule we wrote down: before you scale on a metric, find out whether it can come back down.

– Sergey Shinder