HPA Not Scaling: Diagnosing metrics-server, Thresholds, and Stabilization Behavior

A HorizontalPodAutoscaler stuck at one replica while your pods are visibly saturated is one of the most common — and most misdiagnosed — incidents in a Kubernetes platform. The usual reaction is to lower the CPU target, add replicas manually, or blame "the autoscaler". None of that fixes the root cause.

In practice, an HPA that does not scale almost always falls into one of four buckets: the metrics pipeline is broken, the workload has no resource requests, the target threshold is mathematically unreachable, or the scaling behavior (stabilization window, policies) is deliberately throttling the decision. This article walks through each of them with real commands, then shows how to move past CPU/memory with custom metrics and KEDA.

Context: why the HPA looks broken when it is not

The HPA controller runs inside kube-controller-manager and reconciles every 15 seconds by default (--horizontal-pod-autoscaler-sync-period). On each loop it:

1. Fetches the current metric value for the pods selected by the target workload's selector.
2. Computes a desired replica count with a simple ratio.
3. Applies the behavior policies and stabilization windows.
4. Writes the result to the scale subresource of the Deployment / DeploymentConfig / StatefulSet.

The formula is intentionally boring:

desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue)) 

With a 10% tolerance by default: if the ratio is between 0.9 and 1.1, the HPA does nothing at all. That tolerance alone explains a lot of "it should have scaled" tickets.

Every step above can fail silently from the user's point of view. The HPA object stays Active, kubectl get hpa prints something, and nobody reads the conditions. So start there.

Step 1: read the HPA conditions properly

kubectl get hpa is almost useless for diagnosis. It shows a target column that displays <unknown> on failure, but nothing about why.

kubectl get hpa -n payments # NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE # api-hpa Deployment/api <unknown>/70% 1 10 1 3h 

The real information lives in the conditions and events:

kubectl describe hpa api-hpa -n payments 

Focus on three conditions:

Typical failure messages and what they actually mean:

ScalingActive False FailedGetResourceMetric unable to get metrics for resource cpu: no metrics returned from resource metrics API 

→ metrics-server is absent, unhealthy, or the pods are too young (no metric window yet).

ScalingActive False FailedGetResourceMetric missing request for cpu in container app of Pod api-7d9... 

→ The container has no resources.requests.cpu. A utilization-based HPA is mathematically impossible without a request, because utilization is usage / request.

AbleToScale True ReadyForNewScale ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count 

→ Nothing is wrong; the HPA simply wants fewer pods than minReplicas.

Also check the structured status, which is easier to parse in automation:

kubectl get hpa api-hpa -n payments -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' 

Step 2: verify the metrics pipeline

Is the metrics API actually served?

kubectl get apiservices v1beta1.metrics.k8s.io # NAME SERVICE AVAILABLE AGE # v1beta1.metrics.k8s.io openshift-monitoring/... True 210d kubectl top nodes kubectl top pods -n payments 

If kubectl top fails, the HPA will never work for resource metrics. On vanilla Kubernetes, metrics-server may simply not be installed:

kubectl -n kube-system get deploy metrics-server kubectl -n kube-system logs deploy/metrics-server --tail=50 

Common metrics-server failure: TLS verification against kubelet serving certificates signed by a cluster-internal CA.

unable to fully scrape metrics: unable to extract connection information for node x509: cannot validate certificate for 10.0.3.14 because it doesn't contain any IP SANs 

The correct fix is to enable kubelet serving certificate rotation and approve the CSRs, not to disable verification. If you must run in a lab, the insecure flag exists — but never in production:

# metrics-server args (lab only for --kubelet-insecure-tls) apiVersion: apps/v1 kind: Deployment metadata: name: metrics-server namespace: kube-system spec: template: spec: containers: - name: metrics-server args: - --cert-dir=/tmp - --secure-port=10250 - --kubelet-preferred-address-types=InternalIP,Hostname - --metric-resolution=15s # - --kubelet-insecure-tls # NEVER in production 

On OpenShift 4, metrics-server is managed by the Cluster Monitoring Operator (it replaced Prometheus Adapter as the resource metrics provider in recent releases). You do not install it yourself. If oc adm top pods fails, investigate the monitoring stack instead:

oc get clusteroperator monitoring oc -n openshift-monitoring get pods -l app.kubernetes.io/name=metrics-server oc adm top pods -n payments 

Are metrics fresh enough?

The HPA ignores metrics older than --horizontal-pod-autoscaler-cpu-initialization-period (5 min default) for freshly started pods, and treats pods that are not Ready as unknown. A deployment that restarts constantly will keep the HPA in a permanent "unknown metrics" state.

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/payments/pods" | jq '.items[] | {name:.metadata.name, cpu:.containers[].usage.cpu}' 

Step 3: fix requests and thresholds

No request, no utilization HPA

apiVersion: apps/v1 kind: Deployment metadata: name: api namespace: payments spec: replicas: 1 selector: matchLabels: app: api template: metadata: labels: app: api spec: containers: - name: app image: quay.io/acme/api:1.14.2 ports: - containerPort: 8080 resources: requests: cpu: 200m memory: 256Mi limits: memory: 512Mi 

Note the deliberate absence of a CPU limit: CFS throttling caps usage below the HPA target and can make a saturated pod look idle. If you must set a CPU limit for quota or compliance reasons, keep it well above the request so throttling does not mask load.

Sidecars poison container-less targets

With type: Resource, the HPA sums requests across all containers in the pod. A service mesh sidecar requesting 100m next to an app requesting 100m halves your apparent utilization. Use type: ContainerResource to scope the metric:

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api-hpa namespace: payments spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 2 maxReplicas: 20 metrics: - type: ContainerResource containerResource: name: cpu container: app target: type: Utilization averageUtilization: 70 

An unreachable threshold

If your app is I/O bound and never exceeds 25% CPU, a 70% target means the HPA will never scale up no matter how slow the responses are. Check the real distribution before choosing a target:

kubectl top pods -n payments --containers --sort-by=cpu 

If CPU is not the bottleneck, stop using CPU as the trigger. Jump to the custom metrics section.

Step 4: understand behavior and stabilization windows

This is where "the HPA refuses to scale" is actually correct behavior. Defaults in autoscaling/v2:

• Scale up: stabilization window 0s, policies allow +100% or +4 pods per 15s, whichever is higher.
• Scale down: stabilization window 300s, policy allows -100% per 15s.

The scale-down stabilization window makes the controller pick the maximum desired replica count observed over the last 5 minutes. So after a spike, you wait 5 minutes before the first pod is removed. That is intentional and usually right.

The symptom people hit: someone set a long scale-up stabilization window to "avoid flapping", and now scale-up takes minutes.

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api-hpa namespace: payments spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 2 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 behavior: scaleUp: stabilizationWindowSeconds: 0 selectPolicy: Max policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 4 periodSeconds: 15 scaleDown: stabilizationWindowSeconds: 300 selectPolicy: Min policies: - type: Pods value: 1 periodSeconds: 60 

Read selectPolicy carefully. selectPolicy: Min on scaleUp is a classic foot-gun: it picks the most restrictive policy, so a Pods: 1 / 60s policy will add one pod per minute even during a traffic surge. And selectPolicy: Disabled on scaleUp turns the HPA into a scale-down-only controller — perfectly valid YAML, completely silent failure mode.

Watch the decisions live:

kubectl get events -n payments --field-selector involvedObject.name=api-hpa --sort-by=.lastTimestamp # ... SuccessfulRescale New size: 4; reason: cpu resource utilization above target 

Also check for conflicts: a GitOps controller (Argo CD, Flux) that owns spec.replicas in Git will fight the HPA every sync. Either remove replicas from the manifest or set an ignore rule.

# Argo CD Application excerpt spec: ignoreDifferences: - group: apps kind: Deployment jsonPointers: - /spec/replicas 

Step 5: move to custom metrics with KEDA

CPU is a proxy, rarely the actual signal. Queue depth, request rate, p99 latency, or in-flight connections describe saturation far better. KEDA (the Custom Metrics Autoscaler Operator on OpenShift) generates a standard HPA under the hood and feeds it external metrics.

Install on OpenShift via OperatorHub, then create the controller:

apiVersion: keda.sh/v1alpha1 kind: KedaController metadata: name: keda namespace: openshift-keda spec: watchNamespace: "" operator: logLevel: info metricsServer: logLevel: "0" 

A Prometheus-driven ScaledObject scaling on HTTP request rate:

apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: api-scaler namespace: payments spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicaCount: 2 maxReplicaCount: 30 pollingInterval: 15 cooldownPeriod: 300 advanced: horizontalPodAutoscalerConfig: behavior: scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 2 periodSeconds: 60 triggers: - type: prometheus metadata: serverAddress: https://thanos-querier.openshift-monitoring.svc:9091 query: | sum(rate(http_requests_total{namespace="payments",service="api"}[2m])) threshold: "50" namespace: payments authenticationRef: name: keda-thanos-auth 

Authentication to Thanos uses a dedicated ServiceAccount token, never a static credential:

apiVersion: v1 kind: ServiceAccount metadata: name: keda-thanos namespace: payments --- apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: keda-thanos-auth namespace: payments spec: boundServiceAccountToken: - parameter: bearerToken serviceAccountName: keda-thanos 

Grant the minimum read access:

oc adm policy add-cluster-role-to-user cluster-monitoring-view \ -z keda-thanos -n payments 

Verify KEDA produced a working HPA:

oc get scaledobject api-scaler -n payments oc get hpa keda-hpa-api-scaler -n payments oc describe scaledobject api-scaler -n payments | grep -A10 Conditions oc -n openshift-keda logs deploy/keda-operator --tail=100 | grep -i error 

A frequent KEDA gotcha: with minReplicaCount: 0, the ScaledObject scales to zero and the underlying HPA is removed while idle. If you then run kubectl get hpa and see nothing, that is scale-to-zero working, not a bug.

Never combine a hand-written HPA and a ScaledObject on the same Deployment — two controllers writing the same scale subresource produce oscillation. KEDA refuses with ScaledObject doesn't have correct scaleTargetRef or reports ownership conflicts.

Security best practices

• Scope RBAC for metric readers. cluster-monitoring-view is cluster-wide read on metrics. Prefer a namespaced Role when your Prometheus setup allows it, and always use a dedicated ServiceAccount per ScaledObject rather than default.
• Prefer bound ServiceAccount tokens (boundServiceAccountToken in TriggerAuthentication) over long-lived Secrets. They are short-lived, audience-scoped, and auto-rotated. If an external system (SQS, Kafka SASL) forces a static credential, store it in a Secret referenced by TriggerAuthentication and rotate it, or use a Secrets Store CSI driver.
• Never disable kubelet TLS verification to "fix" metrics-server. Enable RotateKubeletServerCertificate and approve the serving CSRs instead; --kubelet-insecure-tls lets anything on the node network impersonate a kubelet and feed your autoscaler fabricated metrics.
• Treat maxReplicas as a blast-radius control. An autoscaler driven by an attacker-influenced metric (public request rate, unauthenticated queue) is a cost-amplification vector. Pair maxReplicas with a ResourceQuota and a LimitRange in the namespace.
• Protect the metric source. If a ScaledObject queries a Prometheus endpoint reachable without auth, anyone able to push series can drive your scaling. Query Thanos Querier with the tenancy port (9092) when you need per-namespace enforcement.
• Audit HPA mutations. kubectl edit hpa raising maxReplicas from 10 to 500 is a legitimate-looking change with a real financial impact. Keep HPAs and ScaledObjects in Git with review, and disable direct edits via RBAC in production namespaces.

Conclusion: key takeaways

An HPA pinned at one replica is a diagnosis problem, not an autoscaling problem. Work the chain in order:

1. Read the conditions, not the TARGETS column. ScalingActive=False with FailedGetResourceMetric points to metrics; ScalingLimited=True points to your bounds.
2. Validate the metrics pipeline with kubectl top. No top, no HPA. On OpenShift, that means checking the monitoring cluster operator, not installing metrics-server by hand.
3. Set CPU requests, and scope the metric with ContainerResource when sidecars are present. Beware CPU limits that throttle a pod into looking idle.
4. Audit behavior. Default scale-down stabilization is 300s by design. A non-zero scaleUp.stabilizationWindowSeconds, a selectPolicy: Min, or Disabled is usually the actual culprit.
5. Check for an HPA/GitOps conflict on spec.replicas, and never run two controllers against one scale subresource.
6. Graduate to KEDA when CPU is not the bottleneck — queue depth and request rate are honest signals, and KEDA still gives you the full behavior block through advanced.horizontalPodAutoscalerConfig.

Finally, test it. Generate load, watch kubectl get hpa -w and the SuccessfulRescale events, and record how long the first new pod takes to become Ready. An autoscaler you have never observed scaling is an autoscaler you cannot trust during an incident.