MinIO memory usage high on Kubernetes but never OOMKilled? Why the working set sits at 95% of the limit

The 512 MiB cgroup on 2026-11-15 at 13:49 UTC: full, with most of the top made of page cache the kernel can reclaim.

If your MinIO memory usage is high on Kubernetes (kubectl top shows 495Mi against a 512Mi limit, an audit reports 7d memory working set p95 / limit = 0.95) but you never see a MinIO pod OOMKilled (container_oom_events_total is still 0), MinIO is probably not leaking: community releases read the container's own cgroup memory.max and use 90 % of it as their budget (since PR #19839, May 2024, this includes the Go soft memory limit), so the memory fills the limit by design. To see whether yours is the same case, read the pod's cgroup (step 2 below): it is the same case when memory.current is within a few MB of memory.max, anon stays under 90 % of memory.max, and memory.events shows oom_kill 0. The cluster is a 2-node k3s cluster that I run (server-1 and agent-1); MinIO runs as a single-replica StatefulSet minio, pod minio-0, one drive, image minio/minio:RELEASE.2025-04-22T22-12-26Z. All times are UTC, 2026-11-15 unless a date is given.

What we saw: MinIO memory usage high on Kubernetes, 495Mi of 512Mi for 30 days, no MinIO pod OOMKilled

The first signal was an automated audit of the cluster (kubegap, described under the checks below), run on 2026-11-14 at 16:25:29. Its problem 5, translated from Korean: "minio's memory is about to reach its limit." The reason it gave: a small rise in requests ends in OOMKilled, and with one replica the object store stops. The finding was resources.memory-limit-near-p95, pod minio-0: 7d memory working set p95 / limit = 0.95, with a KRR recommendation (KRR computes resource recommendations from Prometheus history) of about 582 MB for the memory request.

The pod at 13:47:18 (names masked, pod IP and container id removed):

$ kubectl -n apps get pod minio-0 -o wide; kubectl -n apps get pod minio-0 -o jsonpath='{.status.qosClass}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}{.status.containerStatuses[0].lastState}{"\n"}{.spec.containers[0].resources}{"\n"}' NAME READY STATUS RESTARTS AGE NODE minio-0 1/1 Running 2 (36h ago) 128d server-1 Burstable 2 {"terminated":{"exitCode":255,"finishedAt":"2026-11-14T01:12:30Z","reason":"Unknown","startedAt":"2026-07-11T00:26:47Z"}} {"limits":{"cpu":"300m","memory":"512Mi"},"requests":{"cpu":"50m","memory":"128Mi"}} $ kubectl top pod -n apps minio-0 --containers POD NAME CPU(cores) MEMORY(bytes) minio-0 minio 30m 495Mi 

The last restart (exit 255, reason Unknown) was the k3s restart of 2026-11-14, not memory. The pod is pinned to server-1 by nodeSelector storage: server-1; server-1 has 12 CPUs and 32207388Ki of memory, with 23638Mi (75 %) requested.

Prometheus at 13:47:29, in MiB (two values where the current and the previous container each have a series):

container_memory_working_set_bytes now 495.1 max_over_time [30d] 498.1 / 505.9 quantile_over_time(0.95) [7d] 494.5 / 486.8 container_memory_rss now 395.6 max_over_time [30d] 413.5 / 440.4 quantile_over_time(0.95) [7d] 395.9 / 434.6 container_memory_cache now 82.3 container_memory_usage_bytes now 510.3 max_over_time [30d] 511.6 / 511.7 container_oom_events_total 0 increase(container_oom_events_total[30d]) 0 / 0 container_memory_failcnt 0 kube_pod_container_status_last_terminated_reason reason="Unknown" 1 

Daily maxima over 30 days (13:48:02, MiB):

container_memory_rss (max per day) 10-16=435 10-17=439 10-18=436 10-19=434 10-20=437 10-21=437 10-22=439 10-23=440 10-24=436 10-25=439 10-26=438 10-27=439 10-28=438 10-29=439 10-30=440 10-31=438 11-01=440 11-02=440 11-03=438 11-04=437 11-05=438 11-06=438 11-07=438 11-08=440 11-09=440 11-10=440 11-11=440 11-12=438 11-13=438 11-14=439 11-15=421 container_memory_working_set_bytes (max per day) 10-16=503 10-17=503 10-18=500 10-19=502 10-20=502 10-21=503 10-22=502 10-23=502 10-24=502 10-25=503 10-26=504 10-27=504 10-28=503 10-29=504 10-30=505 10-31=505 11-01=505 11-02=504 11-03=506 11-04=506 11-05=506 11-06=504 11-07=502 11-08=492 11-09=494 11-10=496 11-11=491 11-12=489 11-13=491 11-14=491 11-15=499 container_memory_cache (max per day) between 55 and 70 every day, 152 on 11-14 (the restart day), 90 on 11-15 

anon's daily maximum was 434–440 MiB every day for 30 days (421 on 11-15, after the restart of 11-14). A leak grows; this did not.

Prometheus, 30 days, captured after the change (values in decimal MB). Flat for a month at 512Mi; the restart of 11-14 and the 768Mi pod and its load test are at the right edge.

Inside the cgroup at 13:49:02. The recorded command was step 2's command with sock and kernel added to the memory.stat filter and a check of environment variable names appended. Below is step 2's command and the part of that output it prints: the same lines without sock 0 (the kernel printed no kernel line) and without the environment check.

$ kubectl -n apps exec minio-0 -- sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) current=$(cat /sys/fs/cgroup/memory.current); cat /sys/fs/cgroup/memory.events; while read k v; do case $k in anon|file|active_file|inactive_file|slab) echo "$k $v";; esac; done < /sys/fs/cgroup/memory.stat' max=536870912 current=535588864 low 0 high 0 max 0 oom 0 oom_kill 0 anon 415117312 file 86994944 inactive_file 30425088 active_file 56569856 slab 31441592 

memory.current was 1.3 MB below memory.max (512 MiB). Of that, anon (memory the process allocated) was 395.9 MiB, file (page cache) 83.0 MiB (53.9 active, 29.0 inactive) and slab 30.0 MiB. oom_kill was 0. The same recording printed every environment variable name in the container that matched GOMEMLIMIT, GOGC or MINIO_API*, and it printed none, so none of them was set. The only memory alert rules in the cluster were KubeMemoryQuotaOvercommit, NodeMemoryMajorPagesFaults and NodeMemoryHighUtilization: nothing for an OOM kill or a container near its limit.

Why the working set is near the memory limit but no OOM: MinIO sizes itself from 90 % of the cgroup limit

MinIO reads its own limit. In the source at tag RELEASE.2025-04-22T22-12-26Z, cgroupMemLimit() reads /sys/fs/cgroup/memory.max (with a cgroup v1 fallback) and availableMemory() returns 90 % of it, available = (limit * 9) / 10 (cmd/handler-api.go lines 62–106). At 512Mi that is 483,183,820 bytes, 460.8 MiB. MinIO uses that number three times:

1. Go soft memory limit. At startup setMaxResources calls debug.SetMemoryLimit(int64(ctx.MemLimit)) (cmd/server-rlimit.go lines 84–86), and MemLimit defaults to that 90 %. The Go documentation says the runtime then adjusts "the frequency of garbage collections and returning memory to the underlying system" to stay under it, and that the limit does not count kernel memory held for the process (runtime/debug/garbage.go lines 181–197). Page cache and slab are outside it. This came in with MinIO PR #19839 (merged 2024-05-30).
2. Concurrent request pool. apiRequestsMaxPerNode = maxMem / (drives × (1 MiB + 32 KiB) + 2 × 1 MiB) (cmd/handler-api.go lines 124–147), 3,178,496 bytes per request on one drive: 152 requests at 512Mi, 228 at 768Mi (computed from the source, not read from the server). Requests over the pool get HTTP 429 TooManyRequests at once.
3. Erasure byte pool. newErasureServerPools sizes it at min(1024, budget / 2 MiB) buffers of 2 MiB capacity (cmd/erasure-server-pool.go lines 93–116): 230 at 512Mi, 345 at 768Mi. Each buffer is allocated with a 2 MiB capacity (internal/bpool/bpool.go line 68), and a buffer that is handed back stays in the pool's channel until the process exits.

What MinIO does with the limit it is given, at 512Mi and at 768Mi. All three budgets grow with the limit; the margin stays 10 %.

The Kubernetes side explains why 95 % was not a kill. The documentation says memory limits "are enforced by the kernel with out of memory (OOM) kills" and "terminations only happen when the kernel detects memory pressure" (Resource Management for Pods and Containers). The working set is usage minus inactive_file (the kubelet excludes it "as it assumes that memory is reclaimable under pressure", Node-pressure eviction), so it "typically also includes some cached (file-backed) memory" (Resource metrics pipeline). Active page cache is counted in the working set but is still reclaimable. That is the difference between container_memory_working_set_bytes and container_memory_rss: on cgroup v2, cAdvisor's container_memory_rss is the cgroup's anon, not the process RSS, and the working set is usage minus inactive_file.

My reading (inference from the numbers above): at 512Mi MinIO had set the Go soft memory limit to 460.8 MiB, so anon stayed between 391.5 and 440.4 MiB over 30 days before the 11-14 restart (daily maxima 434–440 MiB; 255.9 MiB right after the restart), and the rest of the cgroup was page cache (83 MiB, which the kernel can reclaim) and slab (30 MiB). The 95 % working set was MinIO filling the limit it was given, not a leak. The risk was the margin: 512 − 460.8 = 51.2 MiB for everything outside the Go soft limit. Slab is neither anon nor page cache, and its 30 MiB counts against that margin.

What to check in your cluster now

Steps 1 to 4 are read-only. Replace <ns> and <pod>, and minio if your container has another name.

1. Limit and QoS class.

kubectl -n <ns> get pod <pod> -o jsonpath='{.status.qosClass}{"\n"}{range .spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' 

Ours before: Burstable, memory request 128Mi, limit 512Mi. MinIO sizes itself from that limit.

2. What the memory inside the limit is made of (cgroup v2).

kubectl -n <ns> exec <pod> -- sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) current=$(cat /sys/fs/cgroup/memory.current); cat /sys/fs/cgroup/memory.events; while read k v; do case $k in anon|file|active_file|inactive_file|slab) echo "$k $v";; esac; done < /sys/fs/cgroup/memory.stat' 

The MinIO image has no grep; the while read loop does the filtering. Same problem as ours when current is within a few MB of max, anon is under 90 % of max, MinIO's own budget (ours: current 510.8 MiB of 512 MiB, anon 395.9 MiB against 460.8 MiB, file 83.0 MiB, slab 30.0 MiB), and oom_kill is 0. max in memory.events counts the times the cgroup hit its limit and the kernel reclaimed; oom_kill counts kills.

3. Flat or growing. In Prometheus, graph both over 30 days with a 1-day step:

max_over_time(container_memory_rss{namespace="<ns>",pod="<pod>",container="minio"}[1d]) / 2^20 max_over_time(container_memory_working_set_bytes{namespace="<ns>",pod="<pod>",container="minio"}[1d]) / 2^20 

Flat day after day means sized, not leaking. Ours: anon 434–440 MiB every day, working set 489–506 MiB, against 512 MiB.

4. Has it ever been killed?

increase(container_oom_events_total{namespace="<ns>",pod="<pod>",container="minio"}[30d]) kube_pod_container_status_last_terminated_reason{namespace="<ns>",pod="<pod>"} 

Ours: 0, and the last terminated reason Unknown, not OOMKilled.

5. How much MinIO gives itself. 90 % of memory.max. The request pool size is returned in the X-RateLimit-Limit header of S3 calls, including an unauthenticated list on /: kubectl -n <ns> port-forward svc/<svc> 9000:9000 & sleep 2; curl -s -D - -o /dev/null http://127.0.0.1:9000/ | grep -i x-ratelimit. From the source it should read 152 at 512Mi on one drive. We did not run this check on our cluster, so we have no output to show. Computed values on one drive:

With more drives the request pool is smaller (the divisor is drives × 1.03 MiB + 2 MiB). If all of 1 to 4 match ours, the limit is being used as designed; what you decide is how big the margin should be.

To compare every container's working set with its memory limit on a schedule, kubegap, an open-source audit of Kubernetes cluster operations that I maintain, has a check for it: its resources.memory-limit-near-p95 check divides each container's 7-day p95 working set by its memory limit, warns above 0.8 and fails above 0.9. That check found this problem. It does not read memory.stat, so the split of step 2 and the OOM history of step 4 stay manual, or go into alerts like the two in the next section.

What we changed, and what we have not

Commit 1, 13:56:48, in the manifests repository that Argo CD syncs: memory request 128Mi → 768Mi, limit 512Mi → 768Mi. 768Mi was the report's example value, above KRR's 582 MB. The same commit added a PrometheusRule container-oom-rules with two alerts:

- alert: ContainerOOMKilled expr: | increase(kube_pod_container_status_restarts_total[1h]) > 0 and on (namespace, pod, container) kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 - alert: ContainerMemoryNearLimit expr: | max by (namespace, pod, container) (container_memory_rss{container!=""}) / on (namespace, pod, container) max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"}) > 0.9 for: 30m 

Argo CD synced at 13:58:47 and the new pod was Ready at 13:59:07: about 20 seconds without MinIO, because there is one replica. The first commit left the pod Burstable. Its comment said "memory request = limit (Guaranteed)", but the CPU request (50m) was not equal to the CPU limit (300m), and the Kubernetes Pod QoS rules require request = limit for CPU as well as memory on every container.

Commit 2, 13:59:42: CPU request 50m → 300m. The pod was replaced at 13:59:48 and Ready at 14:00:04 (about 16 seconds). At 14:00:12: QoS Guaranteed, requests = limits = cpu 300m / memory 768Mi, cgroup memory.max 805306368, mc ready local answered "The cluster 'local' is ready", and the PrometheusRule existed. memory.current was 233.8 MiB right after the restart; with no load, anon rose to 352.8 MiB in the next 7 minutes and to 381 MiB by 14:34.

The load test. Three attempts did not reach MinIO, and their numbers are not results:

1. 14:00:24: the Job pod could not pull its image (kubectl logs: "image can't be pulled"). The later attempts pinned the Job to server-1, where the MinIO image is cached.
2. 14:07:53: mc alias set read the password, which starts with -, as a flag: mc: <ERROR> Invalid command usage, flag provided but not defined.
3. 14:24:07: the alias was written to ~/.mc, but this image sets MC_CONFIG_DIR, so mc used local paths. 32 "uploads" finished in 0.2 s with 0 errors, and the MinIO pod's anon did not move.

Attempt 4 wrote the alias into $MC_CONFIG_DIR and checked that mc ls reached the server before starting. From 14:34:55 to 14:38:35 it uploaded 8 MiB objects with 32 parallel mc cp per pod, in 1, 2 and then 4 pods (32, 64, 128 concurrent), checked 256 MiB / 32 objects per pod on the server, read them back with 32 parallel mc cat, and removed the bucket. 0 errors at every stage.

memory.current reached 767.6 MiB of 768 at 14:37:39. After the test memory.events showed max 72 (the cgroup hit its limit 72 times and the kernel reclaimed page cache), oom 0, oom_kill 0, and the pod had 0 restarts.

At 768Mi under 32, 64 and 128 concurrent 8 MiB requests: anon above the old 512 MiB limit from 64 concurrent, under MinIO's own 691.2 MiB soft limit, and not back down afterwards.

What this test does and does not show:

• At 64 and 128 concurrent requests anon alone (618–678 MiB) was above the old 512Mi limit. We did not run the same load at 512Mi, so whether the old pod would have been OOMKilled, or held back by its own budget (152 requests, 460.8 MiB soft limit), is not measured.
• MinIO fills the new limit too. memory.current reached 767.6 of 768 MiB, and anon stayed at 618 MiB after the load (14:39:43; 627.6 MiB at 14:46:51), 237 MiB above where it started. That is 80 % of 768 MiB held at rest. That these are buffers kept by the byte pool is an inference from the source; MinIO does not export the pool size as a metric.
• The ContainerMemoryNearLimit alert fires at 90 % of 768 MiB, 691 MiB, which is MinIO's own Go soft limit (inference: it will fire only when MinIO is past its own target).

kubegap after the change. A full run collected at 14:40:19 passed resources.memory-limit-near-p95 for all 92 targets. The same ratio read from Prometheus at 14:45:52 was 0.78; the new pod only existed since 13:59, so that p95 is mostly the load-test hour. The same run reported three other things for MinIO:

• resources.cpu-throttling warn: "throttling ratio now 0.53, 24 h or the pod's life 0.02". This came from the timing: the run started 1.5 minutes after the load test ended, so it measured the test, not the day.
• resources.krr-recommendation now fails on CPU, not memory: "CPU request over by 0.288 cores: CPU request 0.3 → 0.012 cores". The 300m CPU request set to make the pod Guaranteed is far above what MinIO uses. KRR's memory figure, 796 MB, is just under 768Mi (805.3 MB) and rated GOOD.
• resilience.n-plus-one: memory without server-1 moved from OK (30.7GB vs 31.0GB requested) to short (30.7GB vs 31.8GB), because the request grew by 640Mi.

The check that failed before the change passes after it; the rerun started 1.5 minutes after the load test, which is where the new CPU warning comes from.

What we have not changed:

• The CPU request stays at 300m. It was raised to 300m only to make the pod Guaranteed, and it now exceeds what MinIO uses (the KRR result above); we left it as is, a choice for the owner.
• MINIO_API_REQUESTS_MAX and MINIO_MEMLIMIT are not set. They are the in-process way to keep MinIO below its limit. My judgement: at 768Mi the measured run had 0 OOM kills, and what a smaller request pool would cost in throughput is not measured.
• The first load-test pod could not pull the MinIO image (attempt 1). The running pod uses the copy cached on server-1. Outside this problem; not changed.
• No second replica or distributed mode. It needs more drives and is outside this problem. A restart of minio-0 still means 16–20 seconds without the object store, as both commits showed.

Where else this happened

The Incident Library, a searchable collection of public post-incident reports that I maintain at benchlet.com, has no MinIO case and no case of a working set near the limit without a kill. The three cases below show what the two new alerts are for: ContainerMemoryNearLimit warns while a container is still below its memory limit, and ContainerOOMKilled reports a kill like theirs after it happens; each case also shows what the kill took down elsewhere.

• GitLab, 2022-04-25: the Consul agent in every GitLab.com GKE pod had a 500MB memory limit, was killed again and again when its memory spiked, and the Rails pods lost replica discovery and sent every query to the primary database.
• GitHub, 2020-07-13: a container not needed for production traffic exceeded its memory limit, pods were marked unavailable, and restarts failed because a DNS change had cut access to the image registry; 4 h 25 min degraded.
• Buildkite, 2026-05-20: autoscaling raised the number of pods an EKS Prometheus tracked until Prometheus needed more memory than its host, and it was killed in a loop, so services stopped scaling.

More: OOMKilled + container memory limit, 15 cases.

Timeline

FAQ

minio pod OOMKilled

Check first whether it was killed at all. kube_pod_container_status_last_terminated_reason should read OOMKilled, and increase(container_oom_events_total[30d]) should be above 0; memory.events belongs to the running container's cgroup, so after a kill and restart it starts again at 0. Ours read Unknown and 0 at 95 % of the limit. If it was killed, remember MinIO sizes itself from 90 % of memory.max, so 10 % is left for page cache that cannot be reclaimed, slab and kernel memory.

working set near memory limit but no OOM

The working set is usage minus inactive_file, so active page cache counts in it, and the kernel can still reclaim that cache. The kernel kills only when it cannot reclaim enough. Our MinIO sat at 95 % working set for 30 days with oom_kill 0, while the daily maximum of anon, the memory the process allocated, stayed at 434–440 MiB.

minio 512Mi memory limit

At 512Mi MinIO gives itself 460.8 MiB: a Go soft limit, 152 concurrent requests on one drive, 230 two-MiB buffers. Ours ran 30 days unkilled with 51.2 MiB margin. At 768Mi, 64 concurrent 8 MiB uploads pushed anon to 618 MiB; that load was not run at 512Mi. A MinIO maintainer's answer to a 512Mi pod in #20156 was "set --memlimit <pod_limit> for memory, such tiny memory is not what we optimize for"; at this tag a --memlimit above 90 % of memory.max is ignored, so only a lower value changes anything.

MINIO_API_REQUESTS_MAX

It overrides MinIO's concurrent request pool, by default 90 % of the memory limit divided by drives × 1.03 MiB + 2 MiB: 152 at 512Mi, 228 at 768Mi on one drive. It is divided by the node count; requests over it get HTTP 429. The log line "Configured max API requests per node based on available memory" is printed only in distributed mode, so a single-drive MinIO never logs it; read the X-RateLimit-Limit response header instead (step 5, not run on our cluster).

MINIO_MEMLIMIT

It is the environment form of the hidden --memlimit flag (help text: "set global memory limit per server via GOMEMLIMIT"), and it can only lower MinIO's budget. A value above 90 % of memory.max logs "ignoring --memlimit value" and is capped. A lower value shrinks the request pool and the Go soft limit, but not the erasure byte pool, which reads the cgroup limit directly. We have not set it.

Originally published at benchlet.com.