It all started as a routing investigation.
The monorepo (Go, nodejs, C++ with shared proto files) pipeline had been running slowly for some time, and then it started to fail. Bazel builds were being OOM-killed during working hours, the retries lengthened the queue, the longer queue reduced the frequency of pushes, and the CI issues quietly escalated into engineering problems. Upon finally logging into the runner host, I found that the machine, with 6 cores and 12 threads, had a load average of 32.7. Swap space was being used. There was nothing wrong with the box in the conventional sense. It was working exactly as instructed, but the instructions themselves were bizarre.
The cause was a comment in the .bazelrc file describing a different machine. The CI configuration block briefly stated that the runner could run a maximum of two Bazel containers simultaneously, and therefore each build could consume five CPU threads and 24 GiB of memory. The calculation was correct for the target runner at the time it was written. However, the two CI tags used in the monorepo had been resolving to a different runner for quite some time, and that other runner was set to concurrent=10 with no per-job restrictions whatsoever. In other words, the machine with 6 cores and 62 GiB of memory was accepting 9 jobs at once, and each Bazel job among them was requesting the budget promised in the comment. In total, the containers on that host perceived themselves as having 25 CPUs and 120 GiB of memory. There were no cgroup limits that dictated anything else. The only restriction in the whole system was Bazel's own local_resources setting, but this setting is an estimate for the Bazel scheduler, not a hard limit.
This fix was intentionally boring and required three days of readjustment to get it working correctly. First, a hard cgroup limit was imposed on the runner, allowing three jobs with four CPUs and 14 GiB of capacity each to run simultaneously. Next, we ran four jobs using four CPUs and 12 GiB. After that, we ran five jobs using three CPUs and 11 GiB, at which point the processing stabilized. Then, with each step, .bazelrc gradually decreased from 5 CPUs and 24 GiB per build to the following:
# CPU: 3 threads (5 x 3 = 15, 1.25x SMT) # RAM: 9GiB under the 11GiB cap. Measured peak per job was 11.37GiB working # set but only 6.68GiB RSS, the rest reclaimable page cache # Jobs: 5, deliberately above the CPU budget so sub-CPU actions still fill it build:ci --local_resources=cpu=3 build:ci --local_resources=memory=9216 build:ci --jobs=5 Two of the rules within that block took me an egregious amount of time to understand. The value of local_resources must be strictly below the upper limit of cgroup. This is because if there is a margin, resource-intensive processes will be queued instead of being OOM-killed. Furthermore, the memory value should be determined based on the actually measured 6.68 GiB resident memory set, not on the 11.37 GiB working set. This is because the difference between these two values represents the reclaimable page cache that the kernel frees when needed.
There was another surprise on the CPU side. The first version of the caps used a hard CFS quota, so the box quickly went into half-idle mode. Measurements revealed that host utilization was only 50%, and the one job actually running was throttled in 93% of its CFS periods. Hard quotas serve to protect neighbors that don't yet exist, but this is achieved by tightening the limits on currently running jobs. Therefore, while the CPU limit was no longer strictly defined and a fair allocation method was adopted, the memory limit was strictly maintained. This is because memory is the resource that actually kills.
Additionally, some jobs were moved. The Starlark linter and code review helper were occupying build slots on the loaded runner, but since linters shouldn't occupy build slots, both were moved to a lighter runner. The only remaining process was the computation of the affected targets, because this process required the warm repository cache present on that machine.
While investigating the host, I discovered a second leak, older and less noticeable than the first. Our integration tests launch helper containers using the raw Docker SDK, and the CI job mounts the host's Docker socket. This means these containers are siblings of the job on the host and outlive the job. These cleanup processes are performed within a finally block, which is not executed if the job is terminated. And, since the previous pipeline is automatically canceled each time a push is made, the job is terminated periodically. The Reaper shipped by Testcontainers only owns containers that Testcontainers itself launched. The host's garbage collector only removes containers with the Testcontainers label. These helper containers did not have any labels. Its natural lifespan was 300 seconds. The specimens I found on the runner were 4 months, 8 weeks, and 3 days old.
To solve that problem, we needed to modify one line for each calling site, for a total of five lines of (vibe)code.
# Ryuk only owns containers Testcontainers itself started, and the host GC only # removes what carries this label, so without it these leak forever -- specimens # four months old were found on the runner. ITEST_CONTAINER_LABELS = {"org.testcontainers": "true"} container = client.containers.run(image, ..., detach=True, labels=ITEST_CONTAINER_LABELS) The monorepo behind all of this has 816 BUILD files, 151 of which define container images, so the runner isn't a toy and the load is real. However, this story isn't unique to Bazel or our stack.
The lessons I learned from this are that CI hosts follow cgroups, not comments, and all capacity promises must originate from the hosting machine, be verified against the machine's specifications, and reaffirmed when jobs begin to be run elsewhere. And, in parallel with that, every process that CI starts on a shared host needs to be labeled so that the garbage collector can read it, because finally blocks are also promises, and jobs that are terminated do not keep those promises.
