Our nightly patch build reused one cached apt-get upgrade for seven months

Our services run on a base image of our own, built from Ubuntu 22.04. In March we pinned the Ubuntu image by digest, so that two builds of the same commit would always start from the same bytes, and we kept security patches flowing with one line near the top of the Dockerfile:

RUN apt-get update && apt-get -y upgrade && rm -rf /var/lib/apt/lists/*

A scheduled workflow rebuilt the base image every night and pushed it, and every service picked it up on its next build. The workflow uses buildx with a registry cache, cache-from and cache-to pointing at the same reference, to keep builds fast.

In October an external penetration test reported that our customer API was running an OpenSSL with a fix published in the summer. We said that was impossible, because the image had been rebuilt the night before. Its creation date agreed with us. Trivy, run against the same image, agreed with the testers: the package versions inside it were the ones from March.

BuildKit decides whether to run a RUN instruction by looking at the layer underneath it and the text of the command. It does not look at the clock or at the network. The layer underneath was the pinned Ubuntu image, which never changed, and the command was the same string every night, so every night the builder found a matching layer in the registry cache and reused it. The rebuild did run. It took eleven seconds, gave the image a new creation date, and for seven months produced layer digests identical to the night before. Nobody had compared them.

The upgrade now sits in its own stage, and the nightly build passes --no-cache-filter for that stage, so it always runs. Renovate updates the pinned digest when Ubuntu publishes a new image. The workflow records the installed package list and posts the difference from the previous night, so a night with no difference is visible. Trivy runs against every pushed image and fails the build on any critical or high finding that has had a fix available for more than fourteen days.

A cache gives you the answer you got last time. We had written a step whose whole purpose was to get a different answer each night, and handed it to a tool built to make sure it never would.

– Sergey Shinder