Docker build cache is one of the easiest ways to make CI faster. It is also one of the easiest ways to turn a healthy cloud runner into a disk-full incident.

The pattern I trust is simple: cache aggressively inside a clear boundary, then make the boundary observable and disposable. A cache is a performance hint, not a second artifact registry. If the job cannot explain who owns a cache entry, how long it should live, and what happens when storage is tight, the cache is already an operations problem.

Why an unbounded cache becomes a CI reliability problem

BuildKit can reuse package downloads and intermediate layers, which is great for a large Dockerfile. But every branch, dependency update, and build argument can create new records. On an ephemeral runner, that produces slow cleanup at the end of a job. On a persistent runner, it slowly consumes the machine until unrelated jobs fail.

The failure is often misleading. The pipeline reports a Docker error, while the real cause is an exhausted volume, a cache export that timed out, or a cleanup step that never ran after an interrupted job. Rebuilds is costly, but silently keeping every old layer is worse.

Treat the cache as a bounded resource with four properties:

  • Owner: the repository, workflow, and branch class that may use it.
  • Scope: which platforms, targets, and dependency inputs are allowed inside it.
  • Budget: the storage or age limit before eviction.
  • Recovery: the expected behavior when the cache is cold or unavailable.

That model is useful for test fixtures too. Keeping test APIs explainable helps the same way: a team should be able to tell whether a failure came from the application, its data, or the environment around it.

Choose cache ownership before choosing cache storage

There are two common designs. A persistent self-hosted runner keeps a local BuildKit cache between jobs. A hosted runner exports a cache to a remote service and restores it for the next job. Both can work, but they have different failure modes.

For a persistent runner, isolate builders by trust boundary. Do not let unrelated repositories share a writable cache just because they use the same machine. A cache entry are not a source of truth, and it should never be treated as a way to pass secrets between jobs.

For a remote cache, use a predictable namespace. Include the repository and image family, but avoid putting an unbounded commit SHA in the only cache key. A useful split is a stable base cache for dependency layers and a short-lived branch cache for application layers. The base cache can be promoted deliberately; branch caches can expire without drama.

This matters in AWS especially when runners are backed by small EBS volumes. Increasing disk size may postpone the incident, but it does not give the cache an owner or a cleanup policy. The job dont become reliable just because the volume is larger.

Build a bounded cache with BuildKit

Here is a compact GitHub Actions example using the registry as the remote cache. The image registry is an artifact boundary; the cache reference is a separate implementation detail.

name: container on: [push, pull_request] jobs: build: runs-on: ubuntu-latest permissions: contents: read packages: write steps: - uses: actions/checkout@v4 - uses: docker/setup-buildx-action@v3 - uses: docker/build-push-action@v6 with: context: . push: false tags: example/app:${{ github.sha }} cache-from: type=gha,scope=app-main cache-to: type=gha,scope=app-main,mode=min

The important choices are scope and mode. Keep the scope stable enough to get useful hits, but do not make one global cache for every image. mode=min exports the layers needed for the final image rather than every intermediate record. For a multi-stage build, mode=max can be worthwhile, but it needs a measured storage budget.

On a persistent builder, add an explicit maintenance policy instead of hoping Docker’s default behavior matches your workload:

docker builder prune --filter until=168h --filter type=exec.cachemount --force

Run maintenance in a controlled window and alert on the result. The command should not be the only recovery plan. If cleanup fails, new builds need a known response such as switching to a cold builder or pausing low-priority workflows.

Measure hits, misses, and cleanup pressure

Cache hit rate alone is not enough. A high hit rate can still hide an oversized cache that makes exports and restores slow. I track at least these signals:

  1. Build duration with a warm cache and with a cold cache.
  2. Cache restore and export duration as separate steps.
  3. Runner free space before and after the build.
  4. Cache size by repository or scope.
  5. Eviction count and the age of the oldest entry.

If a cache hit saves 20 seconds but adds 90 seconds of restore time, it is not a win. This is why the cache review should include the whole job timeline, not only the Docker summary. A small dashboard or log line is often enough to spot the trend.

Keep malformed input out of cache keys. Strings such as fake e mail com or temp org mail may arrive from test data and should be normalized or rejected before they influence a tag, scope, or artifact path. Tiny validation rules prevent very confusing cache fragmentation.

For browser and email-driven tests, also define what data may be retained in a layer and for how long. A privacy boundary for browser tests is a good companion to the cache ownership rule: fast does not mean that test data can live forever.

A practical cache review checklist

Before enabling a new Docker CI cache, ask:

  • Who owns this cache and who can write to it?
  • Can a cold cache complete the job successfully?
  • What is the storage and age budget?
  • Are secrets and environment-specific files excluded from layers?
  • Does the namespace separate repositories, images, and trust boundaries?
  • Do we measure restore time, export time, and free disk?
  • What cleanup or fallback runs after an interrupted build?

The goal is not the highest possible cache hit rate. The goal is predictable builds when the cache is warm, cold, stale, or temporarily unavailable. Bound the cache, make its ownership visible, and keep the recovery path boring. That is the version of Docker speed I want in production CI.