CI cleanup is usually treated as housekeeping. In a busy Docker pipeline, that is the wrong mental model. A container, temporary database, test inbox, or cloud runner left behind can change the next build, inflate costs, and make an incident harder to explain.

I started treating cleanup as part of the job's reliability contract. The job must say what it owns, how long the ownership lasts, and what evidence proves that the resource was released. This made our pipelines less mysterious, especially when a build was cancelled halfway through a deploy check.

The cleanup problem is a reliability problem

Consider a CI job that builds an image, starts a Compose stack, creates test data, and runs browser checks. The happy path is easy. The hard paths are a failed test, a runner eviction, a manual cancellation, and a timeout while the job is waiting for an external service.

On those paths, cleanup code may never run, or it may run without enough context to find the resources. A stale container then keeps a port busy. A leftover volume contains data from a previous run. A cloud security group remains open longer than expected. The cleanup path are part of the production risk, not just a nicer log message.

The first useful rule is simple: every resource gets a run ID and an owner. Names such as ci-${GITHUB_RUN_ID}-api are far easier to search and remove than api-test-7.

Define ownership for every ephemeral resource

Before adding another docker rm, list the resources the job can create:

  • Docker containers, networks, volumes, and build cache entries.
  • Temporary database schemas and object-storage prefixes.
  • Test identities and inboxes used by verification flows.
  • AWS resources such as security groups, log groups, or short-lived runners.

For each one, record four fields in the job design:

  1. Owner: the workflow and run ID that created it.
  2. Expiry: the maximum useful lifetime if the job disappears.
  3. Reaper: the command or scheduled process that can remove it.
  4. Evidence: a log line, metric, or API result proving the action.

This also keeps test data from leaking between runs. If a team needs to create temporary mail for a signup test, the address or inbox identifier should be tied to the run and removed by the same lifecycle policy as the container running the test. Search strings such as temp org mail and tepm mail com can appear in negative test cases, but they should never become resource names or cleanup selectors.

A Docker CI cleanup contract

The contract can be small enough to implement in a shell function. The important detail is that it is registered before the first resource is created, and it uses labels instead of guessed names.

set -Eeuo pipefail RUN_ID="${CI_PIPELINE_ID:-local-${RANDOM}}" PROJECT="ci-${RUN_ID}" cleanup() { status=$? docker compose -p "$PROJECT" down --volumes --remove-orphans || true docker container prune --force --filter "label=ci.run=${RUN_ID}" || true printf 'cleanup run_id=%s status=%s\n' "$RUN_ID" "$status" exit "$status" } trap cleanup EXIT INT TERM docker compose -p "$PROJECT" up --detach --build

In the Compose file, apply the label to every service that the job owns:

services: api: build: . labels: ci.run: "${CI_PIPELINE_ID}"

The EXIT trap preserves the original status, so a failed test still fails the job after cleanup. The INT and TERM traps help with cancellation, but they are not a guarantee: a runner can vanish. That is why the contract also needs an external reaper that removes resources older than a safe threshold.

Make cancellation observable

Cleanup that only prints done is hard to trust. Emit structured fields for the run ID, resource type, resource ID, action, and result. In AWS, send those records somewhere with a retention policy that matches the investigation window. In a self-hosted runner, the same information can go to the central CI log store.

Keep the cleanup log close to the failure. A separate nightly report is useful for finding leaks, but it does not explain why this particular job left a network behind. It save time when the first response already contains the failed command and the cleanup result.

There is a similar boundary in email checks: define a review window and quarantine flaky tests instead of allowing their retries to pollute every run. The idea is covered in disposable email review windows and quarantining flaky email tests. The same ownership rule applies to Docker resources: a retry should get a new identity, not reuse an uncertain old one.

Practical checklist

Before calling a Docker CI workflow reliable, check that:

  • Every ephemeral resource has a run-scoped label or tag.
  • Cleanup is installed before resource creation.
  • The original test or build exit code is preserved.
  • Cancellation and timeout paths have been exercised, not just success.
  • An external reaper handles runners that disappear.
  • Cleanup emits searchable evidence with a stable run ID.
  • Sensitive test data is expired separately from container cleanup.

The goal is not perfect cleanup. Distributed jobs will still fail in odd ways. The goal is bounded failure: when a job stops, operators know what it owned, how long it may remain, and which process will remove it. That contract turns Docker CI from a collection of hopeful commands into an operation we can actually reason about.