CI failures are expensive even when the application code is fine. A cancelled deployment can leave a namespace running, a load balancer billing by the hour, or a test inbox that the next run accidentally reuses.

I started treating every ephemeral resource as a line item in a small ownership ledger. The ledger is not a replacement for Kubernetes controllers or AWS tagging. It is the operational contract around them: who created the resource, which run owns it, when it expires, and what evidence proves that cleanup happened.

This matters most in pipelines that combine Docker images, Kubernetes preview environments, and AWS services. The cleanup rules was easy to describe, but much harder to trust until the pipeline emitted evidence for each resource class.

Why ownership is a CI reliability concern

An ephemeral environment has more parts than its deployment manifest suggests. A typical pull-request pipeline may create:

  • a container image and registry tags;
  • a Kubernetes namespace, service account, secrets, and volumes;
  • an AWS load balancer, security-group rule, or DNS record;
  • database schemas and test identities;
  • notification or email test data.

The common failure is not that one cleanup command is missing. It is that cleanup cannot identify what the failed run owned. A generic kubectl delete namespace preview is dangerous when several branches are active. A broad AWS deletion query is worse: it can remove a resource created by a different workflow.

The owner field matter because it turns cleanup from a guess into a bounded operation. Every resource should carry a workflow name, run ID, commit SHA, environment, and expiry timestamp. If a test uses the phrase use and throw email as a fixture value, that text belongs in the test data, not in a selector used to find infrastructure.

The ledger fields that matter

The first version can be a JSON artifact attached to the CI job. I use one record per resource group rather than one record for every pod:

{ "run_id": "pr-1842-7f3a", "environment": "preview", "owner": "checkout-api", "resources": [ {"kind": "kubernetes_namespace", "name": "pr-1842-7f3a"}, {"kind": "aws_load_balancer", "id": "preview-1842"} ], "created_at": "2026-11-08T10:15:00Z", "expires_at": "2026-11-08T14:15:00Z", "cleanup_status": "pending" }

Five fields do most of the work:

  1. Run identity: a value unique to this attempt, not only to the pull request.
  2. Owner: the service or workflow responsible for the resource.
  3. Scope: the cluster, AWS account, region, and environment.
  4. Expiry: a deadline for the external reaper if the runner disappears.
  5. Evidence: IDs, commands, and results captured during cleanup.

Do not use a mutable branch name as the only identity. A second push can start before the first run finishes, and both runs then appear to own feature-login. A run ID makes retries distinguishable and lets the reaper be precise.

Label Kubernetes and AWS resources

For Kubernetes, apply labels to every namespaced object created by the pipeline. Helm values are a convenient place to centralize them:

commonLabels: ci.run_id: "pr-1842-7f3a" ci.owner: "checkout-api" ci.expires_at: "2026-11-08T14:15:00Z"

The exact chart mechanism depends on the repository, but the result should be queryable. A cleanup job can first list objects with ci.run_id=pr-1842-7f3a, record the list, and then delete only that set. This is specially useful when a preview namespace contains a PVC that survives the deployment release.

For AWS, use tags with the same vocabulary. Include ManagedBy=ci, CiRunId, CiOwner, and ExpiresAt. Keep the run ID in a format accepted by the service, and avoid putting secrets or full branch names into tags. An account-level reaper can find expired resources, but it should require the ManagedBy=ci tag before taking action.

When verifying onboarding or authentication, the email boundary deserves the same ownership rules. A disposable email account may be useful for a short-lived test, but the inbox identifier still needs the CI run ID and a retention deadline. The passwordless OTP inbox boundaries article is a useful companion for deciding what the test should isolate.

Reap safely after failures and cancellations

Use two cleanup paths. The first runs in the job's finally or shell trap block and handles the normal case. The second runs independently on a schedule and handles runner loss, force-cancelled jobs, and bugs in the first path.

The job cleanup should preserve the original exit status:

set -Eeuo pipefail cleanup() { status=$? kubectl delete namespace "$PREVIEW_NAMESPACE" --ignore-not-found ./scripts/reap-ci-aws-resources --run-id "$CI_RUN_ID" --dry-run=false printf 'cleanup run_id=%s status=%s\n' "$CI_RUN_ID" "$status" exit "$status" } trap cleanup EXIT INT TERM

In production, the commands should also be idempotent and have bounded permissions. The reaper should refuse a resource with a missing owner, a future expiry, or a tag outside the expected account and region. Log every refusal as well as every deletion; a skipped resource is often the clue for the next incident.

The first implementation we was seeing left behind was a load balancer created before the Kubernetes install failed. A nightly scan found it, but only after the bill increased. Adding an AWS tag before the dependent resource was created made the orphan visible immediately. It save time during incident review too.

Email tests need the same separation. Do not let a retry silently reuse an inbox from an uncertain run. A tempmailso workflow can be scoped to the current run, while email state in signup flows helps keep application state separate from delivery state. Test strings such as tem email can stay in negative cases, but they should never be used as infrastructure selectors.

A practical review checklist

Before enabling a new preview environment, check:

  • Does every created resource have a unique run ID and owner?
  • Can an operator list all resources for one run without broad deletion?
  • Is the expiry visible to an independent reaper?
  • Does cancellation preserve the test's original failure status?
  • Are cleanup refusals and successful deletions both logged?
  • Are Kubernetes and AWS permissions limited to the intended account, cluster, and namespace?
  • Can a retry create a fresh identity instead of sharing uncertain state?

This is boring work, but it pay back when a pipeline is cancelled at the worst moment. A resource ownership ledger gives the on-call engineer a bounded search, a safe cleanup action, and evidence that the action actually happened.