Short answer: treat a tenant export as unpublished until the worker has stored the exact completed-object identity in a durable record. A browser download link should be derived from that record, never from a filename guessed when the job was queued. This turns a confusing object-storage 404 into a small set of inspectable state transitions.
For a healthtech system that stores product images and builds a tenant-scoped ZIP, large-file throughput makes the distinction matter. The HTTP request that starts the work and the worker that completes the archive are separate owners. A fast queue can widen that gap; it cannot make an unfinished archive downloadable.
Why does a browser download link hit an object storage 404 after an export?
Start with the requested object key, copied byte for byte from the access log, and compare it with the key on the completed export record. Do not reconstruct it from a display filename, an object listing, or a route parameter. A missing tenant segment, a URL-encoding change, or a retry suffix is enough to make a plausible-looking link point nowhere.
The browser is just the witness.
For multipart uploads, completion is a real boundary: the upload is not an object available for download until the multipart completion operation succeeds. A worker that updates its database row before that operation has created an application-level race, even where the storage system offers strong read-after-write behavior. A HEAD-style metadata lookup after the final write provides evidence about the precise key the worker intends to publish; it does not prove that a CDN, retention worker, or a different retry will honor the same lifecycle.
The failure report I want has six fields: export ID, tenant ID, attempt ID, requested key, recorded key, and lifecycle state. Keep signed query strings out of it. For notification-heavy systems, that restraint matters: a copied URL in an email or SMS log is a credential, not useful diagnostic context.
Do not retry blind.
ADR: publish an immutable object identity, not a predicted path
Decision: the export worker alone moves an export from generating to ready, in the same durable update that records the final key and, where available, its version or generation. The link service reads a ready record scoped to the authenticated tenant. It does not accept an object key from the browser.
The invariants are deliberately boring. One completed attempt has one tenant ID and one object identity. ready means the storage write completed and the identity was persisted. Cleanup first changes ready to expired; only then may it delete the associated bytes. A request that loses that race gets an unavailable response rather than a freshly signed link to an object being removed.
| Publication design | What the 404 means | Throughput effect | Appropriate use |
|---|---|---|---|
| Sign a predicted key at job creation | The queue, retry, or prefix construction disagreed with the link | No finalization wait, but failures surface at click time | A status page link, not a completed file |
| Record the completed identity, then sign it | The record or retention state is stale, and has a narrow audit trail | One metadata check per completed archive | Tenant-scoped asynchronous exports |
| Stream the archive through the application | The application policy or stream failed | Adds application bandwidth and connection pressure | Small artifacts requiring per-download inspection |
The second row adds a pending state that some teams dislike. Keep it. Honest latency is easier to explain than a link that sometimes fails after a notification has already been delivered.
This decision has a cost beyond one database update. The job table becomes part of the download-serving path, so its retention, tenant index, and transition rules deserve the same review as the bucket policy. An export that reaches ready must retain the attempt identity long enough for a support trace to join the browser request to the worker write; deleting that row early recreates the mystery under a different name. The database also needs a unique rule that says which attempt is current for an export. Without it, two workers can both finish successfully, and a later transaction can silently replace the object identity that an earlier notification described. Use a conditional transition rather than an unconditional update, and alert when it affects zero rows. That outcome is useful evidence: either cleanup expired the export, a cancellation won, or another attempt already published. It should not become an automatic retry that writes another archive. For a service sending delivery messages, make the outbox entry reference the committed export ID. The message sender can retry its own delivery without reopening the storage lifecycle.
The critical path is a state transition, not a URL formatter
The adapter below is intentionally generic. It uses a unique attempt key, so a retry cannot overwrite bytes that a reader has already been authorized to fetch. The database predicate is what keeps two workers from publishing competing attempts.
from dataclasses import dataclass from datetime import UTC, datetime, timedelta from uuid import UUID @dataclass(frozen=True) class CompletedObject: key: str size: int version: str | None def archive_key(tenant_id: UUID, export_id: UUID, attempt_id: UUID) -> str: return f"tenants/{tenant_id}/image-exports/{export_id}/{attempt_id}.zip" def finalize_export(db, storage, tenant_id: UUID, export_id: UUID, attempt_id: UUID, archive: bytes) -> None: key = archive_key(tenant_id, export_id, attempt_id) storage.put(key=key, body=archive, content_type="application/zip") metadata = storage.head(key=key) if metadata.size != len(archive): raise RuntimeError("completed archive size differs from uploaded bytes") completed = CompletedObject(key=key, size=metadata.size, version=getattr(metadata, "version", None)) changed = db.publish_if_generating( export_id=export_id, tenant_id=tenant_id, attempt_id=attempt_id, object_key=completed.key, object_version=completed.version, expires_at=datetime.now(UTC) + timedelta(days=7), ) if not changed: raise RuntimeError("another lifecycle transition won") def issue_download(db, storage, tenant_id: UUID, export_id: UUID) -> str: export = db.get_ready_export(export_id=export_id, tenant_id=tenant_id) if export is None or export.expires_at <= datetime.now(UTC): raise PermissionError("export is unavailable") return storage.signed_download(key=export.object_key, expires_in=300) For a large archive, avoid buffering all image bytes in the web process just to preserve this ordering. The worker can stream archive production into the storage writer, then execute its completion and metadata steps before publishing the record. The exact checksum, version, and replacement semantics differ by backend, so persist only identifiers whose documented meaning the adapter can verify. A size check catches truncation; it cannot establish content equivalence on its own.
What should the timeline show before blaming consistency?
Trace the job creation, upload start, upload completion, metadata result, database publication, link issuance, and cleanup transition with one correlation ID. The interesting cases usually become obvious when these events are ordered. A ready row before upload completion is a publisher bug. A requested key that differs from the recorded key is prefix drift. A recorded key that existed at publication but disappeared after expired is a retention question.
Test the awkward timings: cleanup one second before a click, two jobs with the same displayed ZIP name, a retry after a failed multipart completion, and a tenant A request carrying tenant B's export ID. Also test the notification boundary. Send the message after the ready transition commits, and make notification retries reuse that committed export rather than create a new archive.
Watch the count of exports stuck in generating, attempts to sign non-ready records, and storage-not-found results partitioned by lifecycle state. Those three signals identify whether throughput pressure is leaving work unfinished, authorization is bypassing publication, or retention is doing exactly what it was told to do.
The 300-second download lifetime is a policy input, not proof of availability.
Rejected option: deterministic overwrite keys
Using tenants/{tenant}/exports/{export}.zip for every attempt looks tidy and can reduce cleanup bookkeeping. It is a poor fit when a retry can overlap a download, because the system must then define what readers see during replacement and how the old version remains addressable. Choose it only when the storage adapter exposes documented conditional replacement and version semantics that the application actually enforces.
A real limitation of the publication-ledger pattern is its extra durable state and database dependency. The trade-off is not suitable for an export whose only consumer is an internal service and whose bytes are small; choose synchronous generation followed by direct application streaming there. It is not a throughput strategy for a ZIP assembled from many product images. The decision here is about an explicit handoff between asynchronous producers and browsers, not the name on a storage account.

