Troubleshooting Object Storage Races When a Browser Download Link Returns 404 After Export

Short answer: treat a browser 404 after an export as a publication-contract failure, not as proof of eventual consistency: verify the exact object key, require the job to publish a committed key only after upload completion, and generate the download URL from that stored key rather than reconstructing a prefix in the browser.

A bounded production scenario makes the distinction concrete. An export worker accepts a job, writes a large file to object storage, and the status endpoint eventually says complete; the browser then follows a download link and receives HTTP 404. My first move is to trace one immutable job ID through four recorded values: the requested export, the final key returned by the writer, the key persisted with the completed job, and the key used to create the link. I don't start with retries. If those strings differ by one slash, tenant segment, filename normalization rule, or environment prefix, waiting only delays the same failure.

The invariant is blunt: a completed export must name an object that has already been committed and can be addressed by that exact name. Everything else is an implementation detail.

How should you troubleshoot a browser download link to missing object storage exports?

Start at the server-side publication boundary and work outward. Capture the job ID and the failing request, then compare the full object key byte for byte at each handoff. Do not compare only the filename. A key such as exports/acme/42/report.csv is not interchangeable with /exports/acme/42/report.csv, export/acme/42/report.csv, or exports/acme/042/report.csv; an application that assembles those strings in several places has several chances to disagree with itself.

Next, separate lookup failure from authorization failure using the response produced by your own download service and its internal logs. The browser-visible status alone may be a deliberately narrow security response, so the useful evidence belongs at the trusted boundary: job ID, object key, job state, link creation time, and a correlation ID. Avoid logging signed query parameters or credentials. If the link points through an application endpoint, verify that endpoint resolves the persisted key; if it points directly to storage, verify that the signer received the persisted key unchanged.

Then inspect ordering. A worker can report logical work as finished before the storage write has reached its commit point, particularly when multipart upload is involved. Multipart upload has an explicit completion operation; the final object is assembled when that operation succeeds, and incomplete parts are a separate concern. The job state transition therefore belongs after successful completion, not after the last part was merely sent. This is a race in the application state machine, regardless of which storage system sits behind it.

Only after key equality and commit ordering are proven would I investigate consistency semantics. “Eventual consistency” is often used as a catch-all explanation, but it is not a portable diagnosis: behavior depends on the service and operation. Check the current documentation for the storage system you actually run, then design against its stated contract. I'm not sure what guarantees an unnamed storage-compatible gateway provides; a focused create-then-read test against that exact deployment would resolve it.

Fast triage, in order:

1. Reproduce with one job ID and preserve the failing URL without exposing its secret query fields.
2. Read the persisted final key and compare it with the writer's committed key.
3. Confirm upload completion happened before the terminal job-state update.
4. Generate a fresh link from the persisted key and compare only its path and key inputs.
5. Check retention, deletion, and lifecycle events if the object once existed but no longer does.
6. Test documented consistency behavior only after the deterministic checks pass.

Stop guessing.

Make completion a storage-backed state transition

The preventative design is a small state machine: queued, running, publishing, then complete. The complete transition stores the authoritative final key. A download handler does not derive that key from a user-supplied filename, a display label, or a duplicate prefix constant; it reads the key attached to the completed job. If publication fails, the job remains non-terminal and no download link is emitted.

The following Go sketch keeps the storage interface generic and makes the ordering visible. Commit represents the backend's successful finalization point, including multipart completion when the implementation uses multipart upload.

package export import ( "context" "errors" "io" ) var ErrNotReady = errors.New("export is not ready") type Job struct { ID string State string ObjectKey string } type ObjectWriter interface { Commit(ctx context.Context, key string, body io.Reader) error SignRead(ctx context.Context, key string) (string, error) } type JobStore interface { MarkPublishing(ctx context.Context, jobID string) error MarkComplete(ctx context.Context, jobID, committedKey string) error Get(ctx context.Context, jobID string) (Job, error) } func Publish(ctx context.Context, objects ObjectWriter, jobs JobStore, jobID, key string, body io.Reader) error { if err := jobs.MarkPublishing(ctx, jobID); err != nil { return err } if err := objects.Commit(ctx, key, body); err != nil { return err } return jobs.MarkComplete(ctx, jobID, key) } func DownloadURL(ctx context.Context, objects ObjectWriter, jobs JobStore, jobID string) (string, error) { job, err := jobs.Get(ctx, jobID) if err != nil { return "", err } if job.State != "complete" || job.ObjectKey == "" { return "", ErrNotReady } return objects.SignRead(ctx, job.ObjectKey) } 

There is still a distributed-systems edge here. Storage commit can succeed and the job database update can fail, leaving an object without a published job record. Do not reverse the operations, because that creates a completed job that names an uncommitted object. Instead, make publication idempotent, retain the deterministic key, and reconcile non-terminal jobs against committed objects. The repair path must use the same key contract as the normal path.

For capacity planning, budget separately for export CPU, temporary workspace, multipart parts, final-object storage, and concurrent link generation. A queue depth SLO without a publication-latency SLO can look healthy while users wait for files; measure time from job acceptance to committed-key publication, and count terminal jobs whose link resolution does not find the named object. Those signals describe the user-visible contract more directly than raw upload throughput.

Compare the likely causes before changing architecture

The fastest fix depends on which boundary broke. This table is deliberately operational: it maps evidence to the next test, not to a favorite product.

One nuance matters: a 404 returned to the browser does not, by itself, identify the table row. Applications and gateways may intentionally avoid revealing whether a private object exists. That is why the internal correlation trail is part of the design, not optional debugging garnish.

What should the export pipeline own, and what should storage own?

The export pipeline should own job state, canonical key construction, authorization, retention promises, idempotency, and the rule for when a link becomes visible. Storage should own durable object commit and retrieval according to its documented contract. Mixing those responsibilities leads to browser code that guesses prefixes or workers that equate “all bytes sent” with “object published.”

For a buy-versus-build review, I would force the platform team to price on-call load and lock-in alongside request charges. No single answer wins every column.

The catch is that this design is not suitable for a genuinely streaming export where the user is expected to consume bytes before a final object exists. In that case, use a streaming protocol and expose streaming state; don't label the job complete and pretend the endpoint represents a committed file. Likewise, if immutable files must be downloadable indefinitely, short-lived links are still useful, but the application needs a stable authorized endpoint that can mint a new one while retention is in force.

Test the contract, not the happy-path filename

Unit tests should cover key construction with empty segments, Unicode names, repeated slashes, tenant boundaries, and filenames that are display-equivalent but byte-different. Integration tests should hold the job in publishing, assert that no link is returned, complete the storage commit, transition the record, and then resolve the exact persisted key. Add a failure injection between commit and database update to prove reconciliation is idempotent.

The deployment test is smaller: publish one uniquely named object, wait only on the documented commit response, read it by exact key, and delete it under the same lifecycle policy used in production. Run that probe in every environment because prefixes and credentials often differ there. Your mileage may vary for storage-compatible gateways, so record the product version and the observed operation in the test result instead of generalizing from a brand name.

For the SLO, measure successful downloads against eligible, authorized requests, and split failures by internal cause while keeping the public response appropriately narrow. Alert on completed jobs with missing committed keys, reconciliation backlog, and publication latency. A browser 404 is the symptom. The state transition is where correctness lives.

Sources

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• https://cloud.google.com/storage/docs