Give every AI-generated product image render an immutable attempt key, and publish a tenant-scoped pointer only after the render record is durably committed. A retry must create another candidate, never compete for the same object name. That decision preserves a recovery path when an image appears missing even though a storage request succeeded.
Short answer: diagnose the job and its publication record before inspecting the image URL; if a retry reused the same key, no-version storage cannot recover the bytes that the later write replaced.
This is an access-control problem disguised as a filename problem. In a media catalog, the image is addressed by a tenant, product, and render job, while the worker sees a convenient path such as products/42/latest.png. A caller can time out after the upload has completed, retry, and produce a second valid image. Both workers then have a plausible claim on one name. The browser reports “missing” when the catalog points at an object that was never published for that tenant, or when the expected attempt was silently replaced.
The system needs an identity for the work, not just an identity for the bytes.
What should an AI image retry record before storage writes?
Record the tenant ID, product ID, job ID, attempt ID, immutable object key, and publication state before invoking the generator. The attempt ID must exist before the worker starts. A timestamp alone is a weak identity because two workers can share a clock value, and a prompt hash is not an attempt identity because the same prompt can be intentionally rendered twice.
Use a private candidate namespace and a separate canonical reference. For example, the record might map tenant-7, sku-184, and attempt-02 to one candidate key, while a catalog row says which candidate is currently published. The canonical reference is application state; it is not the render destination.
That split gives operators something concrete to compare:
| Signal | What it tells you | Action |
|---|---|---|
| One job, two attempt IDs | A retry or duplicate delivery happened | Keep both candidate records until publication is decided |
| Two attempts, one object key | The key builder is collapsing identity | Stop treating the object name as an idempotency key |
| Candidate exists, no publication row | Storage completed but the workflow did not advance | Replay the state transition, not the image generation |
| Publication row points across tenants | The authorization boundary is wrong | Quarantine the reference and audit the write path |
The important invariant is simple: an object key must not be the only record of ownership. Object metadata can help an investigation, but it does not replace a queryable job table that can answer who requested an image, which attempt won, and when the pointer changed.
Names are evidence, not authority.
Why do same-key collisions make a missing image look random?
Retries create time reordering. Worker A may generate the expected blue shoe image, upload it, and lose its response. Worker B then starts from the same request, writes a different result to the same key, and returns normally. If the catalog update from A arrives after B, the database and the bytes disagree. If A never updates the catalog, the object still exists, but the expected attempt is not visible through the product record.
No mystery is required. Last-writer behavior is enough.
Check ownership first.
The failure is harder to see when the read path derives an object name from request parameters instead of loading the published candidate. A 200 response for latest.png proves only that some bytes occupy that name. It says nothing about tenant ownership, render attempt, content type, or whether the bytes match the catalog version. A 404 may mean a cleanup job removed a candidate that the catalog still references; that is a referential-integrity failure, not evidence that generation failed.
The audit trail should make the answer boring: one job, two attempts, two candidate keys, one publication decision. If it instead shows one key with changing checksums, the overwrite happened before the investigation began, and the missing-image symptom is only the last visible consequence. Preserve request IDs, transition actors, and the catalog pointer in separate fields; collapsing them into a filename leaves the operator guessing about order, ownership, and intent while the next retry is still able to change the evidence.
Treat the pipeline as a small state machine: queued, rendering, stored, published, and superseded. A worker may safely retry the transition into stored only for its own attempt. A transaction or serialized queue should decide the transition into published. The decision should include the attempt ID and an expected job state, so a stale worker cannot publish over a newer decision merely because it finished later.
This is also where SLOs become useful. Track the rate of jobs with more than one attempt, the age of stored candidates waiting for publication, and the number of catalog references that have no matching object. The image-delivery SLO should measure a tenant-authorized, catalog-selected image, not just a successful storage write. Capacity planning follows from the same model: retry bursts multiply candidate objects and database rows, so retention and cleanup must be sized for attempts, not successful products.
A Go boundary that prevents the collision
The storage adapter should accept an immutable key and refuse to invent one from a product name. The following example keeps provider calls behind an interface; the important behavior is the key contract and the tenant check around publication.
package main import ( "crypto/rand" "encoding/hex" "fmt" "path" ) type Candidate struct { TenantID string ProductID string JobID string AttemptID string ObjectKey string } type ObjectStore interface { Put(key string, content []byte, contentType string) error Copy(sourceKey, destinationKey string) error } func newAttemptID() (string, error) { var raw [16]byte if _, err := rand.Read(raw[:]); err != nil { return "", err } return hex.EncodeToString(raw[:]), nil } func newCandidate(tenantID, productID, jobID string) (Candidate, error) { attemptID, err := newAttemptID() if err != nil { return Candidate{}, err } return Candidate{ TenantID: tenantID, ProductID: productID, JobID: jobID, AttemptID: attemptID, ObjectKey: path.Join("candidates", tenantID, jobID, attemptID+".png"), }, nil } func publish(store ObjectStore, candidate Candidate, tenantID string) error { if candidate.TenantID != tenantID { return fmt.Errorf("tenant boundary rejected") } canonicalKey := path.Join("published", tenantID, candidate.ProductID, "image.png") return store.Copy(candidate.ObjectKey, canonicalKey) } The copy in this example is only the delivery step. In a real service, the catalog transaction must record the selected candidate and the canonical reference under the same serialized decision, or the read path must resolve the candidate directly from that record. A retry of Put uses the same candidate only when the attempt itself is being resumed; a new delivery of the job receives a new attempt ID.
Keep tenant authorization outside the key convention as well. Prefixes make audits and lifecycle operations easier, but a prefix check is not a complete access-control policy. The read and publication paths should authorize the tenant from authenticated request context, then verify that the selected product and candidate belong to that tenant. This matters for both accidental cross-tenant reads and cleanup routines that operate by prefix.
How can a runbook verify retry collisions and roll back safely?
Start with the catalog record. Capture the tenant, product, job, attempt, selected candidate key, object checksum if available, and the state transition timestamps. Then list candidates under the job's private namespace and compare them with the database. Do not start by clearing a cache or regenerating the image; that destroys evidence and can add another competing attempt.
Reproduce the sequence in a non-production tenant by forcing a client timeout after the storage adapter has accepted the bytes. Verify that the first attempt remains addressable, that a retry receives a distinct key, and that only one serialized publication decision becomes visible. Test a stale worker too: it should be able to finish storing its candidate, but it must not replace a newer published selection.
Rollback is a database decision plus a retained candidate. If the current published image is wrong, select a known-good candidate under the same authorization and serialization rules, then update the catalog pointer. With no object versioning, an accidental same-key overwrite has no storage-only undo. If the earlier bytes were never retained under another key, recovery may be impossible; the runbook should say that plainly and alert on the risk.
Deletion deserves the same discipline. Maintain an inventory from tenant and job records to candidate keys, then delete candidates only after no catalog reference or retention hold needs them. GDPR Article 17 establishes the right-to-erasure context, but it does not supply the inventory: the application still has to know which objects belong to a person, tenant, or product. An opaque key without a durable mapping makes that work needlessly uncertain.
Your mileage may vary on retention duration. Set it from the product's support window, regulatory obligations, and the maximum time a stale worker can return; then measure storage growth from actual attempt counts. A short retention period reduces cost but narrows the recovery window, while an indefinite candidate archive turns every retry into a permanent capacity commitment.
Which storage boundary fits access control versus delivery simplicity?
This is a buy-versus-build choice about ownership. A managed object store can simplify delivery and durability operations, while a self-hosted or file-system-oriented design may provide controls your team already understands; neither choice fixes a key scheme that conflates a product with a render attempt. AWS describes EFS as a managed file system, which is a useful reminder that file-system semantics and object-key semantics are different contracts.
| Boundary | Good fit | Trade-off to validate |
|---|---|---|
| Object storage with application publication state | Tenant-scoped images, immutable candidates, and controlled delivery | The application owns idempotency, authorization, and catalog consistency |
| Managed file system | Workloads that need shared file-system behavior and existing POSIX-oriented tooling | Delivery, tenant isolation, and concurrent publication still need an explicit policy |
| Self-hosted object service | Teams willing to own capacity, upgrades, repair, and access controls | The on-call burden and recovery testing become platform responsibilities |
The catch is delivery simplicity. A design that is correct for private tenant downloads may be unsuitable for permanent public URLs, browser uploads, or a regulated archive if it lacks the required policy, retention, or replication controls. Stick with the boundary that can meet the image-delivery SLO and recovery requirement your team can actually operate. Do not turn a convenient latest.png path into a durability guarantee.
