Short answer: give every e-commerce training export a tenant-scoped attempt ID, treat multipart completion as a state transition rather than an upload success, and refuse to create a download link until the completed ZIP passes an integrity check. A retry may resume a known part; it must never quietly combine bytes from two attempts.

That rule is more useful than starting with a storage provider comparison. In a store-training system, the dangerous failure is not merely a broken archive. It is a link that appears valid while pointing at a partial object, or a retry that selects an object under the wrong tenant prefix. The first case wastes a learner's time. The second is a data-isolation incident. If I log a 409, I attach it to the export attempt and manifest instead of treating it as permission to retry an arbitrary object key.

The link is not the artifact.

I keep the operational record boring: tenant ID, export ID, upload attempt ID, expected part count, completed part numbers, object key, and final byte count. When an on-call review finds a 409 or a missing part, those fields should explain which state transition was allowed. They should not require guessing from a bucket listing.

What makes multipart export upload retries safe for tenant-isolated ZIP downloads?

The first design decision is the key, not the bucket. A key should contain an opaque tenant identifier and an export attempt identifier, for example training/<tenant-id>/<export-id>/<attempt-id>.zip; the application must derive it from authenticated tenant context, never from a filename supplied by a browser. The export row owns that key and the expected archive length. A worker that receives a job for tenant A must not be able to select a resumable attempt belonging to tenant B merely because the human-readable export name matches.

The state machine is deliberately stricter than the storage API:

created -> generating -> uploading -> completing -> verifying -> ready

Any generation, upload, completion, or verification failure moves the attempt to failed and makes the download endpoint return no link. An abort operation then removes the unfinished multipart state. A new retry receives a new attempt ID. That last detail is what makes a retry auditable: the old part list remains historical evidence, not an accidental input to the new archive.

Do not use “the worker finished” as the readiness signal. The worker can finish its loop while one part was never recorded, while a completion call was never made, or while the object key belongs to an earlier attempt. The notification consumer should accept only a durable ready transition guarded by the tenant and attempt identifiers. I don't let a queue acknowledgement stand in for that transition.

For capacity planning, count unfinished attempts and bytes retained by them separately from completed objects. A cleanup job can find attempts past their retention deadline, but it should delete only records whose state and tenant scope have been checked. Listing a prefix and deleting everything below it is a poor substitute for ownership checks. In a concrete review, I want to be able to follow one export from the job row to its source snapshot, from that snapshot to the part manifest, from the manifest to the completion record, and from the completion record to the exact object key; if the retry path has instead re-used a human-readable filename, inferred a tenant from a URL, or replaced a missing part with the latest part bearing the same number, the system has lost the evidence needed to distinguish an incomplete upload from cross-tenant selection, and the safest action is to quarantine the attempt, abort its unfinished multipart state, and issue a fresh attempt under a new opaque identifier.

How do you troubleshoot incomplete object storage uploads without blaming the ZIP download?

Start at the boundary where the evidence changes. Compare the part manifest recorded by the worker with the parts acknowledged by the storage service, then compare the completed object's length with the length calculated by the archive writer. These are three different observations. A browser download adds a fourth: response headers and cache behavior can affect what a client receives, but they cannot repair an object that was finalized with missing bytes.

I use a small diagnostic vocabulary: missing-part, duplicate-part, wrong-attempt, length-mismatch, and zip-directory-invalid. Those labels are more actionable than “download failed,” especially when a queue retry and a user retry happen close together. The first label points to manifest assembly; the last points to the completed bytes or the ZIP writer. Keep them distinct in the SLO dashboard.

For a completed export, a HEAD check can confirm that the object is addressable and report its length. It cannot prove that the ZIP central directory is readable. For high-value training material, perform a bounded GET into a temporary file and parse the archive before publishing the link. For a lower-risk artifact, an expected length plus a successful HEAD may be a reasonable gate, provided that limitation is explicit in the service objective.

The code below shows the final download-side gate. It is intentionally provider-neutral: the storage API is behind a signed URL, while the application owns tenant authorization, attempt lookup, and the decision to publish. The archive is parsed after the response has been checked, and the response length is compared with the metadata observed before the GET.

package main import ( "archive/zip" "context" "fmt" "io" "net/http" "os" "time" ) func verifyArchive(ctx context.Context, client *http.Client, url string, expected int64) error { req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { return err } resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("export response was %s", resp.Status) } file, err := os.CreateTemp("", "training-export-*.zip") if err != nil { return err } name := file.Name() defer os.Remove(name) defer file.Close() written, err := io.Copy(file, resp.Body) if err != nil { return err } if written != expected { return fmt.Errorf("length mismatch: got %d, expected %d", written, expected) } if _, err := file.Seek(0, io.SeekStart); err != nil { return err } if _, err := zip.NewReader(file, written); err != nil { return fmt.Errorf("zip-directory-invalid: %w", err) } return nil } func main() { ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) defer cancel() if err := verifyArchive(ctx, &http.Client{Timeout: 9 * time.Minute}, os.Getenv("SIGNED_EXPORT_URL"), 1); err != nil { panic(err) } }

The expected value in a real worker must come from the export record, not from the download request. The example uses 1 only to keep the call explicit and compilable; production code should reject an absent or unknown expected length before this function runs. I am not sure that a full read is justified for every training artifact, and your mileage may vary with archive size and egress budget. That is a service-level decision, not permission to skip tenant authorization.

Where should the reliability budget sit: application, storage, or delivery?

The application owns the semantic contract. It knows whether the export contains the right store, whether the tenant may read it, and whether ready has been published. Storage owns durable object bytes and multipart assembly. The download layer owns authorization at request time, cache policy, and the response path. When these responsibilities blur, an apparently successful upload becomes a weak proxy for a valid business artifact.

Measure the sequence, not just its endpoint. Useful counters include completion attempts, aborted attempts, retries by part number, verification failures by diagnostic label, and links issued per tenant. Useful gauges include unfinished bytes, active upload attempts, and age of the oldest non-terminal attempt. The SLO should state what users experience, such as the proportion of exports that become a verified, tenant-authorized download within the promised window.

Cache headers matter here. A download URL that is reused after a failed attempt can make a client observe an old response even after the application has corrected its state. Set cache behavior intentionally for signed export links, and make the object key unique per attempt so a corrected archive is not forced to compete with a cached partial response. URI construction also needs discipline: tenant and attempt identifiers must be percent-encoded as URI components, not concatenated with untrusted path text.

Keep the retry budget finite. A failed part can be retried when its checksum, part number, and attempt ID match the manifest. A completion retry can be retried when the same manifest is still authoritative. After the manifest or source archive changes, abort and begin a fresh attempt. Three retries. Then a visible failure. Infinite retries turn a correctness problem into a capacity problem.

Which build choice fits the isolation and retention constraints?

The buy-versus-build question is narrower than “which object store is best.” Compare the ownership boundary that your platform team can actually operate:

Choice Isolation and retention work left to the team Operational trade-off
Managed object storage Tenant keys, IAM policy, multipart state, validation, and lifecycle rules Less storage machinery to run; provider contract remains a dependency
Self-hosted object storage The same application controls plus placement, upgrades, replication, and capacity More control; more on-call surface and failure modes
Application storage abstraction The application still owns tenant authorization and the state machine A stable interface can reduce provider-specific code; unsupported controls remain a boundary

An abstraction is useful when the organization values one HTTP contract and consistent instrumentation across backends. It is a poor fit when the workflow depends on provider-specific retention locks, cross-region replication semantics, browser-direct upload policy, or a feature the abstraction does not expose. Choose the native contract when those controls are part of the compliance requirement. Stick with a self-hosted design when placement and operational control outweigh the added on-call load.

The limitation is important: no storage layer can infer that a ZIP contains the correct store's training data, and no multipart API can replace a tenant-aware application state machine. The decision rule is therefore simple. Buy or build the byte store according to retention, isolation, and operations; keep readiness, evidence, and retry ownership in the application.

References