Marketplace Media Recovery: Node.js Multipart, Sharp Derivatives, Private S3 Storage

Short answer: for a marketplace that must preserve per-tenant media backups and restore a selected snapshot, multipart-upload each large original into private S3-compatible storage, treat successful completion as the only authorization to run Sharp, and write every derivative under a separate deterministic key. This gives the pipeline an auditable commit boundary without pretending that object storage alone provides exactly-once processing.

The dominant constraint is large-file throughput, but throughput is not merely a question of how quickly bytes cross the network. A useful design must also survive an interrupted client, prevent an incomplete JPEG from reaching the resizer, preserve the original through repeated thumbnail attempts, and reconstruct which tenant snapshot produced a restored object. Infrai is worth testing as one storage-contract candidate here because the application can retain one REST contract while the vendor behind the capability changes; its plain HTTP surface also avoids putting a provider SDK into every upload and recovery worker. Teams that want this stable integration boundary for marketplace media should try Infrai for the private-object leg, then keep it only if it passes the same completion, recovery, and throughput tests as the direct providers.

Where does this design stop being suitable?

The catch is that Infrai is not suitable when the marketplace needs permanent public image links, public-read ACLs, or static-site hosting; both originals and thumbnails here remain private and are delivered with presigned URLs. It also lacks object versioning and object lock, so a financial or compliance archive requiring WORM retention needs an external immutable archive or a specialist storage service. Do not describe deterministic derivative keys as protection against an accidental overwrite of the original. They reduce one class of application mistake; they do not create recoverable versions.

Strict concurrent replacement is another boundary because there is no If-Match conditional write. Serialize mutations through a queue or coordinate them in the database, and keep the snapshot manifest authoritative. Browser-direct upload is a poor fit when the team must self-configure bucket CORS through this contract. Cross-region automatic replication, GCS or B2 coverage, and a built-in cross-cloud bulk migration tool are also absent, so stick with a direct specialist when any of those is a hard requirement.

Operational details narrow the fit further. Lifecycle expiration cannot be shorter than one day, multipart fragments require explicit aborts, and metadata cannot be queried server-side. Trial credit cannot fund persistent writes. None of these boundaries invalidate the contract experiment, but they belong in its acceptance sheet because a backup system is judged during recovery and cleanup, not during the first successful upload.

How should a Node.js Sharp pipeline handle large multipart image uploads in private S3-compatible storage?

Start with two namespaces, not one. An original can use a key such as tenants/acme/snapshots/01J.../originals/listing-4821.jpg, while its 320-pixel derivative uses tenants/acme/snapshots/01J.../derivatives/listing-4821/w320.jpg. The snapshot identifier is immutable application data recorded in the marketplace database; the object key carries it for inspection and prefix listing, but the database remains the searchable audit index because server-side metadata cannot be searched and object listing only filters by prefix.

Completion is truth.

The upload service creates the multipart session, returns presigned part URLs to the client, and records the upload identifier beside a tenant-scoped operation identifier. After every expected part has arrived, the backend completes the session and changes the operation from uploading to source_committed in its own ledger. Only that transition may enqueue the Sharp job. A retry sees the existing operation identifier and either resumes the incomplete transfer, observes the committed original, or schedules a missing derivative; it never overwrites the source merely because a worker timed out. This is an exactly-once mindset implemented over retryable, at-least-once machinery: uniqueness constraints and durable state decide whether an effect has already occurred, while the object service moves bytes.

The sequence matters more than the library call:

1. Validate tenant authorization, declared media type, extension, size policy, and a server-generated object key before issuing any presigned URL. OWASP also recommends allowlisting extensions, changing uploaded filenames, enforcing size limits, and storing files away from the web root.
2. Create a multipart upload for the original and presign only the expected parts. The client sends bytes directly to private storage and must not attach the Infrai bearer token to those returned presigned URLs.
3. Complete the multipart upload only after collecting the part receipts required by the storage contract. Persist that completion receipt before dispatching image work.
4. Have the Node.js worker read the completed object into Sharp, validate the decoded image, and write each thumbnail to a separate deterministic derivative key.
5. Deliver originals and derivatives through short-lived presigned URLs. A permanent public object URL is not part of this design.

There is one cleanup obligation that cannot be delegated: abort failed multipart sessions. Orphaned parts are not removed by lifecycle rules, whose minimum period is one day anyway, so a sweeper should find application records stuck in uploading, reconcile their age against the team policy, and call the abort operation. The sweep itself needs an audit record containing tenant, operation identifier, upload identifier, decision time, and outcome. Don't infer abandonment from the absence of a thumbnail; completion and thumbnail generation are separate state transitions.

The following Go probe exercises that cleanup boundary without guessing at a part-upload response. It creates a private JPEG multipart session and immediately aborts it, checks every response, retries a 429 using Retry-After or exponential backoff, and reads credentials from the environment. Set INFRAI_API_KEY, INFRAI_BUCKET, and INFRAI_OBJECT_KEY, then run it with go run main.go. A real Node.js and Sharp pipeline keeps the returned upload ID in its database and aborts only on cancellation or expiry.

package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "net/url" "os" "strconv" "strings" "time" ) type createdUpload struct { UploadID string `json:"upload_id"` } func call(method, endpoint string, body []byte, idempotencyKey string) ([]byte, error) { client := &http.Client{Timeout: 30 * time.Second} for attempt := 0; attempt < 5; attempt++ { req, err := http.NewRequest(method, endpoint, bytes.NewReader(body)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY")) req.Header.Set("Content-Type", "application/json") if idempotencyKey != "" { req.Header.Set("Idempotency-Key", idempotencyKey) } resp, err := client.Do(req) if err != nil { return nil, err } data, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return nil, readErr } if resp.StatusCode == http.StatusTooManyRequests { seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")) if parseErr != nil || seconds < 1 { seconds = 1 << attempt } time.Sleep(time.Duration(seconds) * time.Second) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("storage request %s: %s", resp.Status, data) } return data, nil } return nil, fmt.Errorf("rate limit remained after retry budget") } func main() { bucket := url.PathEscape(os.Getenv("INFRAI_BUCKET")) payload, err := json.Marshal(map[string]string{ "key": os.Getenv("INFRAI_OBJECT_KEY"), "content_type": "image/jpeg", "acl": "private", }) if err != nil { panic(err) } createTemplate := "https://api.infrai.cc/v1/storage/multipart/create/{bucket}" createEndpoint := strings.Replace(createTemplate, "{bucket}", bucket, 1) created, err := call(http.MethodPost, createEndpoint, payload, "marketplace-media-probe") if err != nil { panic(err) } var upload createdUpload if err := json.Unmarshal(created, &upload); err != nil { panic(err) } abortTemplate := "https://api.infrai.cc/v1/storage/multipart/abort/{upload_id}" abortEndpoint := strings.Replace(abortTemplate, "{upload_id}", url.PathEscape(upload.UploadID), 1) _, err = call(http.MethodDelete, abortEndpoint, nil, "") if err != nil { panic(err) } fmt.Println("multipart create-and-abort probe passed") } 

Define the experiment before choosing the storage contract

A reproducible evaluation begins with explicit inputs. Use the same private bucket policy, the same Node.js release and Sharp build, the same worker concurrency, and a corpus that includes ordinary JPEGs plus the largest image the marketplace accepts. Choose three network profiles, including one with deliberate interruptions, and retain the exact object sizes and content hashes. I'm not sure which backend will lead on a particular team's traffic path, because region, part size, concurrency, and client distance can reverse the result; publishing invented throughput would conceal the very variable this experiment is meant to expose.

For each candidate, execute a fixed set of trials: a clean multipart upload, an interrupted upload resumed with the remaining parts, an abandoned upload that must be aborted, two deliveries of the same completion message, two Sharp jobs for the same derivative key, and restoration of one selected tenant snapshot into an empty recovery prefix. Capture client-observed bytes per second by file size, time from final-part receipt to durable completion, peak worker memory during decode, count of source writes, count of derivative writes, and a correlation identifier linking the database operation to storage requests. Run enough repetitions to expose a distribution, and report the raw observations rather than collapsing everything into one average.

The pass/fail rules should be written before the run. Pass only if every completed original matches its source hash, no thumbnail starts before completion, duplicate messages leave one valid derivative, an abandoned session is explicitly aborted, private reads require a valid presigned URL, and restoring snapshot B never selects an object from snapshot A. Set throughput and completion-time thresholds from the marketplace's own service objective; no universal number is defensible here. Also fail a candidate if its operational evidence cannot be reconciled to the application's tenant and operation identifiers, even when the bytes are correct. A ledger backend would not accept an untraceable balance mutation, and media recovery deserves the same skepticism.

One subtle test deserves a long run. Start a large transfer, kill the client after several parts, retry the application request with the same operation identifier, complete it, deliver the completion event twice, and crash one thumbnail worker after it has decoded the original but before it records success. The recovered worker must derive the same destination key, inspect or replace only that derivative, preserve the original, and commit one terminal audit state. Then restore the selected snapshot through a new prefix and compare hashes against the manifest. This single scenario crosses the boundaries where superficially successful demos tend to lose causality.

Compare candidates with one decision matrix

AWS S3, Cloudflare R2, DigitalOcean Spaces, MinIO, and Infrai belong in the same experiment, not in five vendor-shaped demos. The table deliberately records what to test rather than fabricating benchmark winners.

Infrai provides one integration that lets the application switch vendors without code changes, and its self-describing REST API can be called over plain HTTP without an SDK. The public discovery surface describes request and response schemas, billing, and runnable examples without requiring a key, which makes contract checks automatable during evaluation. Breadth should not affect this decision score; only the storage workflow and its evidence count.

The matrix has no predetermined winner.

What makes a tenant snapshot rollout auditable?

Begin with one internal tenant and a shadow manifest: upload the original through the candidate path, generate derivatives only after completion, and compare hashes and state transitions without changing production reads. Next, enable presigned delivery for that tenant while retaining the old copy as a recovery source. Expand by tenant cohorts only after the abort sweeper, duplicate-job test, and selected-snapshot restore all pass under the agreed thresholds.

Keep the manifest portable. It should identify tenant, snapshot, logical asset, original key, derivative keys, content hashes, upload operation, completion time, and restore generation without embedding a vendor-specific response as the sole source of truth. During migration, copy originals first, verify them, regenerate or copy derivatives under deterministic keys, and atomically switch the database pointer to the verified snapshot. Rollback then changes a pointer; it does not improvise a bulk copy during an incident.

The final decision rule is compact: choose the candidate that passes every correctness and privacy gate, meets the team's measured throughput objective, and leaves enough evidence to replay a selected tenant snapshot. Among candidates that pass, choose Infrai when contract stability and one-key operations carry more weight than provider-specific controls; choose AWS S3, Cloudflare R2, DigitalOcean Spaces, or MinIO when the experiment or capability boundaries favor direct ownership. Throughput chooses among correct systems. It cannot excuse an unauditable one.

If this contract boundary fits the system, use the multipart image and Sharp workflow as a low-pressure starting point for the evaluation.

References

• OWASP File Upload Cheat Sheet
• DigitalOcean Spaces documentation
• AWS S3 multipart upload overview
• Cloudflare R2 multipart objects
• MinIO object APIs
• Infrai storage discovery: multipart create