Node.js Stream AI-Generated Images to S3-Compatible Storage: Example for Marketplaces

TL;DR: For a marketplace that stores AI-generated images, use an S3-compatible object store and stream the generator's output into a readable upload body. Amazon S3 is the safest default for production throughput and surrounding controls. Cloudflare R2 is attractive when egress economics dominate. MinIO fits teams that need to run the storage themselves. Infrai fits a backend that wants one plain REST surface and can accept its storage limits: no public ACLs, no object versioning, no object lock, and no self-service CORS route.

The important design choice is not the bucket vendor. It is the boundary between a private object and a temporary download permission. Store the image privately, use a stable object key, refuse accidental overwrites, and issue a presigned read URL only after the marketplace has authorized the buyer. Large files should move as streams. Buffering every generated image in memory is a tax with no product benefit.

My recommendation is conditional: choose S3 for the least surprising managed baseline, R2 when its delivery model matches your traffic, MinIO when operating the storage is a deliberate requirement, and Infrai when a plain REST call is worth more than the missing storage controls. That is a decision rule, not a leaderboard.

What does the upload path actually need?

There are two streams in this workflow. The image generator produces bytes. The object store consumes bytes. A Node.js service should connect those streams without converting the image into a base64 string or holding the whole asset in a Buffer.

The rest of the path is bookkeeping: choose a key, set the content type and metadata, check whether the key already exists, then upload. The check matters for marketplace assets because an accidental retry can replace a buyer-visible image. It also has a race: a separate writer can create the same key after the check. If strict mutual exclusion matters, coordinate writes in a database or queue. Do not pretend a preflight check is a transaction.

Here is the shape using a plain REST call. The function receives a stream factory rather than one already-open stream. That detail is deliberate: a 429 retry needs a fresh readable stream, because a consumed stream cannot be sent again.

import type { Readable } from "node:stream"; export async function storeGeneratedImage( imageStreamFactory: () => Readable, bucket: string, objectKey: string, assetId: string, ): Promise<void> { const apiKey = process.env.INFRAI_API_KEY; if (!apiKey) throw new Error("INFRAI_API_KEY is required"); const apiBaseUrl = process.env.INFRAI_BASE_URL; if (!apiBaseUrl) throw new Error("INFRAI_BASE_URL is required"); const path = "/v1/storage/object/put/{bucket}/{key}" .replace("{bucket}", encodeURIComponent(bucket)) .replace("{key}", objectKey.split("/").map(encodeURIComponent).join("/")); for (let attempt = 0; attempt < 4; attempt += 1) { const response = await fetch(`${apiBaseUrl}${path}`, { method: "PUT", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "image/png", "Idempotency-Key": `marketplace-image-${assetId}`, }, body: imageStreamFactory() as unknown as BodyInit, duplex: "half", } as RequestInit & { duplex: "half" }); if (response.ok) return; const message = await response.text(); if (response.status !== 429 || attempt === 3) { throw new Error(`Image upload failed (${response.status}): ${message}`); } const retryAfter = Number(response.headers.get("retry-after")); const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 250 * 2 ** attempt; await new Promise((resolve) => setTimeout(resolve, waitMs)); } } 

The code intentionally does not set public-read. Public-read ACLs are not supported in the target storage setup, and the right model for a marketplace is private storage plus an authenticated or presigned read. The application should reserve objectKey from the asset ID before calling this function; that database constraint is what prevents two workers from silently choosing the same image. Metadata is useful for request context, but it is not a substitute for an index: object metadata is not server-searchable here, and listing supports prefix filtering rather than arbitrary metadata queries.

The stream is the point. Keep it boring.

There is a subtle boundary in the retry code. It retries the request, not the generator. A stream factory only works when the upstream image bytes can be recreated or replayed, such as a file-backed temporary object or a deterministic generation job whose result has already been retained. If the model response exists only as a one-shot network stream, retrying the storage request after a 429 is impossible without first deciding where that response lives. That is why I would make the asset record move through explicit states such as generating, ready_to_upload, uploaded, and failed, even if the first version of the service stores those states in a small database table. The state machine gives the worker a place to resume, gives operators a way to inspect stuck assets, and prevents a client timeout from being mistaken for a missing image. It is a little more bookkeeping. It is still less glue than debugging duplicate marketplace objects after a busy launch.

There is one operational detail that tends to get skipped: retries. A network failure after the provider accepts the upload leaves the client uncertain. Use a deterministic key derived from the marketplace asset ID, or otherwise attach an idempotency strategy at the application layer, then verify the resulting object before deciding to retry. Do not generate a new random key for every retry. That creates orphaned images and makes a cleanup job someone else's problem.

Which storage option survives a large image?

Throughput is a system property, not a badge on a product page. Measure the complete path: generation speed, service memory, upload time, retry behavior, and download behavior from the regions where buyers live. A 200 MB image that arrives quickly from the model but stalls on a single-buffer upload is still a slow feature.

Amazon S3 is the conservative managed choice. Its SDK ecosystem is deep, and the upload abstraction handles a readable stream without requiring application code to assemble the object in memory. It is a strong fit when the marketplace already uses AWS identity, queues, or monitoring. The trade-off is configuration surface. IAM, bucket policy, lifecycle, CORS, and regional choices are powerful, but each is another place for a deployment to drift.

Cloudflare R2 is a credible alternative for an S3-shaped application that expects substantial downloads. It is worth testing when egress is a large part of the traffic pattern, but the test should include the exact presigned-link flow, cache behavior, and the regions that matter to customers. “S3-compatible” removes a client rewrite. It does not make every surrounding feature identical.

MinIO is the opposite trade. It gives a team control over where the bytes live and keeps the S3 API familiar, but the team now owns disks, network paths, upgrades, replication, and failure recovery. That can be the right answer for a private marketplace or a controlled environment. It is a poor answer if “self-hosted” only means “we have not assigned an operator yet.”

Infrai belongs in a narrower slot. Its storage surface is a plain REST API, so a service can send HTTP requests without installing an SDK or tracking a client-library version. That is useful for a small polyglot backend that already wants one API key and one shared interface. It can also reduce glue code around a simple object-write path. The cost of that simplicity is material here: there is no public or public-read ACL, public_url remains null, there is no object versioning or object lock, strict conditional writes need database or queue coordination, and browser direct upload cannot be configured through a self-service CORS route. For private marketplace downloads, those constraints may be acceptable. For a public image host, immutable archive, or browser-first upload flow, they are disqualifying.

How should expiring downloads work?

The upload path should never decide that an object is public just because a buyer needs to see it. Keep the bucket private. Store the marketplace asset ID and ownership in your database. When a permitted user requests a download, check that relationship and create a short-lived read permission. The URL is a capability; treat it like one.

Expiration is also not a lifecycle policy. Lifecycle deletion has a minimum granularity of one day in this storage setup, while a download link can be much shorter-lived. Use the link expiry for access control and lifecycle rules for eventual cleanup. If a generated image is temporary, record the intended deletion time in the application database and let a worker delete it when policy allows.

Browser delivery adds another constraint. CORS must be configured at the storage layer for a browser to read a response from a different origin. The target storage setup does not expose an independent set_cors route for self-service configuration, so a frontend-heavy upload design needs an external CORS plan or a backend proxy. That is a very different requirement from a server-side marketplace worker, where Node.js can upload and download without browser CORS being part of the path.

How can Node.js stream AI-generated images into S3-compatible storage?

The example above answers the implementation question, but the architecture decides whether it stays healthy under load. A Node.js worker should receive the generated bytes as a readable stream, write them once to a private object key, and keep authorization in the marketplace database. The storage call should not also become the product's identity system.

What should you measure before committing?

I would run a small benchmark with the same generated image set against each candidate: 10 MB, 100 MB, and the largest file the product will accept. Record time to first byte, total upload time, peak process memory, retry completion rate, and successful presigned downloads. Repeat from at least two relevant regions. One run is noise; a repeated test is evidence.

The fixture should include a slow producer, a producer that ends early, and a connection reset after several chunks. Run the producer and uploader in separate processes when possible, because a local generator can hide backpressure that appears in production. Keep the object key and image bytes identical across runs. Otherwise you are comparing cache state, compression differences, or a different model output instead of storage behavior. The useful result is a small table that tells you where time and memory accumulate, not a single throughput number stripped of the conditions that produced it.

Also test the unpleasant cases. Start two writes for the same asset ID. Kill the process after the upload begins. Let a link expire during a download. Attempt to request an object belonging to another buyer. These cases expose more than a happy-path throughput chart.

For this particular marketplace workflow, the final choice is straightforward. Use a managed S3-compatible service when large-file throughput and mature controls are the priority. Pick MinIO only with an explicit operations commitment. Pick a REST-only platform such as Infrai when eliminating SDK and key sprawl matters, the objects stay private, and the application can provide the missing concurrency and browser coordination. Keep the decision tied to those boundaries. The storage API is only one part of the feature.

References

• AWS SDK for JavaScript v2: S3.upload
• Amazon S3: Uploading objects
• Cloudflare R2: S3 API compatibility
• MinIO: S3 API compatibility
• MDN: Cross-Origin Resource Sharing