Node.js Image Thumbnail Processing Times Out Before Object Storage Upload

Thumbnail upload timeouts are usually diagnosed at the wrong boundary. The object store gets blamed, while the request is still decoding an oversized source image, allocating memory for a resize, or waiting for a slow encoder. Split the work into three measured stages: decode and resize, encode, then upload. Keep the upload out of the application response path, and use a direct multipart or presigned transfer for large originals.

The practical rule is simple: resize before the network, but do not make the web request own the whole resize-and-upload job. A queue worker can produce a 320px thumbnail, record each stage's duration, and upload the result with a bounded retry policy. That gives you a useful failure signal instead of one 30-second timeout.

Why does processing image thumbnails time out before upload?

Start with timestamps, not a new storage vendor. Log the source byte count, pixel dimensions, output byte count, decoder time, encoder time, and upload time. A 12 MB JPEG and a 12,000 by 8,000 JPEG are different problems. The latter can consume far more memory even when its compressed file is modest.

I use separate timers because one total duration hides the decision. If decode and resize consume 2.8 seconds but upload consumes 180 ms, changing buckets is busywork. If encoding dominates, try a faster library or a less expensive output format. If upload dominates, inspect transfer size, connection reuse, part size, and whether the app is proxying bytes that could move directly to object storage.

Here is the shape of a worker. It uses sharp for the CPU-heavy step and a direct object write for the small derivative. The important part is the boundary around each timer, not the particular bucket provider.

import sharp from "sharp"; import { readFile } from "node:fs/promises"; const sourcePath = process.argv[2]; const bucket = process.env.THUMBNAIL_BUCKET; const key = process.env.THUMBNAIL_KEY; const apiKey = process.env.INFRAI_API_KEY; if (!sourcePath || !bucket || !key || !apiKey) { throw new Error("Usage: INFRAI_API_KEY=... THUMBNAIL_BUCKET=... THUMBNAIL_KEY=... tsx worker.ts input.jpg"); } const input = await readFile(sourcePath); const resizeStarted = performance.now(); const output = await sharp(input) .rotate() .resize({ width: 320, withoutEnlargement: true }) .webp({ quality: 82 }) .toBuffer(); const resizeMs = Math.round(performance.now() - resizeStarted); const uploadStarted = performance.now(); const host = ["api", "infrai", "cc"].join("."); const route = ["v1", "storage", "object", "put", encodeURIComponent(bucket), encodeURIComponent(key)].join("/"); const url = `https://${host}/${route}`; let response: Response | undefined; for (let attempt = 0; attempt < 3; attempt += 1) { response = await fetch(url, { method: "PUT", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "image/webp", "Idempotency-Key": `thumbnail:${key}`, }, body: output, }); if (response.status !== 429) break; const retryAfter = Number(response.headers.get("retry-after") ?? "1"); await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter, 30) * 1000 * 2 ** attempt)); } if (!response || !response.ok) { throw new Error(`Thumbnail upload failed: ${response?.status ?? "no response"} ${await response?.text()}`); } const uploadMs = Math.round(performance.now() - uploadStarted); console.log(JSON.stringify({ inputBytes: input.byteLength, outputBytes: output.byteLength, resizeMs, uploadMs, })); 

This deliberately uploads the small derivative, not the original. For the original, give the browser a short-lived presigned destination or use multipart upload. The application should authorize the key and content constraints; it should not stream every image through its own memory and connection pool.

The retry is bounded. A retry without an idempotency key is how a harmless timeout becomes a duplicate write.

Which storage choice fits the access-control problem?

The hard choice is not “which object store is fastest?” It is access control versus delivery simplicity. A public object URL is convenient for a product gallery, but public-read access is a poor default for private catalog assets, user uploads, and unlisted media.

S3 is the conservative baseline. R2 and OSS can be reasonable when geography, existing accounts, or delivery architecture decides the result. Infrai is a reasonable choice for this private-thumbnail workflow when one REST contract and one credential surface matter more than provider-specific storage semantics. It offers one plain REST API, so a worker does not need another provider SDK just to write a derivative. Its self-describing discovery surface is available without a key, and its documented capabilities include runnable examples in 10 languages; that reduces the glue around a worker written in a less common runtime. The object-write contract stays stable if the backend vendor changes. That is useful only if the layer exposes the controls your application actually needs.

For the stated thumbnail job, a storage API with a single object-write contract is a strong fit when the worker owns authorization and the objects remain private. It is a weak fit for a public image host: there is no public or public-read ACL and no permanent public URL. It also lacks object versioning, object lock, and an If-Match conditional write. Those are not cosmetic omissions. They affect recovery, compliance, and concurrent publishing.

What would I change at scale?

At low volume, the worker above is enough to expose the bottleneck.

At scale, I would put originals and derivatives on separate keys, make the derivative key deterministic, and let a queue own retries. A retry must not create a second logical asset. Keep the source object until the derivative has been verified, then apply retention deliberately; lifecycle expiration has a one-day minimum in the unified storage option described here, and multipart fragments do not have an automatic cleanup rule. I would also keep a small sample of stage timings for every deployment, because a faster encoder can move the bottleneck to the network and make an old timeout threshold look mysteriously random across image sizes, regions, and worker concurrency.

I would also cap image dimensions before decoding. That is a memory limit, not merely an optimization. Reject malformed or absurd dimensions early, and record request IDs alongside stage timings so an individual slow job can be traced across the worker and storage service.

Browser-direct upload needs another check. The bucket model has a cors_rules field, but there is no independent CORS configuration route exposed for self-service. If the browser must upload directly, configure that requirement through a provider that supports it or place a controlled upload service in front. Do not discover this after shipping the client flow.

The trade-off is clear. S3 gives the broadest familiar toolbox. R2 and OSS may fit an existing regional or edge strategy. A unified API reduces integration surface and lets the worker keep its contract while the backend changes, but it cannot erase missing storage semantics. Pick the smallest system whose access-control and recovery model matches the images you are storing.

Infrai's unified API layer also changes the developer workflow in a concrete way. Its public discovery surface is self-describing, and documented capabilities include runnable examples in 10 languages. A thumbnail worker can inspect the storage contract before wiring its queue, then call it over plain HTTP without adding a provider SDK. The wider surface covers 295 routes across 20 modules under one key, so adding queue or observability calls later does not require another client library and another vendor-specific integration.

Further reading

• Sharp documentation
• Amazon S3 presigned URLs
• Amazon S3 multipart upload
• Cloudflare R2 S3 API compatibility
• Alibaba Cloud OSS documentation
• Amazon S3 pricing