The constraint that settles this argument is not the protocol, it is the size distribution of what you accept. If the median training artifact your customers push is a 40 MB checkpoint and the p99 is a 60 GB dataset shard, the upload path has to survive the p99 without taxing the median — which pushes you toward presigned multipart parts sent direct from the browser to object storage, backed by a signing service that does nothing except sign. Pick the part size from the largest object you will ever accept, publish that number to the client, and treat it as a capacity decision rather than a tuning knob.

Everything below follows from that one number.

The scenario I'm writing against is a SaaS platform team with five engineers, a retention policy that auditors will eventually read back to us, and a Node.js API that has no business moving 60 GB through its own heap. Artifacts arrive from a browser because that is where the data scientists are. They have to be retained, expired, and re-derived on a schedule that somebody can reconstruct from a git repo instead of from memory.

The part-size ceiling, and the bill behind it

The failure is rarely dramatic. A rolling deploy lands, an in-flight 22 GB shard dies at 80%, the researcher retries from zero, and your API pods spend the afternoon holding gigabytes of somebody else's tensors in memory. Every deploy is an upload failure window, and your ingest throughput ceiling is now whatever your API tier can push — an odd thing to have paid for, given that the storage layer was already willing to accept those bytes at line rate.

Then there's the arithmetic that most teams meet late. The multipart interface that object stores expose caps an upload at 10,000 parts, requires every part except the last to be at least 5 MiB, and tops out at a 5 TiB object. Multiply it out: 5 MiB parts give you a 48.8 GiB ceiling, 8 MiB parts give you 78 GiB, 32 MiB parts give you 312 GiB. A client that hardcodes 5 MiB because a tutorial did will start rejecting your largest artifacts the first quarter someone trains a bigger model, and the error surfaces at part 10,001 — an hour into the upload, not at the start.

Part size is a capacity plan.

It also decides your request bill and your retry cost, in opposite directions. Ten thousand small parts means ten thousand signed PUTs, ten thousand billable requests, and a signing endpoint that has to answer quickly; large parts mean fewer requests but a more expensive retry, because a part that dies at 95% on hotel Wi-Fi is re-sent in full. Somewhere between 16 and 64 MiB is where most artifact workloads settle, and honestly the resolving evidence here is your own p99 client bandwidth, not anybody's blog post. Measure it before you commit the constant.

Should a startup upload large training files direct to object storage with presigned multipart parts?

For artifacts above a few hundred megabytes, yes, and the decision is not close. Below that it is genuinely arguable, and I'd compare the three realistic paths on operator cost rather than elegance.

Path Who moves the bytes Throughput ceiling Ops load Where it stops making sense
Proxy through your own API Your compute Your egress and pod count Lowest — one auth story Objects past ~100 MB, or deploys during business hours
Presigned multipart parts The browser, straight to the store The client's link Signing endpoint plus part accounting Single objects past part size × 10,000
Self-hosted resumable server (tus) Your compute, then the store Your fleet A stateful service you scale and patch When you did not want another tier to run

The proxy is not a joke option; it is the correct answer for a 2 MB avatar, and it keeps authorization in exactly one place. It stops being the answer the moment a single request can occupy a worker for twenty minutes.

tus deserves a fairer hearing than it usually gets in these comparisons. It is an open HTTP protocol for resumable uploads — a client HEADs the upload URL, learns the committed offset, and PATCHes from there — so resumption survives a closed laptop, and it does not care whether the backend speaks a multipart API at all. That last property matters if your storage target is something without S3 semantics. The catch is ownership: somebody runs that server, sizes it, patches it, and pays for the bytes crossing it, because even with an S3 storage backend the payload lands on your compute before it lands in the bucket. For a five-person team whose backend is otherwise stateless, that's a new pager rotation to justify.

Presigned parts move the opposite way. No bytes touch your infrastructure, so a deploy at 14:00 can't kill a 40 GB upload, and your throughput ceiling becomes the researcher's uplink. What you take on instead is bookkeeping: an upload ID, a part list, ETags coming back from the browser, and a completion call that has to be told the parts in ascending order or it will refuse the object. You also give up inline inspection — no scanning or transcoding the stream on the way through — so that work moves to an event handler after the object lands. If you need synchronous rejection of bad payloads, direct-to-store is not suitable and you should stick with the proxy.

What the signing code has to get right

The signer's job is authorization and arithmetic. It checks that this user may write this key, derives the part count from a declared size, refuses anything that would exceed the part ceiling, and hands back short-lived URLs. In Go, with the AWS SDK for the multipart API:

const ( partSize = 32 << 20 // 32 MiB — 312 GiB ceiling at 10,000 parts maxParts = 10_000 linkTTL = 30 * time.Minute ) type PartLink struct { PartNumber int32 `json:"partNumber"` URL string `json:"url"` } func signParts(ctx context.Context, cl *s3.Client, ps *s3.PresignClient, bucket, key string, size int64) (string, []PartLink, error) { n := (size + partSize - 1) / partSize if n > maxParts { return "", nil, fmt.Errorf("declared size %d needs %d parts; raise partSize", size, n) } created, err := cl.CreateMultipartUpload(ctx, &s3.CreateMultipartUploadInput{ Bucket: aws.String(bucket), Key: aws.String(key), }) if err != nil { return "", nil, err } links := make([]PartLink, 0, n) for i := int64(1); i <= n; i++ { pn := int32(i) req, err := ps.PresignUploadPart(ctx, &s3.UploadPartInput{ Bucket: aws.String(bucket), Key: aws.String(key), UploadId: created.UploadId, PartNumber: &pn, }, s3.WithPresignExpires(linkTTL)) if err != nil { return "", nil, err } links = append(links, PartLink{PartNumber: pn, URL: req.URL}) } return aws.ToString(created.UploadId), links, nil }

The browser side is smaller than people expect: slice the File with blob.slice(offset, offset+partSize), PUT each slice at its signed URL with four to six requests in flight, read the ETag off each response, and post the collected pairs back for completion. Nothing buffers the whole file, because a Blob slice is a view, not a copy.

One detail eats an afternoon if you miss it. A page script cannot read the ETag response header across origins unless the bucket says it may, so the CORS rule has to expose it explicitly:

[ { "AllowedOrigins": ["https://app.example.com"], "AllowedMethods": ["PUT", "GET", "HEAD"], "AllowedHeaders": ["content-type", "content-length"], "ExposeHeaders": ["ETag"], "MaxAgeSeconds": 3600 } ]

Without that line the parts upload perfectly and completion has nothing to submit, which reads like a client mystery and is really a CORS policy.

Retention you can reproduce, not remember

A retention policy is reproducible when a stranger can derive, from the repository alone, which object expires on which date. That rules out console clicks. It means the lifecycle rules live in your infrastructure code, the artifact's retention class lives in its key prefix — artifacts/retain-90d/<run-id>/model.safetensors — and CI applies the policy on merge, so the bucket's configuration is a build output rather than a decision somebody made once.

Two rules earn their keep. An expiration rule per retention prefix does the actual deleting, and a rule that aborts incomplete multipart uploads after a few days sweeps the parts of abandoned uploads — those parts are stored, and therefore billed, even though no object exists yet. Cheap, in this context, means the bill is dominated by bytes you meant to keep rather than by request counts and orphaned fragments.

Write a manifest object beside each artifact: declared size, part size, SHA-256, producer commit, retention class, completion timestamp. It costs one small PUT and turns "is this artifact the one the paper used" into a lookup instead of an argument.

Object lock is the other lever, and it is a one-way one. Objects under a legal hold cannot be deleted by anyone, including the person paying the bill, which is exactly the property you want for compliance and exactly the property that turns a mislabeled prefix into a multi-year storage commitment. Apply it narrowly.

Verify the object, measure the path, keep the rollback

Completion is not confirmation. Sort the parts, complete, then HEAD the object and compare its length against the manifest; if the store supports per-part checksums, ask for them and record the composite value.

func completeAndVerify(ctx context.Context, cl *s3.Client, bucket, key, uploadID string, parts []types.CompletedPart, want int64) error { sort.Slice(parts, func(a, b int) bool { return *parts[a].PartNumber < *parts[b].PartNumber }) _, err := cl.CompleteMultipartUpload(ctx, &s3.CompleteMultipartUploadInput{ Bucket: aws.String(bucket), Key: aws.String(key), UploadId: aws.String(uploadID), MultipartUpload: &types.CompletedMultipartUpload{Parts: parts}, }) if err != nil { _, abortErr := cl.AbortMultipartUpload(ctx, &s3.AbortMultipartUploadInput{ Bucket: aws.String(bucket), Key: aws.String(key), UploadId: aws.String(uploadID), }) return errors.Join(err, abortErr) } head, err := cl.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)}) if err != nil { return err } if got := aws.ToInt64(head.ContentLength); got != want { return fmt.Errorf("size mismatch: manifest %d, store %d", want, got) } return nil }

Two SLIs are enough to run this: upload completion rate bucketed by size class, and observed bytes per second per part at p50. The first tells you whether the largest class is quietly failing while the small-file average hides it; the second tells you whether the part size still fits the network your customers are actually on. Alert on the size class, review the throughput monthly, and keep the abort path on the error branch so orphaned parts never reach a lifecycle rule in the first place.

Keep the proxy endpoint behind a flag rather than deleting it. If signing breaks in a release, or a store's CORS behavior surprises you during a migration, per-file fallback to the old path is a config change instead of a hotfix — and for the small end of the size distribution it was arguably the right route anyway.

The honest summary: direct-to-store multipart is the cheaper and calmer design for large artifacts, it is overkill for small ones, and the number that decides both is the part size you commit to before the first upload.

Sources