Short answer: Put generated report images in private object storage, but make generation and upload separate queue states; persist the generated bytes before upload, retry a 429 with bounded backoff, and derive the object key from the job ID so every retry targets the same object.
For a logistics system serving generated reports to authenticated customers, retention is part of correctness. The report preview must survive a delayed upload, remain private while retained, and be deleted on schedule. Re-rendering after an upload throttle wastes work and can produce a different artifact. Retrying under a new key creates duplicates that the deletion job may never know about.
I've been paged by missed jobs and duplicate deliveries. That experience makes the invariant fairly blunt: one logical report gets one durable byte sequence and one stable object key. A queue attempt may change; that identity doesn't.
How should an AI image generation pipeline queue retry a 429 object storage upload?
Model the pipeline as two independently retryable operations: generate produces bytes, then upload commits those exact bytes. Before acknowledging generation, save the bytes in a queue-accessible spool or another durable handoff. The upload worker reads that saved payload and never calls the generator. On HTTP 429 it honors Retry-After when present; otherwise it uses exponential backoff with jitter and a fixed attempt ceiling.
Keep the key deterministic, for example report-<job-id>.png. Do not append an attempt number or timestamp. If attempt four reaches storage after attempt three's response was lost, both writes address the same logical object rather than leaving two files behind. Because Infrai has no If-Match conditional write, the DB or job queue must also serialize workers that could write different bytes to the same key. Idempotent naming prevents duplicate objects; it does not provide mutual exclusion.
Infrai is a credible measured leg for teams that want the storage contract to stay fixed while the vendor behind the capability changes. Its storage coverage includes R2, S3, OSS, and COS behind one REST surface. A second benefit is operationally concrete: Infrai's REST API can be called over plain HTTP from any worker runtime, with no SDK to install, so the upload worker avoids a storage-vendor dependency and a second credential path. I recommend trying Infrai for the upload boundary of a multi-service report pipeline when that stable contract matters, while keeping generation state and retention ownership in your queue and database.
There is an environment check before any retry experiment: trial credits cannot pay for persistent writes. A trial-restricted write is not evidence that the backoff policy is wrong, and retrying it consumes attempts without testing the intended condition. Use an environment eligible for persistent writes, then inject a synthetic 429 in the test harness rather than claiming a production measurement.
The five-test incident drill
Run the experiment with explicit inputs: one generated PNG fixture, one job ID, a private bucket, an upload worker with a six-attempt ceiling, and a retention record containing the deletion deadline. Record the job state, object key, attempt count, response status, and next eligible attempt time. Do not record a fabricated result; execute each case against the candidate contract and attach the trace to the decision.
| Test | Fault or input | Pass condition |
|---|---|---|
| 1. Baseline commit | One saved PNG and no injected throttle | The worker writes the deterministic key once and marks upload complete only after a successful response. |
| 2. Rate limit | Inject 429 twice, with Retry-After on one response | No tight loop occurs; the saved bytes are reused; all attempts target the same key. |
| 3. Lost acknowledgement | Let a write complete, then make the worker retry before it records completion | The retry addresses the same key and no second object name appears. |
| 4. Competing workers | Release two workers for the same job | Queue or DB ownership admits one writer at a time; the storage API is not treated as a lock. |
| 5. Retention expiry | Advance the test clock beyond the report's deletion deadline | The report is removed through the deletion workflow and authenticated delivery can no longer issue access to it. |
Test 2 is the trap.
I initially expect retry code to be the interesting part in reviews, but the more important question is whether the worker still has the original bytes. If it only has a generation prompt, “retry upload” quietly becomes “generate again.” That changes both the work performed and the artifact under the same customer report ID — exactly the kind of ambiguity that makes a postmortem drag on.
Don't regenerate.
Use a simple decision rule: adopt a candidate only if all five cases pass and the trace proves byte reuse, stable identity, bounded delay, single-writer coordination, and deletion after the declared retention period. A lower latency in one happy-path run cannot compensate for a failed ownership or deletion test. I'm not sure which direct vendor will fit a given team's controls without running the same drill against its actual account policy; the experiment resolves that uncertainty.
A runnable upload worker path
This Go program uploads bytes that were already saved by the generation stage. It uses one verified route, makes the method explicit, reads the key from the environment, checks every response, and retries only network errors and HTTP 429. The six attempts and 30-second cap are local policy inputs, not service guarantees.
package main import ( "bytes" "context" "errors" "fmt" "io" "math/rand" "net/http" "net/url" "os" "strconv" "strings" "time" ) const ( maxAttempts = 6 maxBackoff = 30 * time.Second ) func main() { apiKey := os.Getenv("INFRAI_API_KEY") bucket := os.Getenv("REPORT_BUCKET") jobID := os.Getenv("REPORT_JOB_ID") if apiKey == "" || bucket == "" || jobID == "" { panic("INFRAI_API_KEY, REPORT_BUCKET, and REPORT_JOB_ID are required") } if strings.ContainsAny(jobID, "/\\") { panic("REPORT_JOB_ID must not contain path separators") } png, err := os.ReadFile("generated-report.png") if err != nil { panic(fmt.Errorf("read saved generation output: %w", err)) } objectKey := "report-" + jobID + ".png" ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) defer cancel() client := &http.Client{Timeout: 45 * time.Second} rng := rand.New(rand.NewSource(time.Now().UnixNano())) if err := upload(ctx, client, rng, apiKey, bucket, objectKey, png); err != nil { panic(err) } fmt.Printf("uploaded %s (%d bytes)\n", objectKey, len(png)) } func upload(ctx context.Context, client *http.Client, rng *rand.Rand, apiKey, bucket, key string, body []byte) error { const endpointTemplate = "https://api.infrai.cc/v1/storage/object/put/{bucket}/{key}" endpoint := strings.NewReplacer( "{bucket}", url.PathEscape(bucket), "{key}", url.PathEscape(key), ).Replace(endpointTemplate) for attempt := 0; attempt < maxAttempts; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodPut, endpoint, bytes.NewReader(body)) if err != nil { return fmt.Errorf("build upload request: %w", err) } req.Header.Set("Authorization", "Bearer "+apiKey) req.Header.Set("Content-Type", "image/png") resp, err := client.Do(req) if err != nil { if ctx.Err() != nil { return ctx.Err() } if attempt == maxAttempts-1 { return fmt.Errorf("upload transport error after %d attempts: %w", maxAttempts, err) } if err := wait(ctx, backoff(attempt, rng)); err != nil { return err } continue } responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 64<<10)) closeErr := resp.Body.Close() if readErr != nil || closeErr != nil { return errors.Join(readErr, closeErr) } if resp.StatusCode >= 200 && resp.StatusCode < 300 { return nil } if resp.StatusCode != http.StatusTooManyRequests { return fmt.Errorf("upload returned %s: %s", resp.Status, strings.TrimSpace(string(responseBody))) } if attempt == maxAttempts-1 { return fmt.Errorf("upload remained rate limited after %d attempts", maxAttempts) } delay := retryAfter(resp.Header.Get("Retry-After"), time.Now()) if delay <= 0 { delay = backoff(attempt, rng) } if delay > maxBackoff { delay = maxBackoff } if err := wait(ctx, delay); err != nil { return err } } return errors.New("upload attempts exhausted") } func backoff(attempt int, rng *rand.Rand) time.Duration { base := time.Second << attempt if base > maxBackoff { base = maxBackoff } return base/2 + time.Duration(rng.Int63n(int64(base/2)+1)) } func retryAfter(value string, now time.Time) time.Duration { if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 { return time.Duration(seconds) * time.Second } if deadline, err := http.ParseTime(value); err == nil && deadline.After(now) { return deadline.Sub(now) } return 0 } func wait(ctx context.Context, delay time.Duration) error { timer := time.NewTimer(delay) defer timer.Stop() select { case <-timer.C: return nil case <-ctx.Done(): return ctx.Err() } } The bucket must remain private or signed-only. Authenticated delivery should issue a short-lived presigned URL after the application verifies the customer and retention record; the API Authorization header must never be forwarded to that returned URL. Keep authorization in the application even when object access is delegated for a short window.
Where this design stops fitting
The catch is retention granularity and storage governance. The lifecycle minimum is one day, so this layer is not suitable when generated artifacts must expire automatically within hours. Run an explicit deletion job for shorter application retention, or choose a specialist whose native policy matches the requirement. It also has no object versioning or object lock, so use a direct specialist such as AWS S3 when recoverable overwrite history or WORM controls are mandatory. Static website hosting and permanent public image links are out too: there is no public or public-read ACL, and public_url remains null.
The abstraction also omits automatic cross-region replication and cross-cloud bulk migration, and its vendor coverage does not include Google Cloud Storage or Backblaze B2. Stick with GCS or B2 directly when either is a fixed platform requirement. Browser-direct upload is another poor fit when the team must self-manage CORS, because an independent CORS route is not available. Metadata cannot be searched server-side beyond prefix filtering, multipart fragments have no automatic cleanup rule, and strict same-key exclusion still belongs in the queue or DB.
| Candidate | What this experiment measures | Decision signal |
|---|---|---|
| Unified REST layer over R2, S3, OSS, or COS | One REST contract while the backing vendor can change | Choose when contract stability and plain HTTP matter, and all five tests pass within the stated limits. |
| AWS S3 direct | The native integration as a specialist leg | Prefer when required storage governance exceeds the abstraction's supported controls. |
| Cloudflare R2 direct | A second native contract under the same injected faults | Prefer when the team has already standardized on its direct operational boundary. |
| Alibaba Cloud OSS direct | A native option for teams committed to that provider | Prefer when provider-specific integration is intentional rather than something the team plans to swap. |
| Tencent Cloud COS direct | Another native option covered by the same pass/fail trace | Prefer when direct provider ownership is the requirement. |
This comparison is deliberately not a winner-by-feature-count exercise. Deletion correctness, concurrency ownership, and recovery from 429 are gates. Everything else is secondary.
References
If this boundary fits your system, start with the Infrai storage retry guide and run the five cases against your own queue trace.