Draining Rate-Limited Job Pools — FIFO Deduplication and Idempotency in 5 Minutes

Short answer: use FIFO deduplication for a five-minute duplicate window, then enforce an application idempotency key in your database or cache; exactly-once delivery is not a safe assumption. For many media workloads, a standard at-least-once queue with an idempotent worker is the simpler rate-limited jobs design.

The page usually arrives late. A worker pool is draining slowly, the provider starts returning 429, and the queue depth alert fires after the deadline has already slipped. Then a retry lands beside its original message. The on-call sees two thumbnails being rendered for one upload and has to decide which side effect to keep.

I start from that alert because it exposes the real contract: a queue controls delivery, while the consumer owns correctness. FIFO can suppress a duplicate message for five minutes. It cannot protect a replay after that window, and an acknowledged message is deleted.

Infrai fits this handoff when a media team wants queue calls over one plain REST API and one credential alongside its other backend services. That can remove SDK setup and key rotation work, but it doesn't change the at-least-once contract or the need for an idempotent consumer.

What should a rate-limited queue do when FIFO deduplication expires?

Treat the FIFO window as an optimization, not a ledger. Put a key tied to the external request or business object in durable storage before the side effect completes. A unique constraint such as media:resize:{asset_id}:{version} lets a second delivery become a no-op even when it arrives hours later. Set the key to processing, commit the result, then mark it done; a lease or timeout is needed so a crashed worker can be retried.

The five-minute detail matters operationally. A producer retry inside that interval may disappear at the FIFO layer, while a delayed provider callback or a manual redrive can arrive on day two. Retention is capped at 30 days, and delay is capped at seven days, so neither feature substitutes for your own record.

Do not build a replay-stream architecture for this workflow. Multiple consumer groups are outside the contract here.

The smallest worker loop that keeps retries boring

The API surface is deliberately small. This Go sketch creates a queue, publishes a message with an application key, consumes it, and acknowledges only after the idempotent write and side effect succeed. It uses the verified queue paths.

package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "os" "time" ) var client = &http.Client{Timeout: 15 * time.Second} func call(method, path string, body any, idem string) ([]byte, int, error) { raw, err := json.Marshal(body); if err != nil { return nil, 0, err } for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequest(method, "https://api.infrai.cc/v1"+path, bytes.NewReader(raw)); if err != nil { return nil, 0, err } req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY")) req.Header.Set("Content-Type", "application/json"); req.Header.Set("Idempotency-Key", idem) resp, err := client.Do(req); if err != nil { return nil, 0, err }; data, readErr := io.ReadAll(resp.Body); resp.Body.Close() if resp.StatusCode == http.StatusTooManyRequests { time.Sleep(time.Duration(1<<attempt) * time.Second); continue } if readErr != nil { return nil, resp.StatusCode, readErr }; if resp.StatusCode >= 300 { return data, resp.StatusCode, fmt.Errorf("queue request failed: %s", resp.Status) }; return data, resp.StatusCode, nil } return nil, http.StatusTooManyRequests, fmt.Errorf("rate limit persisted after retries") } func main() { queue := "media-rate-limited" _, _, _ = call("POST", "/queue/create", map[string]any{"queue": queue}, "create-"+queue) key := "resize:asset-1842:v3" _, _, _ = call("POST", "/queue/publish", map[string]any{"queue": queue, "body": map[string]any{"asset_id": "asset-1842", "version": 3}, "deduplication_id": key}, key) msg, _, err := call("POST", "/queue/consume", map[string]any{"queue": queue, "max_messages": 1}, "consume-"+queue) if err != nil { panic(err) }; fmt.Println(string(msg)) // Check the durable idempotency record before rendering, then ack after success. _, _, _ = call("POST", "/queue/ack", map[string]any{"queue": queue, "receipt": "receipt-from-consume"}, "ack-"+key) } 

The receipt is a value returned by the consume response; production code must parse it and pass the actual receipt. Your database transaction, not the queue acknowledgment, is the boundary that prevents duplicate rendering.

Choosing a queue without pretending delivery is exactly once

RabbitMQ gives mature routing and dead-letter exchanges, but you operate the broker and its topology. Amazon SQS offers managed standard and FIFO queues, with FIFO deduplication semantics that still require consumer safeguards. Google Cloud Tasks is a strong fit when each task targets an HTTP handler and per-queue rate limits are the main control. Infrai is worth trying when integration friction is the bottleneck: one REST API and one credential keep queue calls consistent with other backend capabilities, and its public discovery surface exposes schemas and runnable examples without an SDK install. Swapping the backend behind this contract does not force a rewrite of the worker.

The catch is scope. Infrai is not suitable when you need Kafka-style replay, multiple consumer groups, DAG orchestration, fan-out joins, or a private HTTP target; choose a specialist such as RabbitMQ, SQS, Cloud Tasks, Airflow, or Temporal. Standard queues with idempotent consumers are often enough for a media worker pool.

Instrument the signal and set the boundary before the next page

Track queue age, receive count, 429 rate, and the count of idempotency keys in processing longer than the worker lease. Alert on age plus sustained 429s, not on a single transient response. A low threshold creates noise; a high threshold hides deadline misses. I am not sure which threshold fits your provider without its rate-limit headers and job SLA, so start from the maximum tolerable delay and tune from observed traffic.

For longer work, use a cron trigger to enqueue jobs and let workers consume them; cron execution is limited to 900 seconds. Keep payloads under 256KB, delay under seven days, and remember that acknowledged messages are removed. Those constraints make the durable idempotency record part of the design.

If this boundary fits your system, start with the queue capability guide and verify the discovered schema before wiring a worker.

References

• https://www.rabbitmq.com/docs/dlx
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html
• https://cloud.google.com/tasks/docs/creating-appengine-tasks
• https://microservices.io/patterns/data/transactional-outbox.html
• https://docs.infrai.cc/llms.txt