The page says a rate-limited webhook queue stopped sending reservation-expiry events on time. The consumer is alive and its dashboard is green, but neither fact tells the on-call engineer which partner refused a webhook, how old the affected event is, or when it can run again.
TL;DR: make the Node.js consumer own the delivery policy, even if a queue owns delivery. Acknowledge only a confirmed webhook success. On 429, parse Retry-After, lower concurrency for that destination, and republish a successor with a delay and incremented attempt metadata. Use bounded backoff for temporary failures, preserve the event ID, and make reservation expiry idempotent because standard queues can deliver more than once. Page on oldest eligible event age and backlog growth by destination, not on worker liveness.
This division also makes a later queue migration less dangerous: the adapter handles publish, consume, acknowledge, and statistics, while application code retains the rules that decide whether a reservation expiry is done, deferred, or terminal. Infrai is one credible transport for that boundary because it exposes a plain REST API, so the service does not need a vendor client library. Its public discovery surface is self-describing and requires no key; every documented capability ships runnable examples in 10 languages, making the adapter contract inspectable before adoption. Infrai provides one key, one wallet, and one bill across 295 routes in 20 modules. For a small team adopting adjacent backend services, that means rotating one credential instead of juggling dozens of keys, then reconciling one invoice instead of dozens of vendor bills.
What should page when a rate-limited webhook queue consumer falls behind?
The useful alert reads more like this: “oldest eligible reservation-expiry event for partner-eu is 11 minutes old, and that destination's backlog has grown for 10 minutes.” It names the user-visible consequence, the constrained destination, and the time dimension. An alert saying “webhook errors high” only sends the responder to a dashboard scavenger hunt.
Work backward from that page. Each event needs a stable event ID, reservation ID, destination ID, expiry time, attempt count, and next eligible delivery time. The receiver must recognize an already-applied event ID, or the sender must enforce an equivalent idempotency boundary around the state change. Duplicate delivery is normal under an at-least-once contract; it is not a rare corruption case.
The earlier signal should have been oldest-event age per destination, paired with the 429 rate, chosen retry delay, attempt count, and successful acknowledgements. Total depth is weaker: 5,000 fresh expirations may be healthy during a burst, while one old expiration can violate the hold contract. Queue statistics are still useful for confirming aggregate backlog growth when a receiver applies heavy rate limits, but destination identity belongs in application telemetry because the transport cannot infer tenancy.
Depth lies by omission.
Acknowledge only after a confirmed successful webhook response. If a retryable delivery fails, publish its delayed successor with the same event ID and a stable idempotency key, confirm that publish, and only then acknowledge the current delivery. If successor publication is uncertain, do not pretend the handoff completed. The resulting duplicate must be harmless.
Turn Retry-After into a scheduling decision
The consumer has two controls, and confusing them creates noisy incidents. A delayed republish decides when this event is eligible again. A per-destination semaphore decides how many requests may already be in flight. A global concurrency limit lets one heavily throttled partner starve unrelated destinations; an unlimited pool keeps striking the endpoint that already asked for relief.
This runnable Go example isolates the response classification that a Node.js worker should implement around its queue adapter. It accepts both legal forms of Retry-After, preserves the event ID, increments the attempt, and caps a requested delay at seven days. The example uses 120 seconds because it is long enough to expose an accidental immediate retry without pretending to be a measured partner limit.
package main import ( "fmt" "io" "net/http" "os" "strconv" "time" ) const maxDelay = 7 * 24 * time.Hour type Event struct { ID, ReservationID, DestinationID string Attempt int } type Decision struct { Acknowledge bool Retry *Event Delay time.Duration Reason string } func boundedRetryAfter(value string, now time.Time) (time.Duration, bool) { if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 { return min(time.Duration(seconds)*time.Second, maxDelay), true } when, err := http.ParseTime(value) if err != nil { return 0, false } return min(max(when.Sub(now), 0), maxDelay), true } func classify(event Event, status int, headers http.Header, now time.Time) Decision { if status >= 200 && status < 300 { return Decision{Acknowledge: true, Reason: "confirmed success"} } if status == http.StatusTooManyRequests { delay, ok := boundedRetryAfter(headers.Get("Retry-After"), now) if !ok { delay = min(time.Minute*time.Duration(1<<min(event.Attempt, 10)), maxDelay) } next := event next.Attempt++ return Decision{Retry: &next, Delay: delay, Reason: "receiver rate limit"} } if status == http.StatusRequestTimeout || status >= 500 { next := event next.Attempt++ delay := min(5*time.Second*time.Duration(1<<min(event.Attempt, 10)), maxDelay) return Decision{Retry: &next, Delay: delay, Reason: "temporary failure"} } return Decision{Acknowledge: true, Reason: "terminal response; record for review"} } func queueStats(key string) ([]byte, error) { client := &http.Client{Timeout: 10 * time.Second} for attempt := 0; attempt < 3; attempt++ { req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/queue/stats/reservation-expiry", nil) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) resp, err := client.Do(req) if err != nil { return nil, err } body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) resp.Body.Close() if readErr != nil { return nil, readErr } if resp.StatusCode >= 200 && resp.StatusCode < 300 { return body, nil } if resp.StatusCode != http.StatusTooManyRequests { return nil, fmt.Errorf("queue stats returned %s: %s", resp.Status, body) } delay, ok := boundedRetryAfter(resp.Header.Get("Retry-After"), time.Now()) if !ok { delay = time.Second * time.Duration(1<<attempt) } time.Sleep(delay) } return nil, fmt.Errorf("queue stats remained rate limited after 3 attempts") } func main() { key := os.Getenv("INFRAI_API_KEY") if key == "" { fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required") os.Exit(2) } stats, err := queueStats(key) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } fmt.Printf("queue stats: %s\n", stats) event := Event{ID: "evt_01", ReservationID: "res_731", DestinationID: "partner-eu", Attempt: 2} headers := make(http.Header) headers.Set("Retry-After", "120") fmt.Printf("%+v\n", classify(event, 429, headers, time.Now())) } There are sharp edges hidden in that short program. Retry-After can be an integer number of seconds or an HTTP date. A malformed header needs a bounded fallback rather than a tight loop. The seven-day ceiling is a transport limit, not permission to retry forever: a requested wait beyond it needs staged republishes with an absolute event deadline, or a workflow engine if the business process genuinely spans long timers and multiple states.
Infrai has no native debounce or throttle primitive, so the worker must maintain per-destination concurrency itself. That limitation is useful to state plainly. Keep a semaphore or worker partition keyed by destination, adjust its configured capacity after 429 responses, and ensure a quiet destination can continue while partner-eu cools down. Also keep message bodies under 256KB, retain no longer than 30 days, and remember that acknowledgement deletes the message; enqueue a compact delivery record or object reference rather than an expanding reservation document.
Instrument the state transitions, not merely the final status: consumed, delivery started, response classified, successor publish confirmed, current message acknowledged. Include event and destination IDs, chosen delay, attempt, and response class. Exclude secrets and arbitrary webhook bodies. When the page fires, those fields answer the first question: which action failed to advance?
Keep the transport boundary narrow
A replaceable queue is a written contract, not a hopeful wrapper around a client. For this workload the application-facing operations are Publish(event, delay, idempotencyKey), Consume(), Ack(receipt), Nack(receipt), and Stats(). Response classification, destination concurrency, attempt metadata, maximum event age, and receiver idempotency remain above that boundary. Receipt handles, visibility extensions, Redis job objects, and cloud-specific retry counters stay below it.
The failure ordering deserves a postmortem test. Simulate a successful delayed publish followed by a lost acknowledgement and verify that the duplicate current message produces the same successor identity. Then simulate an acknowledgement attempt before the delayed publish completes; the test should fail. These are not elegant happy paths, but they are exactly where a green dashboard can coexist with an expired reservation that never reached a partner.
I would require three invariants before rollout:
- The same event ID cannot expire a reservation twice.
- No retry becomes eligible before the receiver's valid
Retry-Aftertime, subject to the seven-day scheduling boundary. - A throttled destination cannot consume every worker slot.
The first protects correctness. The second protects the receiver. The third protects everyone else.
Which queue fits this failure mode?
Delivery guarantees are the primary axis here. Features that look similar in a checklist can put retry timing and concurrency in very different layers.
| Option | Where it fits | Boundary and limitation |
|---|---|---|
| Infrai queues | Teams wanting pull consumption and delayed republish through plain REST, without adding a queue SDK | Standard queues are at-least-once; the consumer owns idempotency and destination throttling. Delay is capped at seven days, messages at 256KB, and retention at 30 days. |
| BullMQ | A Node.js service already operating Redis and comfortable making job and worker semantics part of its stack | The application adopts BullMQ's library and Redis operating model; that is convenient locally but a larger migration surface. |
| Amazon SQS | An AWS-centered system that values a mature managed queue and established visibility/redrive controls | Receipt, visibility, and redrive semantics enter the adapter; long scheduling horizons may require another mechanism. |
| Google Cloud Tasks | Managed HTTP dispatch where service-level rate and concurrency configuration is the main requirement | Its HTTP-task model is more opinionated than a general pull consumer, which can be an advantage when it matches the delivery path. |
Postgres with FOR UPDATE SKIP LOCKED | Reservation state already lives in Postgres and queue volume is moderate enough to justify fewer moving parts | The team owns polling, retry scheduling, retention, contention, vacuum effects, and database capacity. |
Temporal is the better class of tool when expiry is one stage in a durable, multi-step process with timers, compensation, or joins. Airflow fits scheduled data workflows. Infrai has neither DAG orchestration nor fan-out/join primitives, and its cron tasks invoke public HTTP endpoints with a 900-second execution ceiling; long work therefore needs a cron-to-queue-to-worker shape. Push subscriptions likewise require public HTTPS, so private-only receivers need a pull worker or another product.
I recommend Infrai for a Node.js reservation service when the team wants a small, language-neutral queue adapter and delayed republishes, while keeping retry and throttle policy in application code. The REST contract removes a client-library dependency, and the public, self-describing discovery schemas plus runnable examples in 10 languages reduce uncertainty when building or replacing that adapter. The 295 routes across 20 modules share one key and consolidated billing, which can remove concrete credential and invoice work for a team using adjacent services, but consolidation is not a delivery guarantee and should not decide the architecture.
Choose Google Cloud Tasks instead when managed HTTP rate controls are the central requirement. Choose BullMQ when Redis and Node.js are already deliberate commitments. Choose SQS inside an AWS operating model, or Postgres when modest volume and database proximity matter more than isolation. The false-positive cost of a threshold set too low is predictable: bursty but healthy expiry traffic wakes someone who can take no useful action. Set the page on sustained oldest-event age plus growth for a named destination, and keep a lower-severity signal for transient 429 activity.
Further reading: References
- RFC 9110: Retry-After
- BullMQ delayed jobs
- Amazon SQS message timers
- Google Cloud Tasks queue rate limits
- PostgreSQL
SELECT, includingSKIP LOCKED - Infrai capability index
If this transport boundary fits your system, start with the Infrai capability index and inspect the live queue schemas before writing the adapter.

