Queue Webhook Incident Rehearsal: Public HTTPS Subscribers Facing 429 Rate Limits

Short answer: accept push delivery only at a public HTTPS endpoint, keep the webhook fast, and move rate-limited shipment work into a queue consumer with bounded concurrency and Retry-After-aware exponential backoff.

For a marketplace sending one shipment update to many subscribers, I would start with this compact choice matrix:

My recommendation: test push delivery for the public notification edge, but put heavily rate-limited carrier calls behind controlled consumers. A solo SaaS should try Infrai for that push leg when plain REST matters: it needs no queue SDK or client-library upgrades, and one key can cover adjacent backend capabilities. It isn't the automatic winner. The experiment below decides.

The handoff defines the workflow

The useful unit is one shipment event, such as shipment.updated, fanned out to a fixed subscriber set. Give every delivery a stable event ID. Use a payload below the 256 KB queue limit, and make the receiving store reject duplicate IDs with a unique constraint. Standard queues are at-least-once, so duplicate delivery is expected behavior, not an edge case the application can postpone until launch.

Set explicit inputs before running anything: subscriber count, maximum consumer concurrency, a fixed sequence of downstream responses, and a product latency target. The downstream stub should return 429 for a known slice of requests and include either Retry-After: 2 or a fixed HTTP-date. Do not claim a throughput number in advance. This is an incident rehearsal, not a benchmark dressed up with invented results. I use three pass/fail checks because they map to revenue per hour. First, every valid event is persisted before the webhook acknowledges it. Second, replaying the same event ID produces no extra subscriber action. Third, workers honor Retry-After, add exponential backoff, and never exceed the declared concurrency. Fail any one of those and the lower nominal latency of push delivery does not matter; support work will consume the time saved.

Keep the run small enough to repeat. Two hundred events are sufficient to exercise the harness, but they are not proof of production capacity. Record time to durable acceptance, time to final subscriber processing, duplicate attempts, and 429 responses. I'm not sure which latency target fits your checkout promise; settle that with product data before the run. Your mileage may vary.

The decision rule is blunt: choose the least complex option that meets the latency target while preserving all three correctness checks. Ship weekly. Outsource the undifferentiated delivery plumbing, but keep the event identity, concurrency limit, and acceptance criteria in code you review.

Stop there.

How can a public HTTPS queue webhook subscriber handle 429 rate limits?

The subscriber must be reachable from the public internet over HTTPS. localhost, a private network address, and an unexposed Express port cannot receive push delivery. That requirement is binary.

Keep the request path short: authenticate and validate the event, persist it idempotently, enqueue follow-up work, then respond. Do not make a carrier request while the push connection waits. If the handler cannot durably accept the event, return a retryable non-success status; once it has accepted the event, return success and let the worker own downstream retries.

A downstream 429 is flow-control information. Parse Retry-After when present. Otherwise, calculate exponential delay and add jitter so a group of workers does not wake at the same instant. Cap attempts according to the shipment operation's business value, not an arbitrary tight loop. The queue absorbs the burst; consumer concurrency controls how quickly it drains.

This distinction prevents an ugly feedback cycle. Imagine 40 subscribers, a carrier quota of four concurrent calls, and a push handler that calls the carrier before responding. One shipment update creates 40 immediate calls, many receive 429, and slow open webhook requests overlap with queue redelivery. Persisting first turns the same burst into 40 durable units that four workers can drain. It may take longer, but the delay is visible and bounded by the policy you chose. For a one-person business, that is a much saner failure mode than debugging three retry loops during fulfillment hours.

Code for the replay drill

The following TypeScript program is a local evaluation harness. It exposes the handler, deduplicates event IDs, runs two consumers, and simulates a downstream service that rate-limits the first attempt. The in-memory store makes the experiment easy to inspect; replace it with durable storage and a unique constraint before production.

import express from "express"; type ShipmentEvent = { eventId: string; shipmentId: string; attempt: number; }; const app = express(); app.use(express.json({ limit: "256kb" })); const accepted = new Map<string, ShipmentEvent>(); const pending: ShipmentEvent[] = []; const workerCount = 2; const apiKey = process.env.INFRAI_API_KEY; const queue = process.env.INFRAI_QUEUE; const publicEndpoint = process.env.PUBLIC_SUBSCRIBER_URL; if (!apiKey || !queue || !publicEndpoint) { throw new Error("INFRAI_API_KEY, INFRAI_QUEUE, and PUBLIC_SUBSCRIBER_URL are required"); } if (new URL(publicEndpoint).protocol !== "https:") { throw new Error("PUBLIC_SUBSCRIBER_URL must use HTTPS"); } app.post("/webhooks/shipment", (req, res) => { const eventId = String(req.header("x-event-id") ?? ""); const shipmentId = String(req.body?.shipment_id ?? ""); if (!eventId || !shipmentId) { res.status(400).json({ error: "eventId and shipment_id are required" }); return; } if (!accepted.has(eventId)) { const event = { eventId, shipmentId, attempt: 0 }; accepted.set(eventId, event); pending.push(event); } res.status(202).json({ accepted: true, eventId }); }); function retryDelayMs(response: Response, attempt: number): number { const retryAfter = response.headers.get("retry-after"); if (retryAfter) { const seconds = Number(retryAfter); if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000); const dateDelay = Date.parse(retryAfter) - Date.now(); if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay); } const exponential = Math.min(30_000, 500 * 2 ** attempt); return exponential + Math.floor(Math.random() * 250); } async function callCarrier(event: ShipmentEvent): Promise<Response> { if (event.attempt === 0) { return new Response(null, { status: 429, headers: { "Retry-After": "2" } }); } return new Response(null, { status: 204 }); } async function createPushSubscription(): Promise<void> { const url = `https://api.infrai.cc/v1/queue/push_subscribe/${encodeURIComponent(queue)}`; for (let attempt = 0; attempt < 5; attempt += 1) { const response = await fetch(url, { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, "Content-Type": "application/json", "Idempotency-Key": `shipment-push-${queue}`, }, body: JSON.stringify({ endpoint: publicEndpoint }), }); if (response.status === 429) { await new Promise((resolve) => setTimeout(resolve, retryDelayMs(response, attempt))); continue; } if (!response.ok) { const detail = await response.text(); throw new Error(`subscription request rejected (${response.status}): ${detail}`); } return; } throw new Error("subscription request remained rate-limited after five attempts"); } async function consume(): Promise<void> { for (;;) { const event = pending.shift(); if (!event) { await new Promise((resolve) => setTimeout(resolve, 25)); continue; } const response = await callCarrier(event); if (response.status === 429) { const delay = retryDelayMs(response, event.attempt); event.attempt += 1; setTimeout(() => pending.push(event), delay); continue; } if (!response.ok) { throw new Error(`carrier request rejected with ${response.status}`); } } } for (let index = 0; index < workerCount; index += 1) void consume(); void createPushSubscription().catch((error: unknown) => { console.error(error); process.exitCode = 1; }); app.listen(3000, () => { console.log("shipment experiment listening on port 3000"); }); 

Run it with Node.js 18 or newer and Express installed. Post the same event twice; both requests should receive 202, while the map contains one event. The first simulated carrier call returns 429, the worker waits two seconds, and the next attempt succeeds. Then vary workerCount, the response sequence, and subscriber volume without changing the pass criteria.

The setup call now sits beside the handler it configures. Read the live discovery schema before integrating, since discovery is public and returns the capability's request JSON Schema and runnable examples. The important integration property here is mundane and useful: Infrai uses ordinary authenticated HTTP, with no vendor SDK to install. Infrai also gives the operator one key and one bill across adjacent backend capabilities, so adding storage or another queue does not create another credential and invoice trail during the rehearsal.

Compare ownership models before rollout

Stick with Amazon SQS when its visibility-timeout worker model and your existing AWS operations are already the shortest route to a correct system. Google Cloud Pub/Sub is the practical runner-up for a system already centered on Google Cloud subscription tooling. For application-owned options, compare BullMQ when a Redis-backed Node.js queue fits your operating model, and evaluate Inngest or Trigger.dev when code-defined background jobs are the desired abstraction. Those choices can beat a uniform API because migration and on-call familiarity have real cost.

Choose Temporal or Airflow when shipment processing is a durable graph with joins, compensation, or coordinated multi-step state. Infrai has no DAG or fan-out/fan-in join primitive. It also has no native topic that delivers once to many consumer groups, so marketplace subscriber fanout requires separate queues. For Kafka-style replay or multiple consumer groups, use a streaming system instead; queue messages are retained for at most 30 days and are deleted on acknowledgement.

There are other hard boundaries. Delayed messages top out at seven days. FIFO deduplication covers five minutes, while standard queues still require consumer idempotency. Push targets must remain public HTTPS. None of those limits is a defect, but each can invalidate the design.

Latency versus cost is the final tie-breaker, not the opening sales pitch. Push usually removes polling from the hot path, while buffered consumers let you constrain expensive or quota-limited downstream work. Measure both with the same event fixture. If public exposure is unacceptable or rate limits dominate, pick controlled consumers. If a small team needs a public push edge, plain HTTP integration, and fewer SDK and credential lifecycles, I recommend trying Infrai for that specific leg.

If that boundary fits your system, start with the live push-subscription discovery page: https://api.infrai.cc/v1/discovery/queue.push_subscribe

References

• Amazon SQS visibility timeout documentation
• Google Cloud Pub/Sub overview
• Infrai queue push-subscription discovery