A delayed webhook worker for game events should acknowledge the queue message only after the delivery attempt has a durable outcome. The deciding constraint is delivery guarantee: a timeout must produce a retryable state, not a lost cleanup job or an unbounded retry storm.

Short answer: separate the queue lease from the webhook request timeout, persist an attempt record before making the request, and make the receiver idempotent. A five-second HTTP timeout can be useful, but it is not a delivery guarantee. It only bounds one attempt.

That distinction matters.

For a periodic cleanup job, I use three explicit outcomes: delivered, retryable, and terminal. The worker owns the transition between them. That small state machine is more dependable than treating a queue timeout as proof that the remote game service did or did not receive the request.

How do message queue timeout issues affect delayed webhook delivery?

There are two different clocks. The queue lease controls how long one consumer may work before another consumer can receive the message. The HTTP timeout controls how long that consumer waits for a response. They can expire in either order.

Suppose a cleanup message targets expired match-room records. The worker sends the webhook, the remote endpoint commits the request, and the response is delayed. The HTTP client reports a timeout. If the worker immediately requeues the message, the same cleanup may run twice. If it marks the message complete, a timeout caused by a dropped response can hide a real delivery failure. Neither choice can infer the remote side's state from the local timer alone.

That is the first trap. A timeout is an observation about waiting, not an observation about side effects. The receiver needs an idempotency key such as cleanup:<roomId>:<periodStart>, and it must record that key before applying the cleanup again. The key belongs to the logical job, not to an individual retry.

The small state machine I would ship

Keep it explicit.

The queue payload should contain a stable job identifier, the game scope, and the scheduled period. It should not be the only record of progress. A database row or equivalent durable store can keep the current state, attempt count, next eligible time, and last error category.

The worker flow is deliberately boring:

  1. Claim the job under the queue's lease.
  2. Persist an attempt-start record with the same idempotency key used for the webhook.
  3. Send the request with a bounded timeout.
  4. Classify the result before acknowledging the queue message.
  5. Acknowledge only after the durable state says delivered or terminal.

Here is the boundary I want in a Node.js worker. The queue adapter is intentionally generic; the important contract is the classification, not a vendor-specific API.

type DeliveryResult = | { kind: 'delivered'; status: number } | { kind: 'retryable'; reason: string } | { kind: 'terminal'; reason: string }; async function deliverCleanup( job: { id: string; roomId: string; periodStart: string }, send: (request: { idempotencyKey: string; body: unknown; signal: AbortSignal }) => Promise<{ status: number }>, persist: (state: { jobId: string; state: string; detail: string }) => Promise<void> ): Promise<DeliveryResult> { const idempotencyKey = `cleanup:${job.roomId}:${job.periodStart}`; const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), 5000); await persist({ jobId: job.id, state: 'attempting', detail: idempotencyKey }); try { const response = await send({ idempotencyKey, body: { roomId: job.roomId, periodStart: job.periodStart }, signal: controller.signal }); if (response.status >= 200 && response.status < 300) { await persist({ jobId: job.id, state: 'delivered', detail: String(response.status) }); return { kind: 'delivered', status: response.status }; } if (response.status === 408 || response.status === 429 || response.status >= 500) { return { kind: 'retryable', reason: `remote status ${response.status}` }; } await persist({ jobId: job.id, state: 'terminal', detail: String(response.status) }); return { kind: 'terminal', reason: `remote status ${response.status}` }; } catch (error) { const reason = error instanceof Error ? error.message : 'unknown delivery error'; return { kind: 'retryable', reason }; } finally { clearTimeout(timeout); } }

The caller still needs to persist a retryable result and release or extend the lease according to the queue contract. That choice matters. A retry scheduled for 30 seconds later is not helpful if the original lease remains held for 10 minutes, and a lease that expires during a live request invites concurrent workers.

A delayed request needs integrity protection as well as retry handling. HMAC authenticates a message with a shared secret and a cryptographic hash; RFC 2104 defines the construction using the inner and outer padding values. Sign the exact bytes the receiver will verify, and include the stable idempotency key and an expiration value in the signed representation.

The receiver should reject an expired timestamp and verify the signature before performing cleanup. It should then atomically check the idempotency key. A valid signature does not make a request safe to repeat by itself. Authentication and deduplication solve different failure modes. The trade-off is extra durable state and a little more work per delivery, but that cost buys an explainable retry boundary.

Keep the signing input explicit. JSON serialization can change if field order or whitespace changes, so the sender and receiver need one canonical representation. The key must stay outside the job payload and logs.

How do you test the timeout boundary?

A happy-path test proves very little here. I test the transition after each meaningful boundary: before the request leaves the process, after the receiver commits but before the response arrives, after the HTTP timeout, and after the queue lease expires. The useful assertion is that repeated delivery with the same key produces one cleanup effect.

I also force a slow receiver, a dropped response, a 429 response, a 500 response, malformed JSON, and a process stop during the attempt. Those cases should produce an intentional state, not a generic catch block that hides whether retrying is safe.

Metrics should separate queue age, lease extensions, request duration, retry count, terminal failures, and duplicate deliveries. One aggregate “webhook failed” counter cannot explain a timeout problem. Alert on jobs approaching their deadline and on retry age, because request latency alone misses work that is quietly cycling.

The decision rule

Use at-least-once delivery when the receiver can deduplicate by a stable key and the cleanup is safe to repeat. Use a terminal state for validation errors or other responses that retrying cannot repair. For an operation that cannot tolerate duplicates, the receiver needs an atomic idempotency record or a transactional handoff; increasing the client timeout does not create exactly-once behavior.

Before copying the five-second example, measure the receiver's normal and tail latency, the queue lease duration, the cleanup deadline, and the time needed to persist state. The timeout should leave room for durable bookkeeping and lease renewal. The number is a starting constraint, not a promise. The limitation is fundamental: a client timeout cannot prove whether the receiver committed the side effect.

This is the part I would review before choosing a queue implementation: can it make leases visible, schedule a retry without losing the job, and expose enough timing data to explain a duplicate? If those answers are unclear, the queue's feature list is less useful than a small, explicit state machine.

References