TL;DR: Azure Service Bus and AWS SQS retry logic can recover failed property-management webhooks, but neither queue can prevent a duplicate business effect. Choose the acknowledgement and redelivery semantics you can operate, then put idempotency at the receiving boundary. The sender records an immutable attempt, the receiver deduplicates a stable event ID, and a retry worker changes only delivery state. This is the operational recovery decision; queue brand comes later.
A lease-renewal event is a good example. A worker sends lease.updated to a partner that updates a resident portal or starts a compliance workflow. The HTTP request can time out after the partner commits the change. Retrying the request is correct only if the partner sees the same idempotency key and treats it as the same event. A queue cannot infer that business fact for us.
The queue is not the idempotency boundary.
What must remain true after a worker crashes?
I start the design with invariants, because retry policy expressed as a number of attempts is incomplete. For each event, the system needs one stable event ID, one immutable payload or payload digest, and a delivery record that can explain every attempt. The business effect may happen once even though transport attempts happen many times.
The boundaries matter:
- The producer transaction creates the event and its outbox row together.
- A dispatcher claims an outbox row for a bounded lease, sends the event, and records the outcome.
- The receiver atomically checks the event ID before applying a side effect.
- A timeout means unknown outcome, so it is retried rather than guessed to be a failure.
- A poison event is quarantined after a policy-defined limit, with enough context for replay.
This is exactly-once thinking without claiming exactly-once delivery. The audit trail says what the system attempted; the receiver's idempotency record says what business effect it accepted. Those are different facts and should not be collapsed into a single sent=true column.
For payment and ledger systems I use the same distinction for reconciliation: a transport acknowledgement is evidence about a message, not proof that a downstream business transaction is absent or present. Property workflows deserve the same discipline when a duplicate notification can trigger a fee, a notice, or a compliance task.
How should Azure Service Bus and AWS SQS retry failed webhooks?
The practical comparison is about failure ownership. A queue service can retain an unacknowledged message and redeliver it, while the application still owns the delay schedule, idempotency key, poison-message decision, and reconciliation view. A managed broker may expose dead-lettering and delivery counts; a simpler queue may expose visibility timeout and leave more policy to the worker. Neither removes the need for a durable application record.
| Decision point | Queue-managed redelivery | Application-scheduled retry |
|---|---|---|
| Crash after send, before acknowledgement | Message becomes visible again after its lease or visibility interval | Retry row remains pending and is claimed again |
| Backoff precision | Bounded by the queue's delay and lease controls | Explicit next-attempt timestamp |
| Duplicate protection | Not provided by transport semantics | Enforced with event ID at the receiver |
| Operational recovery | Inspect delivery count and dead-letter state | Inspect attempt history and replay a selected event |
| Main risk | Lease expires while a slow handler is still running | Scheduler or retry table becomes a new failure surface |
For AWS SQS, the visibility timeout is the interval during which a received message is hidden from other consumers; it is not a successful-delivery marker. Extending that interval while work continues can reduce concurrent redelivery, but the receiver still needs an idempotency check. The same mental model applies when a broker offers lock renewal under a different name.
Azure Service Bus and SQS therefore should not be compared as if one retry setting answers the whole design. Their controls differ, and their service limits and failure semantics must be verified against the current documentation before production configuration. The portable contract is narrower: stable IDs, explicit acknowledgement timing, bounded attempts, durable history, and a replay path.
Neither option is universally suitable. The limitation of queue-managed redelivery is clear when the team cannot inspect lock or visibility expiry; the trade-off is that an application retry table becomes a poor fit when nobody owns its scheduler, retention, and replay permissions. A small property manager with no operational coverage may prefer fewer moving parts; a regulated portfolio may accept the extra records because an auditable replay decision matters more than a short implementation.
Here is the critical path in Go. It deliberately acknowledges only after the outbound call returns, and it records an ambiguous timeout as retryable. The receiver-side key is the event ID, not a newly generated request ID.
type Delivery struct { \tEventID string \tAttempt int \tNextAfter time.Time \tPayload []byte } func deliver(ctx context.Context, d Delivery, send func(context.Context, string, []byte, string) error, mark func(string, int, string) error) error { \tkey := d.EventID \terr := send(ctx, "https://partner.example/events", d.Payload, key) \tif err == nil { \t\treturn mark(d.EventID, d.Attempt, "acknowledged") \t} \t// A timeout has an unknown remote outcome; preserve the event for retry. \tif errors.Is(err, context.DeadlineExceeded) || temporary(err) { \t\treturn mark(d.EventID, d.Attempt, "retryable") \t} \treturn mark(d.EventID, d.Attempt, "quarantined") } The snippet omits storage details on purpose. In production, mark must be conditional so a late worker cannot overwrite a newer state, and an attempt row should include timestamps, response class, and a redacted error category. Payloads may contain tenant or resident data, so auditability has to coexist with retention and access controls; keeping an unbounded copy of every body is not a compliance strategy.
Why does a retry still duplicate a webhook?
Because the failure occurs between two systems. Suppose the partner commits the update at 11:02:14.120, then the connection dies before the sender receives the response. The sender sees a failure and sends the same event at 11:02:19. A receiver that performs the side effect first and deduplicates later has already lost the race.
The receiving transaction should create an idempotency record with a unique constraint on (tenant_id, event_id), apply the business change only for a newly inserted record, and return the stored result for a repeated event. If the side effect is external to that transaction, use a receiver outbox as well. Do not use a timestamp, delivery attempt, or queue receipt handle as the deduplication key; each can change during recovery.
This is the part that tends to surprise people.
A useful test matrix includes acknowledgement loss, worker termination after the HTTP write, visibility expiry during a slow handler, duplicate delivery on two workers, malformed payloads, and replay after quarantine. I would also inject a response delay of 2 seconds and a client deadline of 1 second. The expected result is one business effect and two auditable delivery attempts, not a magically absent timeout.
Which option should the architecture record reject?
The rejected option is “let the queue retry the HTTP request and trust the delivery count.” It is attractive because it has little application code. It fails to distinguish a partner-side commit from a network failure, and delivery count cannot prove whether the business effect happened. It is valid for handlers whose operation is intrinsically idempotent, such as writing a content-addressed blob or refreshing a cache entry, provided the loss and replay policy is explicit. It is not sufficient for a lease notice, charge request, or compliance task.
A second tempting design is a new scheduled job for every retry. That can be useful when retry dates are part of a business workflow, but it creates a fan-out and cancellation problem for ordinary transport recovery. One durable event with append-only attempts is easier to reconcile. The queue then becomes a worker wake-up mechanism rather than the system of record.
Operationally, I would page on age of the oldest retryable event, quarantine growth, and repeated ambiguity for one tenant, not on every transient timeout. Dashboards should correlate event ID, tenant ID, attempt number, and receiver response class while avoiding sensitive payloads. Recovery is complete only when the event is acknowledged or deliberately quarantined with an operator-visible reason.
The decision rule is therefore modest: select the transport whose acknowledgement, delay, retention, and dead-letter behavior fit the recovery workflow your team can inspect; implement the invariant outside that transport. For the property-management webhook, the durable contract and receiver idempotency boundary matter more than whether redelivery is called visibility timeout or lock renewal.
References
- AWS, “Amazon SQS visibility timeout,” https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- AWS, “Amazon SQS message quotas,” https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html
- Microsoft Learn, “Azure Service Bus message transfers, locks, and settlement,” https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement
- Microsoft Learn, “Azure Service Bus dead-letter queues,” https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues
- IETF, “HTTP Semantics: Idempotent Methods,” https://www.rfc-editor.org/rfc/rfc9110.html#name-idempotent-methods
- Stripe, “Idempotent requests,” https://docs.stripe.com/api/idempotent_requests
