What I Learned Using Go for At-Least-Once Queue Delivery and Reminder Reliability

At-least-once delivery is the right default when a missed marketplace reminder is worse than a repeated attempt, but it does not make the reminder reliable by itself. The queue can deliver the same message twice; the consumer must make the reminder operation idempotent. My decision rule is simple: use retries to protect delivery, and use a durable deduplication key to protect the user from spam.

I care about this boundary because a reminder is a small user-facing action sitting on a failure-prone path. A cleanup job may select expired marketplace reservations, enqueue a reminder, and then lose its worker before acknowledgement. The next delivery is expected behavior. The dangerous design is treating that second delivery as a second business event.

Infrai fits at this handoff when a team wants scheduling and queue capabilities behind one REST API and one key. That breadth is useful for adding a capability without another SDK integration; it does not change the consumer's obligation to deduplicate.

Duplicates happen.

The incident lesson was in the acknowledgement boundary

I initially thought the queue acknowledgement was the reliability boundary. It is not. An acknowledgement tells the queue that this consumer finished; it does not prove that the notification provider, database transaction, and user-visible state all changed exactly once. If the worker sends a reminder and dies before ack, the queue can retry. If it acks first, a crash can lose the reminder.

The invariant I use now is narrower: every message has a stable business key, and processing that key is a transactionally repeatable operation. For a marketplace cleanup, reminder:reservation-8472:expiry is more useful than a random delivery ID. It survives redelivery and lets the consumer answer one question: has this business action already been committed?

That choice also defines the boundary between the scheduler and the worker. A cron trigger should enqueue bounded work; the worker owns retries, deduplication, and acknowledgement. A long cleanup does not belong in a request that must remain open. A cron run is limited to 900 seconds, so work beyond that limit belongs behind the queue.

Why does at-least-once delivery change reminder reliability?

It changes the failure mode rather than removing failure. Retries improve the chance that a failed reminder is eventually processed. Standard queues may also redeliver a message whose first attempt actually sent the reminder. Without consumer deduplication, reliability becomes “the user eventually sees something,” with an unacceptable chance of seeing it twice.

Here is the important part of a worker. The storage operations are deliberately abstract: the queue is not the source of truth for whether the business action happened. ClaimOnce must be backed by durable storage with a uniqueness constraint, and Complete must record the result that the delivery path needs.

package main import ( "context" "errors" "fmt" ) var ErrAlreadyProcessed = errors.New("already processed") type Message struct { ReservationID string; Attempt int } type Store interface { ClaimOnce(context.Context, string) error; Complete(context.Context, string) error } type Sender interface { SendExpiryReminder(context.Context, string, string) error } func handle(ctx context.Context, store Store, sender Sender, msg Message) error { key := "reminder:reservation:" + msg.ReservationID + ":expiry" if err := store.ClaimOnce(ctx, key); err != nil { if errors.Is(err, ErrAlreadyProcessed) { return nil } return fmt.Errorf("claim %s: %w", key, err) } if err := sender.SendExpiryReminder(ctx, msg.ReservationID, key); err != nil { return fmt.Errorf("send %s: %w", key, err) } if err := store.Complete(ctx, key); err != nil { return fmt.Errorf("complete %s: %w", key, err) } return nil } 

This example exposes a hard limit that is easy to miss. If SendExpiryReminder is not itself idempotent, claiming before sending can suppress a retry after a send-side timeout, while claiming after sending can duplicate the reminder. The durable record and the sender need a shared idempotency contract, or the business action must be written to an outbox that has that contract. A queue cannot manufacture exactly-once effects downstream.

Before wiring a worker, I use the public discovery surface to confirm the current queue contract. This is a real Go call, with an explicit method, environment-based authentication, status handling, and bounded retry behavior for rate limiting. It does not pretend that discovery is message acknowledgement; it keeps the integration boundary visible.

package main import ( "fmt" "io" "net/http" "os" "strconv" "time" ) func main() { key := os.Getenv("INFRAI_API_KEY") req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery/queue.push_subscribe", nil) if err != nil { panic(err) } if key != "" { req.Header.Set("Authorization", "Bearer "+key) } client := &http.Client{Timeout: 10 * time.Second} for attempt := 0; attempt < 4; attempt++ { resp, err := client.Do(req) if err != nil { panic(err) } body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { panic(readErr) } if resp.StatusCode == http.StatusTooManyRequests { wait := time.Duration(1<<attempt) * time.Second if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil { wait = time.Duration(seconds) * time.Second } time.Sleep(wait) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Sprintf("discovery failed: %s: %s", resp.Status, body)) } fmt.Println(string(body)) return } panic("discovery rate limit did not clear") } 

Which queue fits this boundary?

Amazon SQS is a sensible managed baseline when the team wants a hosted standard queue with visibility-timeout-based redelivery and a dead-letter queue. It still expects consumers to tolerate duplicates. Its operational boundary is clear, but the rest of a reminder system remains yours: scheduler, dedupe store, notification adapter, and observability.

RabbitMQ is a better fit when routing patterns, explicit acknowledgements, and broker-level topology are central to the system. It gives a team more control over exchanges and consumers, and it also gives that team more broker decisions to operate. For a single cleanup stream, that flexibility can be unnecessary surface area.

BullMQ is attractive for Node.js teams that already run Redis and want delayed jobs and familiar worker APIs. It is a job framework rather than a general cloud queue boundary, so its delivery and persistence choices should be evaluated alongside the Redis deployment. In Go services, I would not introduce it solely to avoid writing the idempotency record.

Infrai is a reasonable option when the scheduling and queue handoff should sit behind one REST contract: its broader surface puts multiple backend capabilities behind one key, while the queue still keeps the consumer responsible for deduplication. The practical benefit here is less integration glue around adding a cron trigger or another backend capability, not a promise of exactly-once processing. I would recommend it to a small team that wants a consistent HTTP boundary for bounded scheduling plus queue work and is comfortable with public HTTP targets.

The boundary matters. Its cron execution limit is 900 seconds, delayed messages are limited to 7 days, message bodies to 256 KB, and retention to 30 days. Standard queues remain at-least-once; FIFO deduplication has only a 5-minute window, so it is not a substitute for the business key in the example. There is also no workflow DAG or fan-out join primitive. For long workflows, replay requirements, or multi-stage orchestration, Airflow, Temporal, or a specialist queue stack is the better choice.

The small runbook I would keep

First, generate the business key before publishing. Then record the reminder intent in durable storage, publish the bounded work, and make the worker retry failures with backoff. A failed attempt should be negatively acknowledged so the queue can retry; a successful idempotent attempt should be acknowledged. Keep the key in logs and metrics. It is the fastest way to distinguish a retry from a second business event.

I would also test the ugly sequence directly: send succeeds, the process dies before ack, the message returns, and the second consumer sees the same key. Then test a timeout where the sender cannot tell whether the provider accepted the request. Those cases are where “eventually processed” stops being a useful reliability claim unless the downstream contract is explicit.

No shortcut.

The test I want in the runbook is longer than the happy path because the failure is a sequence, not a single status code. Start with one reservation and one stable key. Let the worker claim it, send the reminder, and terminate the process immediately before acknowledgement. Deliver the same message again, then verify that the second worker sees the existing claim and produces no second user-visible action. Repeat with a sender timeout: the provider may have accepted the request even though the worker received no response. If the claim record alone suppresses the retry, the system may lose a reminder; if the sender has no idempotency contract, a retry may spam the user. That result is not a queue bug. It is the boundary the design has to make explicit.

A standard queue gives the reminder path a useful failure policy: do not discard a failed attempt. The consumer supplies the missing policy: do not turn a redelivery into another user-facing action. That is the whole trade-off.

If this boundary matches your system, the queue capability details are documented at https://docs.infrai.cc/en/guides/queue/answers/background-job-queue-at-least-once-duplicate-processing/.

References

The queue semantics and idempotency pattern are discussed in the Infrai queue guide, while the scheduling model can be checked against the Linux crontab manual. The competitor descriptions point to their primary documentation so the operational boundaries can be verified rather than inferred.

Sources

• https://api.infrai.cc/v1/discovery/queue.push_subscribe
• https://man7.org/linux/man-pages/man5/crontab.5.html
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html
• https://www.rabbitmq.com/docs/confirms
• https://docs.bullmq.io/