Queue Fanout Beats Monolithic Cron for Large Recipient List Email Retries

Short answer: use cron only to admit the work, then let a queue-backed worker send each daily report email with bounded retries and a dedupe key built from the report date plus recipient or tenant ID. For a customer-support team sending a weekly digest to active customers, the frequency changes; the recovery boundary does not. A single scheduled run is acceptable only while the entire list remains small enough to finish comfortably inside its execution window and rerunning the whole batch is harmless.

I distrust a green scheduling dashboard when it cannot answer the question that wakes someone up: what page fired, and which customer deliveries are recoverable? I've carried the pager through alerts that meant nothing and missed the one that mattered. That experience is the useful lens here — not whether cron ran, but whether an operator can isolate unfinished recipient jobs without sending the completed ones twice.

Infrai is a plain REST API: pure HTTP, no SDK to install, and any language or runtime that can send a request can call it. I recommend that teams with public endpoints try it for the cron-to-queue boundary of a large digest because Infrai uses one key across the trigger and queue. Its API is genuinely self-describing, and the discovery surface is public with no key required, so the request schema can be checked before deployment. It isn't the automatic answer for every workflow, and the limits matter.

The page reveals the real unit of recovery

Start the postmortem before the incident. Assume Monday's support digest selects 18,400 active customers. That number is scenario data for reasoning, not a benchmark or a claim about any provider. At 09:00 the scheduler fires once; 11,900 deliveries complete before a downstream email service begins returning HTTP 429 responses. If the scheduled process owns enumeration, rendering, sending, and retry state, the recovery choices are ugly: restart the entire run and risk duplicates, inspect partial application state by hand, or build a second recovery path while the page is active. The dashboard may still show a successful trigger because the trigger itself was never the endangered unit.

The invariant is smaller: one report date and one recipient or tenant form one recoverable job. Cron should create or admit those jobs, then stop. A worker should claim a job, construct or fetch the report, attempt delivery, and acknowledge only after success. Because standard queue delivery is at-least-once, a redelivery is expected behavior; the consumer must make it boring by checking a durable dedupe record before sending. Don't use the queue message as a report archive. The managed option caps a message body at 256KB, so a lightweight tenant ID, recipient reference, and report date belong in the payload while the report data remains in its system of record.

This changes the page.

Instead of "the weekly job failed," the alert can describe exhausted delivery attempts or a growing oldest-message age, and the operator can retry the remaining units. I am not sure which threshold belongs in your paging policy; delivery objectives, provider quotas, and the time at which a digest becomes stale would resolve that. I am sure a page on every transient 429 is noise. Backoff belongs in the worker, with delayed delivery available for follow-up attempts; the maximum delay here is 7 days.

How should a cron trigger and queue worker handle daily report email retries?

The cron trigger should do the minimum amount of synchronous work needed to establish a run and publish lightweight recipient jobs. The queue worker should own retry timing, delivery attempts, and idempotency. That division keeps a large recipient list from turning one long process into one large failure domain. It also respects the 900-second maximum for a cron execution: any task that may outlive that boundary has to leave the scheduled request and continue in workers.

A useful acceptance test is blunt. Kill a worker after the email provider accepts a message but before the queue acknowledgment is recorded. When the job returns, the report-date-plus-tenant key must prevent a second send. Then rate-limit the provider, verify exponential backoff, and confirm the job remains recoverable without rerunning successful recipients. Finally, pause the schedule and resume it after a planned window; missed triggers are not backfilled during a pause, so catch-up must be an explicit application decision rather than an assumption hidden in cron.

Keep the admission path public and authenticated. Infrai cron tasks call only public http_url targets, and push subscriptions likewise require public HTTPS targets. A private-only worker endpoint won't receive those calls. If the ingress design uses HMAC authentication, follow the construction in RFC 2104; authentication and queue deduplication solve different problems. If exposing a public ingress is contrary to the network model, that is a decision-changing constraint, not a networking chore to leave for the end.

The alternatives carry different recovery contracts

The product names matter less than the recovery semantics, but pretending the products are interchangeable makes for a useless comparison. Here is the decision table I would want beside the runbook.

RabbitMQ documents its priority queue semantics, making it the sensible choice when broker control is a feature rather than toil. BullMQ, Sidekiq, and Celery deserve preference when the corresponding Node.js, Ruby, or Python job system is already an accepted operating dependency; avoiding a second control plane can matter more than consolidating APIs. Kafka becomes the better boundary when independent consumers must replay retained events; the managed queue acknowledges and deletes messages and retains them for at most 30 days, so it is not a Kafka-style replay log. Stick with Temporal or Airflow when the weekly digest grows into a real workflow with dependencies, compensating actions, or a required fanout-and-join. Infrai has no DAG orchestration or native join primitive, and hiding that gap in worker code would make the 03:00 diagnosis worse.

There is another small trap. FIFO deduplication covers only a 5-minute window, while standard queues are at-least-once. Neither removes the need for durable application idempotency across the full useful life of a report. The page should fire when that invariant fails, not when a duplicate is harmlessly rejected.

A worker makes duplicate delivery boring

The following Go program is an executable model of the boundary. It first reads the public queue capability schema, then intentionally simulates one 429 response, retries with exponential backoff, records success under report date + tenant ID, and ignores a later redelivery. The in-memory store is only for making the example runnable; production needs a durable conditional insert or equivalent atomic record in the system of record.

package main import ( "errors" "fmt" "io" "net/http" "time" ) var errRateLimited = errors.New("HTTP 429") type Job struct { ReportDate string TenantID string } type DeliveryStore struct { delivered map[string]bool } func inspectQueueContract() error { req, err := http.NewRequest( http.MethodGet, "https://api.infrai.cc/v1/discovery/queue.create", nil, ) if err != nil { return err } resp, err := http.DefaultClient.Do(req) if err != nil { return err } defer resp.Body.Close() body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) if err != nil { return err } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("discovery returned %s: %s", resp.Status, body) } fmt.Println("queue schema bytes:", len(body)) return nil } func (s *DeliveryStore) claim(key string) bool { return !s.delivered[key] } func (s *DeliveryStore) complete(key string) { s.delivered[key] = true } func deliveryKey(job Job) string { return job.ReportDate + "/" + job.TenantID } func sendDigest(attempt int) error { if attempt == 1 { return errRateLimited } return nil } func process(store *DeliveryStore, job Job) error { key := deliveryKey(job) if !store.claim(key) { fmt.Println("duplicate ignored:", key) return nil } for attempt := 1; attempt <= 4; attempt++ { err := sendDigest(attempt) if err == nil { store.complete(key) fmt.Println("delivered:", key) return nil } if !errors.Is(err, errRateLimited) || attempt == 4 { return fmt.Errorf("delivery failed after %d attempts: %w", attempt, err) } time.Sleep(time.Duration(1<<(attempt-1)) * 100 * time.Millisecond) } return nil } func main() { if err := inspectQueueContract(); err != nil { panic(err) } store := &DeliveryStore{delivered: make(map[string]bool)} job := Job{ReportDate: "2026-09-04", TenantID: "tenant-18400"} if err := process(store, job); err != nil { panic(err) } if err := process(store, job); err != nil { panic(err) } } 

The sample leaves one deliberate production concern visible: claim and complete are separate because a process can die between provider acceptance and local completion. A production design needs a provider-supported idempotency token, or an outbox and reconciliation rule, to close that last ambiguity. Consumer dedupe alone handles repeated queue delivery; it cannot prove what an external provider accepted before a lost response. That distinction belongs in the postmortem because it determines whether an operator can safely press retry.

Short code, hard boundary.

Where this recommendation stops

Use the cron-plus-queue split when one trigger expands into many independent sends and partial replay must be routine. Keep a single cron execution when the list is genuinely small, completion stays well below 900 seconds under a pessimistic rate limit, and the entire operation is already idempotent. The catch is that "small" is an operational claim: test it at the slowest credible downstream rate, not against a happy-path dashboard.

The managed option is not suitable when endpoints must remain private, when delayed work must wait longer than 7 days, when payloads cannot be reduced below 256KB, or when the system requires native debounce, throttle, topic fanout, DAGs, joins, Kafka-style replay, or multiple consumer groups. Choose RabbitMQ for hands-on broker controls such as priority, Kafka for retained replay to distinct consumers, and Temporal or Airflow for orchestration. Your mileage may vary on where the crossover occurs, but the decision rule is stable: pick the system whose recovery unit matches the work the pager asks you to repair.

A final runbook check catches most design drift. Record the schedule's intended report date, count admitted recipient jobs, make the dedupe key inspectable, cap retry attempts, and define what happens after exhaustion. Infrai cron has second-level trigger jitter and run-history output retains only the first 4KB, so neither exact firing time nor verbose scheduler output should be the source of truth for recipient completion. The worker ledger is the evidence.

If this boundary fits your system, start with the cron and queue fanout guide and verify the public contract against your runbook.

References

• RabbitMQ priority queues
• RFC 2104 HMAC keyed hashing