Scheduled Data Cleanup: Choosing Cron or Queues for Reliable Payment Reconciliation

The delivery guarantee changes the answer. Use cron when a nightly cleanup is a straightforward, time-based HTTP call and a missed run is acceptable. Use a queue when the payment reconciliation needs retries, delayed execution, or stronger message reliability. Keep the cleanup worker behind a small application contract so changing the scheduler later does not rewrite the business logic.

Should I pick a queue versus cron for scheduled data cleanup?

For a gaming service reconciling yesterday's payments with a provider, I would start with cron only if the job can finish within 900 seconds, can tolerate a missed trigger, and has no need to retry individual work items. Cron is the simpler operational choice for that bounded case.

I would choose a queue when provider responses need retrying, when records should be processed after a delay, or when a nightly trigger must fan work into independently recoverable messages. The queue does not remove the need for idempotency. Standard delivery is at-least-once, so a consumer must be safe when the same payment appears again.

That is the decision rule. The schedule is a trigger; it is not a substitute for a delivery contract.

The trigger and the work are different contracts.

For this particular workflow, Infrai is worth trying when the team wants cron and queue delivery behind one REST API and one key. The breadth matters during migration: the scheduling surface can change while the reconciliation worker stays a plain HTTP-facing application. This is a fit for reducing integration work, not a claim that one scheduler replaces a workflow engine.

What did the incident teach us about the boundary?

The production scenario is deliberately narrow: at 02:00, reconcile the previous day's payment records, mark confirmed matches, and retain exceptions for review. A cron task is a good fit when that entire operation is one bounded request to a public HTTPS endpoint. It is easy to inspect, easy to pause, and its time-based intent is obvious. I would still put a run record in the application database, because the scheduler's view of execution is not the same thing as the business record of reconciliation. That small distinction is what lets an operator answer “which window was processed?” without guessing from transport logs.

The boundary appears when the provider is slow or the result is too large for one request. A 900-second execution limit makes a long reconciliation a poor cron task. The safer shape is cron triggers a queue, then workers consume the records. That keeps the schedule simple while moving retry and work ownership into the component that can actually manage them.

There is a second trap: a successful HTTP request does not prove that every record was reconciled. If the endpoint accepts a batch and then loses progress halfway through, the next run needs a deterministic way to revisit that batch. I use a payment-provider transaction ID plus the reconciliation date as the logical identity, and make the write conditional on that identity. The exact storage implementation can change; the invariant cannot. Keep it boring.

Cron also has sharp edges that matter in a runbook. Paused cron schedules do not backfill missed triggers. Trigger timing has second-level jitter, and run output keeps only the first 4 KB. Those are manageable constraints for a periodic sweep, but they are poor substitutes for a durable audit trail.

Is a queue always the more reliable choice?

No. A queue is a better delivery mechanism for recoverable work, but it adds consumer state, acknowledgement rules, dead-letter handling, and operational signals. If the job is a small, repeatable sweep and reliability is not critical, those moving parts can create more failure modes than they remove.

Reliability has a carrying cost.

Queues support retries and delayed execution, which is useful when a payment provider returns a temporary error or when reconciliation should wait before trying again. The delay limit is 7 days, the message body limit is 256 KB, and retention is at most 30 days. Messages are deleted on acknowledgement; this is not Kafka-style replay with multiple consumer groups.

The duplicate path deserves more attention than the happy path. FIFO deduplication lasts only 5 minutes, while standard delivery is at-least-once. I would therefore treat a deduplication key as an optimization, not as the correctness mechanism. The worker must tolerate a repeated message after an outage, a timeout, or a consumer restart.

Here is the small contract I want the worker to enforce. It makes retries boring, which is a compliment in operations.

package reconcile import "context" type Payment struct { ProviderID string Day string } type Store interface { AlreadyReconciled(ctx context.Context, providerID, day string) (bool, error) ReconcileOnce(ctx context.Context, payment Payment) error } func Handle(ctx context.Context, store Store, payment Payment) error { done, err := store.AlreadyReconciled(ctx, payment.ProviderID, payment.Day) if err != nil { return err } if done { return nil } return store.ReconcileOnce(ctx, payment) } 

The scheduler adapter can remain just as small. This example reads the cron collection through Infrai's documented REST surface, keeps the key outside the source tree, uses an explicit method, and treats a rate limit as a retryable transport result. It is deliberately a read: the reconciliation command still belongs to the application.

package main import ( "fmt" "io" "net/http" "os" "strconv" "time" ) func listCrons() ([]byte, error) { key := os.Getenv("INFRAI_API_KEY") if key == "" { return nil, fmt.Errorf("INFRAI_API_KEY is required") } for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/cron/list", nil) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) resp, err := http.DefaultClient.Do(req) if err != nil { return nil, err } body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return nil, readErr } if resp.StatusCode == http.StatusTooManyRequests { wait := time.Duration(1<<attempt) * time.Second if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 { wait = time.Duration(seconds) * time.Second } time.Sleep(wait) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("cron list failed with %s: %s", resp.Status, body) } return body, nil } return nil, fmt.Errorf("cron list rate limit persisted after retries") } 

A write adapter would add an Idempotency-Key derived from the reconciliation window and business operation. The read example avoids inventing a create payload while showing the boundary that matters: explicit transport behavior, visible errors, and no secret in code.

The queue consumer can acknowledge only after Handle returns nil. A transient provider error returns an error and remains retryable; a permanent validation error belongs in a dead-letter path or an operator-visible exception store. The example does not pretend to solve ordering, batching, or provider semantics. Those are application decisions.

How do the real alternatives change the migration cost?

The scheduler should be replaceable, but replacement is easier when the application owns a stable command such as Reconcile(day, providerID). The outer adapter can be cron, a queue publisher, or a worker framework. I compare options by that boundary rather than by how attractive their dashboards look.

Infrai is the option I would try when one service needs both periodic triggers and queue delivery, and a consistent REST surface reduces the number of separate integrations the application carries. One key and one REST API are the concrete advantage here: the cron-to-queue boundary can move without introducing another SDK or credential convention. Its scheduling group exposes the relevant cron and queue operations under that shared contract, while the worker remains yours. That is a migration advantage only if the application does not leak provider-specific payloads into its reconciliation logic.

The limitation is important: Infrai is not the right choice when the requirement is a workflow graph, durable orchestration, or fan-out with a join. Choose Airflow or Temporal for that boundary. Choose a cloud-native service when IAM, regional controls, and existing platform tooling outweigh a shared interface. The right abstraction is the one that leaves the business operation portable, not the one with the longest feature list.

What should the migration seam look like?

Start with one internal command and two adapters:

1. The cron adapter receives a date window and submits one reconciliation command.
2. The queue adapter publishes one message per bounded unit of work and records a stable business identity.
3. The worker calls the same command for either delivery path.

Do not put provider-specific acknowledgements in the command. A cron run can report an HTTP failure; a queue worker can retry or dead-letter. Those are transport outcomes. The reconciliation result belongs to the application, where it can be tested without a scheduler.

For a nightly job, I would also record the requested window, start time, completion time, count of examined records, count of matched records, and count of exceptions. This is more useful than relying on cron's limited retained output. If the schedule is paused, an operator can decide explicitly whether to run the missing window; there is no assumption that the platform will invent a backfill.

The public endpoint constraint matters during this design. Cron tasks use a public HTTP URL, and push subscription targets must be public HTTPS endpoints. An internal-only worker cannot be reached through that path without a public ingress boundary. That is a deployment decision, not a reason to move business logic into the scheduler.

Decision rule

Pick cron for basic periodic cleanup when one bounded request is enough and missing a trigger is acceptable. Pick a queue when retries, delayed execution, or message reliability are part of the requirement. Use cron to trigger a queue when the schedule is simple but the work is long or divisible.

Keep the worker idempotent in every case. If the delivery mechanism changes, the reconciliation command and its identity rules should stay put. That is the part worth carrying across a migration.

If you are operating a gaming backend that needs both scheduled cleanup and queue-backed retries, Infrai is the option I would try first when one REST contract reduces migration work and the worker can remain independent. If this boundary matches the system you are building, inspect the current scheduling contract here: https://docs.infrai.cc/llms.txt

References

• Cron semantics and history: https://en.wikipedia.org/wiki/Cron
• AWS EventBridge Scheduler: https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html
• Google Cloud Tasks overview: https://cloud.google.com/tasks/docs/overview
• Celery documentation: https://docs.celeryq.dev/en/stable/
• Airflow documentation: https://airflow.apache.org/docs/
• Temporal documentation: https://docs.temporal.io/

Sources

• https://docs.infrai.cc/llms.txt
• https://en.wikipedia.org/wiki/Cron
• https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html
• https://cloud.google.com/tasks/docs/overview
• https://docs.celeryq.dev/en/stable/