For rate-limited job processing, choose cron or a queue based on recovery state, then enforce the per-minute limit where a Node.js backend makes the downstream call. The useful choice is the smallest system that can replay interrupted work without repeating an external effect.

Short answer: use a scheduled scan for bounded, reconcilable work; add durable per-item state when jobs need independent pacing, retries, and an audit trail.

I treat this as an experiment, not a throughput contest. A simple scheduled process can look perfect in a notebook, then hide the hard question: what records that item 1,137 completed when the process stops between the downstream call and its own state update? If the answer is "the next run will find it," that can be a sound design. If the answer depends on remembering each delivery, the state has to be explicit.

How should a Node.js backend process rate-limited jobs per minute?

Put the rate limit immediately before the downstream request. A producer-side limit only controls admission; retries, another producer, or a restarted worker can still create a burst at the API boundary. One worker can keep timestamps locally. Multiple workers need shared coordination or a deliberately partitioned allowance, because four workers each allowed 30 calls per minute are not a 30-per-minute system.

The dispatch decision should be separate from the enqueue decision. Enqueue time says work exists. Dispatch time says it is safe to make the external call. That distinction makes backpressure observable: pending count and oldest-job age describe stored work, while dispatch starts and 429 responses describe pressure on the dependency.

Here is a deliberately small worker contract. It records the start of each downstream attempt in a sliding window, acknowledges only after the idempotent business action returns, and leaves queue storage behind an interface so the scheduling choice does not leak into application logic.

import time from collections import deque from dataclasses import dataclass from typing import Protocol @dataclass(frozen=True) class Delivery: receipt: str body: bytes class Queue(Protocol): def receive(self) -> Delivery | None: ... def ack(self, receipt: str) -> None: ... class SlidingWindow: def __init__(self, allowed: int, seconds: float = 60.0) -> None: self.allowed = allowed self.seconds = seconds self.starts: deque[float] = deque() def wait_for_slot(self) -> None: while True: now = time.monotonic() while self.starts and now - self.starts[0] >= self.seconds: self.starts.popleft() if len(self.starts) < self.allowed: self.starts.append(now) return time.sleep(self.seconds - (now - self.starts[0])) def run(queue: Queue, limiter: SlidingWindow) -> None: while True: delivery = queue.receive() if delivery is None: time.sleep(0.25) continue limiter.wait_for_slot() apply_idempotently(delivery.body) queue.ack(delivery.receipt)

Short code. Important contract.

For an HTTP response with status 429, inspect Retry-After when it is present and delay the next eligible attempt accordingly. A limiter should also reserve capacity when interactive traffic shares the same downstream quota. Otherwise background work can be perfectly regular while users receive the limit response.

The experiment: scheduled scan or durable delivery state

A scheduled scan fits when a source database can reliably answer "what remains?" on every run. Consider a nightly process that recomputes derived summaries: the input rows are still present, operations are idempotent, and a later scan can repair a missed interval. The scheduler is merely a clock; the database is the recovery mechanism. This path has a small operational surface, which matters for a service that already runs a database and does not need per-item progress history.

Durable delivery state becomes worthwhile when each item has its own retry deadline, pacing decision, or operator-visible outcome. A large enrichment run is the common case: an item may complete its external call and lose its acknowledgement, or it may be deferred because the per-minute budget is full. Writing a job record with a stable logical ID, state, attempt count, and next eligible time makes those outcomes inspectable. A schedule can still discover work and create those records; the two patterns are complementary.

The catch is that delivery durability is not exactly-once business behavior. A worker can repeat a delivery after an interruption. Make the handler use an idempotency key near the business state transition, and retain enough result data to recognize a completed logical job before issuing the side effect again. Don't make an expensive model call the only record of success.

For repeatedly unsuccessful messages, a dead-letter destination can isolate items for inspection. It is not a substitute for a failure policy. AWS warns that attaching a dead-letter queue to a FIFO queue can break exact order, so workflows whose correctness depends on strict sequence need a reconciliation plan or a deliberate stop-the-line policy instead of casually moving one item aside.

What should the replay evaluation measure before a backend is copied?

Run the same test against each candidate shape. Seed representative jobs, stop a worker after the external action but before acknowledgement, restart it, and check that each logical job reaches one terminal business result. Then include a malformed payload and a simulated rate-limit response. The goal is not to crown a backend; it is to find the smallest design whose recovery behavior the team can explain from its records.

Use a compact test matrix:

Case Expected evidence
Worker interrupted after side effect Replay reuses the idempotency result or safely detects completion
Payload violates schema Terminal rejection with a useful reason; no blind retry
Dependency responds with 429 Deferred retry respects Retry-After when supplied
Per-minute allowance is exhausted Job remains pending and oldest-job age increases visibly

For AI workloads, add input tokens, output tokens, and downstream calls per successful logical job. A retry that repeats a generation may be invisible in raw queue throughput yet very visible in prompt spend. I am not sure a single completion-latency target transfers between workloads; measure p50 and p95 completion latency, oldest-job age, retry count by reason, terminal rejection count, and duplicate-effect detections on the corpus that resembles production.

Choose the operational boundary, then keep it observable

The cheapest design is often the one that avoids building recovery machinery twice. A scheduled scan plus source database is appropriate for short, repeatable batches that can be reconciled. A job table is useful when transactional proximity to existing application data matters. Independent delivery state earns its complexity when partial completion, paced dispatch, and per-item investigation are requirements.

Not every workload needs a queue. Stick with a schedule when an item can be safely rediscovered and rerun, and choose durable job state when the system must preserve each item's retry history and timing. Before adopting either, run the interruption test, define terminal outcomes, and watch completion age alongside request volume. The architecture that survives that evaluation is the one to operate.

References