Use the cron trigger as a pacer, not as a starting gun. On a live game backend the tick should admit only as many background jobs as the worker pool can actually finish before the next tick fires, and every long-running job should checkpoint a cursor so the 15-minute execution limit ends a chunk instead of killing a job. Do that, and a backlog of 40,000 replay cleanup jobs and cohort reports turns into a queue with a drain time you can predict and alert on.

The constraint that makes this measurable comes first: you can't pace what you haven't measured.

The system I have in mind is the boring half of a live-service game. Matches end, and each one leaves work behind — screenshots to thumbnail, replay blobs to summarize with a model call, expired session rows to sweep, a weekly report per guild. None of it is user-facing. All of it shares one worker pool, and that pool sits behind an upstream inference API with a per-minute request budget it did not negotiate.

The failure mode: a tick that enqueues everything at once

The simple version is one query and one loop. A scheduled job wakes up, selects every row whose due_at has passed, pushes them all onto the queue, and lets the workers sort it out. Under steady load that's fine, because the number of rows that come due in five minutes is roughly the number the pool finishes in five minutes.

Recovery is where it falls apart.

Pause the consumers for a patch deploy, or lose an hour to a bad migration, and the first tick after the outage admits the entire accumulated backlog at once. Now every worker in the pool is calling the same upstream at the same moment, so the upstream starts refusing requests. The refusals get retried on a fixed delay, which re-synchronizes them into the next spike. Meanwhile the messages that are genuinely in flight sit past their visibility timeout, get redelivered to a second worker, and the guild that was owed one weekly report gets two. The queue depth graph goes flat while the cost graph keeps climbing — which is the tell that you're paying to redo work rather than to finish it.

The execution cap turns that from expensive into unbounded. Serverless function runtimes commonly stop an invocation at a hard wall — 900 seconds is the maximum configurable timeout on Lambda — and a job that starts from row zero every time it's killed will burn its share of the rate limit forever without ever reaching the end of its input. Each restart repeats the same model calls, so the token spend for that job grows linearly while its progress stays at zero.

One more thing worth designing around: scheduled triggers are best-effort. GitHub Actions documents a shortest schedule interval of 5 minutes and warns that scheduled workflows can be delayed during periods of high load, which is typical of hosted schedulers generally. Treat the tick as "at least once, roughly on time" and the whole design gets simpler, because you stop writing logic that assumes an exact interval.

How should a cron trigger enqueue long-running cleanup jobs without stalling the worker queue?

Three decisions, in this order: admission control at the tick, continuation at the worker, idempotency at the queue.

Admission control means the tick computes a budget instead of a set. Take the rate the pool can sustain without tripping the upstream limit, multiply by the tick interval, subtract what's still in flight, and admit that many rows — highest priority first. If the upstream allows 60 requests per second and one chunk of replay summarization makes four calls, your sustainable rate is somewhere near 8 jobs per second, and the tick admits about 2,400 of them. The rest stay in the table. They are not lost, they are not in the queue, and nothing in the system is holding a connection open on their behalf.

Continuation means a job is a cursor, not a duration. The worker processes chunks in a loop, writes the new cursor after each one, and checks the clock before starting the next: if the remaining budget is thinner than a chunk takes at p95, it stops, re-enqueues itself with the current cursor, and returns cleanly. The invocation that gets killed at the wall is the one that failed to do this. The one that stops two minutes early loses nothing.

Idempotency is what makes the first two safe to run twice. Key every message on (job_id, cursor) rather than on job_id. FIFO queues help here — the message group ID gives you ordering per match while still letting different matches drain in parallel, and content-based deduplication drops a repeat of the same deduplication ID inside a 5-minute window. That window is the part people misread. It protects you from a tick that fires twice, not from a redelivery an hour later, so the durable check still belongs in your own store: write the cursor and the result under a unique constraint, and let the second attempt lose the race harmlessly.

Add one lane split and you're done: a recovery lane and a live lane, with a fixed share of the budget reserved for live traffic. Matches that just ended shouldn't queue behind a two-day backlog because the sort order says the old rows go first.

The pacer tick in about thirty lines of code

The tick itself is arithmetic. That's deliberate — it means the whole thing is unit-testable in a notebook, with no broker running, which is how I'd rather develop this kind of code before it becomes an on-call surface.

import math import time from dataclasses import dataclass TICK_SECONDS = 300 # the scheduled trigger fires every five minutes LIVE_SHARE = 0.3 # reserved for matches that just ended MAX_INFLIGHT = 4_000 @dataclass class TickResult: admitted: int deferred: int eta_minutes: float def drain_once(store, queue, drain_rate: float) -> TickResult: """Admit at most one tick's worth of work. Everything else waits in the table.""" inflight = store.count_inflight() budget = max(0, min(int(drain_rate * TICK_SECONDS), MAX_INFLIGHT - inflight)) live_budget = int(budget * LIVE_SHARE) batch = store.claim_live(limit=live_budget) + store.claim_backlog(limit=budget - live_budget) for job in batch: queue.send( body={"job_id": job.id, "cursor": job.cursor}, group_id=job.match_id, # ordering per match, parallelism across matches  dedup_id=f"{job.id}:{job.cursor}", # a double tick enqueues one message  ) pending = store.count_pending() eta = pending / (drain_rate * 60.0) if drain_rate > 0 else math.inf return TickResult(admitted=len(batch), deferred=pending, eta_minutes=round(eta, 1))

drain_rate is a parameter, not a constant, because the tick is also the control loop. When the upstream refusal rate for the last interval goes above a small threshold, halve it; when a full interval passes clean, add a little back. Multiplicative decrease and additive recovery, applied once per tick, beats per-request retry tuning — it converges instead of oscillating, and there's exactly one number to look at when someone asks why the backlog is draining at the speed it is.

The worker side is the other half of the contract:

CHUNK_BUDGET_SECONDS = 90 # p95 cost of one chunk, measured, not guessed  def handle(message, deadline_at: float) -> str: job = load(message["job_id"]) cursor = message["cursor"] while cursor is not None: if time.time() + CHUNK_BUDGET_SECONDS > deadline_at: requeue(job.id, cursor) # a continuation, not a restart  return "paused" cursor = process_chunk(job, cursor) # returns the next cursor, or None at the end  store_cursor(job.id, cursor) # committed before the next chunk starts  return "done"

The examples are Python because that's where the job code lives, though nothing here depends on it. Ported to Node.js the arithmetic is identical, and so is the failure it prevents.

What to measure before copying this

Queue depth is the metric everyone graphs and the one that tells you least. Depth going up during an incident is not information; it's the definition of the incident. Drain ETA — pending work divided by the completion rate observed over the last several ticks — is the number that answers the question an incident channel actually asks, which is when this will be over. Alert on the ETA crossing a threshold, and on its second derivative, since an ETA that keeps growing means admission is still outrunning completion.

Four signals are enough to run this in production:

Signal What it tells you What to change
Drain ETA rising across ticks Completion is slower than admission Lower drain_rate until the ETA flattens, then climb back slowly
Deferred count pinned at zero The pacer isn't the binding constraint Raise the budget, or drop the machinery — it's not earning its complexity
p95 chunk duration near the wall Chunks are sized too close to the execution limit Split the chunk; a checkpoint you never reach isn't a checkpoint
Refusal rate above a few percent Admission is outrunning the upstream budget Halve the rate this tick, recover additively over the next few

Two of those want a test rather than a dashboard. Record a day of real job payloads, replay them in staging against a fake limiter that refuses above a set rate, and assert two properties: the tick never admits more than its budget, and a worker killed mid-chunk resumes at the stored cursor with no duplicate output. Both assertions run in under a second on a laptop because neither one needs the broker. The cost per drained job belongs in the same harness — a backlog drain multiplies model spend by the number of pending items, and finding out afterwards is a bad way to find out.

When this pattern is the wrong tool

The design assumes a job can be cut into chunks. If yours can't — one long transcode, one call to a model that streams for eleven minutes and can't resume — pacing the enqueue won't save you, because the unit of work is larger than the runtime's wall. Stick with a long-lived process on a container or VM where nothing kills the invocation at a fixed timeout, and spend your effort on making that process restartable instead. The alternative most teams reach for first is a bigger worker, and that just moves the wall a little further out.

Strict global ordering is the other bad fit. Put every message in a single group and the queue serializes them; drain time becomes the sum of job durations and no amount of admission tuning changes it. That's not a limitation of the pattern so much as the price of the guarantee, and it's usually cheaper to relax the ordering requirement to per-entity than to buy more workers that won't run.

And if the pool never saturates, don't build this. A fixed concurrency cap in the worker is two lines and no operational surface. I'm genuinely unsure where the crossover sits — it depends on how spiky your backlog is and how much of your budget one job consumes — but the deferred-count signal above will tell you which side of it you're on within a day.

Further reading