6 myths about calling rate-limited APIs from background jobs

Background jobs and third-party APIs have a complicated relationship. Your queue can produce work as fast as your workers can pull it. The API on the other side has a limit: so many requests per second, per minute, or per day. And the moment those two numbers disagree, things get interesting.

Here are six beliefs about rate limits that tend to hold right up until the first incident.

Myth 1: "More workers means faster processing"

Up to the API's limit, yes. Beyond it, more workers just means more rejected requests.

If an email API allows a certain number of requests per second for your account, it doesn't matter whether you run 5 workers or 50. Above the limit, the extra requests come back as 429 Too Many Requests, consume retries, and often make total throughput worse, because every rejected request is wasted work that has to be redone.

Instead: size concurrency against the downstream limit, not against your queue depth. If the limit is the bottleneck, the number of workers calling that API should be set from the limit.

Myth 2: "The rate limit is per worker"

Every worker obeys "no more than 10 requests per second", so you're fine. Then you deploy 8 replicas, and the API sees 80 per second from your account.

Rate limits are almost always enforced per API key, account or IP, not per process. A limit enforced locally in each worker doesn't add up to a global limit.

Instead: coordinate the limit across workers. Common approaches:

A shared token bucket in Redis or your database that every worker takes from before calling the API.
A dedicated queue for that API with a fixed number of consumers, each with a local limit, so the total is known.
A single "dispatcher" that hands out permission at the right rate.

# a minimal shared limiter: N requests per window, across all workers (Redis) import time, redis r = redis.Redis() def acquire(key="emailapi", limit=10, window=1.0): while True: slot = int(time.time() / window) k = f"rl:{key}:{slot}" n = r.incr(k) if n == 1: r.expire(k, int(window) + 1) if n <= limit: return time.sleep(window - (time.time() % window))

A fixed window like this is simple and good enough for many cases; it can allow bursts at window boundaries, so use a sliding window or token bucket if the API is strict.

Myth 3: "On a 429, retry immediately"

An immediate retry after a 429 is almost guaranteed to get another 429. Multiply that by every worker and you've built a loop that hammers the API at full speed while accomplishing nothing.

Instead: respect Retry-After if the API sends it (as seconds or an HTTP date). If it doesn't, use exponential backoff with jitter. And crucially, delay the job rather than sleeping in the worker. A worker sleeping for 30 seconds is a worker not processing anything else. Re-enqueue the job with a delay and let the worker move on.

def handle(job): resp = call_api(job.payload) if resp.status_code == 429: wait = parse_retry_after(resp.headers.get("Retry-After")) or backoff(job.attempt) queue.enqueue_in(wait, handle, job.payload, attempt=job.attempt + 1) return resp.raise_for_status()

Myth 4: "A 429 is a failure, so count it as one"

If 429s count towards the job's maximum attempts, a busy afternoon can push perfectly good jobs into the dead-letter queue. Nothing was wrong with them; the API was just busy.

Instead: treat rate-limit responses differently from real errors. A 429 means "not now", not "never". Many teams give rate-limited retries their own budget, measured in time ("give up after 6 hours") rather than attempts, and keep the attempt counter for genuine failures like 500s or timeouts.

Myth 5: "We're well under the limit, so we don't need to think about it"

Average rate and peak rate are different things. You might average 2 requests per second against a limit of 10, and still hit the limit every morning at 09:00 when the scheduled jobs all fire at once, or after an outage when the backlog drains at full speed.

The backlog case is the sneaky one. The API was down for 20 minutes, jobs piled up, it comes back, and your workers immediately try to process 20 minutes of work in 20 seconds.

Instead: rate-limit even when you're "comfortably under", so that bursts and backlog drains are smoothed automatically. Spread scheduled work rather than firing it all on the hour.

Myth 6: "The API's documented limit is the limit"

Documentation describes the policy. Reality includes limits per endpoint, per resource, daily quotas, concurrent connection limits, and limits that change by plan or under load. Some APIs tell you in response headers; the common pattern is something like X-RateLimit-Remaining and X-RateLimit-Reset, although names vary between providers.

Instead: read the headers the API actually sends, log them, and slow down proactively when the remaining budget gets low instead of waiting for a 429. And keep the configured limit somewhere you can change without a deploy, because you will need to.

The pattern underneath

All six myths come from treating a shared, external constraint as if it were local. The fix is always some version of the same thing: make the rate limit a first-class part of your job system, shared across workers, applied before the call, and paired with delayed retries rather than sleeping or spinning.

That last part, "try this again in 40 seconds, without holding a worker hostage", is the bread and butter of delayed job scheduling, and it's exactly the kind of thing we built delay.run to handle.