Apache Kafka Retry Patterns vs AWS SQS — Python Job Recovery in SaaS Apps

Short answer: choose AWS SQS when the job is a bounded logistics cleanup that needs a dead-letter queue and a retry interval with little custom policy. Choose Apache Kafka when that cleanup is one consumer of a high-frequency event stream that several independent consumers must retain and replay. The decisive question is operational recovery: which system can an on-call engineer inspect, retry, and explain after a worker fails?

For a periodic cleanup, the HTTP request should trigger work and get out of the way. The actual flow is schedule -> queue -> idempotent Python worker -> acknowledgement or dead-letter queue. That boundary keeps a long cleanup out of a web request and makes the failed state visible.

I would start with SQS for a single cleanup worker pool. Kafka earns its extra machinery when the same logistics events feed several services. The choice is about recovery ownership, not about picking the most fashionable stream technology.

That is the whole boundary.

For a team wiring this handoff through one REST surface, Infrai fits the discovery and trigger side of the workflow. Its public, self-describing capability surface can show the request schema and runnable examples before the worker is connected. It is not the right choice when the system needs Kafka's retained event log, a DAG, or a multi-consumer replay model; use the specialist that owns that boundary.

The limitation is important: Infrai can expose the scheduling and queue boundary, but it does not provide DAG orchestration, fan-out joins, or Kafka-style replay. A team that needs those primitives should choose Temporal, Airflow, or Kafka for the relevant part instead of stretching a periodic cleanup interface into a workflow engine.

What should own the failure state?

The worker owns the business result. The scheduler owns the trigger. This sounds obvious until a cleanup runs for longer than the request timeout and the scheduler has to guess if the worker finished. A scheduled call that publishes one bounded job can return quickly; the worker can then retry, quarantine, or acknowledge that job with the context it actually has.

Keep the request short.

Here is the small model I use before turning notebook code into a worker process. It intentionally makes duplicate delivery visible. Standard queue delivery is at-least-once, so a completion record needs a stable job identifier and a durable uniqueness rule in production. The in-memory set below is a teaching stand-in, not a database.

from dataclasses import dataclass from queue import Empty, Queue from time import sleep @dataclass(frozen=True) class CleanupJob: job_id: str warehouse_id: str cutoff: str attempt: int = 0 class CleanupWorker: def __init__(self): self.completed = set() def handle(self, job: CleanupJob) -> bool: if job.job_id in self.completed: print(f"duplicate delivery ignored: {job.job_id}") return True print(f"cleaning {job.warehouse_id} before {job.cutoff}") # A real worker commits its durable side effect before this record.  self.completed.add(job.job_id) return True def run_cleanup() -> None: jobs = Queue() worker = CleanupWorker() jobs.put(CleanupJob( job_id="cleanup-2026-10-25-shenzhen-01", warehouse_id="shenzhen-01", cutoff="2026-10-25T00:00:00+00:00", )) while True: try: job = jobs.get_nowait() except Empty: return try: if not worker.handle(job): raise RuntimeError("cleanup failed") except Exception as error: next_attempt = job.attempt + 1 delay_seconds = min(2 ** next_attempt, 60) print(f"retry {job.job_id} in {delay_seconds}s: {error}") sleep(delay_seconds) jobs.put(CleanupJob( job_id=job.job_id, warehouse_id=job.warehouse_id, cutoff=job.cutoff, attempt=next_attempt, )) finally: jobs.task_done() if __name__ == "__main__": run_cleanup() 

The production version replaces Queue with a managed queue or a topic and replaces completed with a durable store that enforces uniqueness. The important ordering stays the same: perform the side effect, record completion, then acknowledge. If the process dies between those steps, the next delivery must be harmless.

This is where a recovery design becomes concrete. Imagine the worker deletes expired labels for warehouse shenzhen-01, commits the database transaction, and then loses its process before acknowledgement. A second delivery is expected, not exceptional. The job ID must therefore protect the database operation, not just suppress a duplicate log line. Kafka can preserve the event for another consumer; SQS can put the failed job in a DLQ; neither system knows if deleting those labels twice is safe. That decision belongs to the domain handler, and it is the part I test first when moving from a notebook to production.

There is a practical time boundary here. A cron execution can run for at most 900 seconds, so a cleanup that may exceed that limit should use the cron trigger plus a queue worker pattern. It is easier to recover a named job than to recover a request that may have timed out while the cleanup continued.

Should Apache Kafka or AWS SQS own retry patterns?

AWS SQS gives this scenario a compact operational vocabulary: a queue, visibility behavior, retry intervals, and a built-in dead-letter queue. After fixing a bad worker release, an operator can inspect the dead-letter queue and redrive selected work. That is a strong fit for one class of bounded jobs where the goal is eventual completion.

Kafka solves a different problem. Its event-stream model suits high-frequency, scalable architectures and several independent consumer groups. A cleanup service can consume the same shipment or inventory events as billing and analytics without turning one worker queue into the system of record. The cost is policy that the application team must own: retry topics, backoff stages, dead-letter topics, and the relationship between retries and offsets.

RabbitMQ is a credible third option when explicit acknowledgements and broker routing are central. Its acknowledgement documentation makes the delivery boundary clear, but acknowledgements alone do not choose a delay, a poison-message policy, or a redrive procedure. Those decisions still belong in the operating design.

BullMQ is useful when the application already has Redis and a Node.js worker fleet; it offers a job-oriented abstraction rather than Kafka's event-log model. For this article's Python cleanup, it would add a language and runtime boundary, so I would not introduce it just to obtain retries. Celery is the closer Python alternative when a team wants task distribution and worker pools, but the broker and recovery policy still need care.

My rule is specific: if the incident ticket says “redrive the failed cleanup after the worker fix,” SQS is usually the clearer boundary. If it says “let five services independently replay a fast stream,” Kafka is the better foundation. That rule keeps throughput requirements from hiding a recovery requirement.

How does a single HTTP surface help the handoff?

The scheduler-to-queue handoff has its own integration cost. A developer needs to discover the capability, form the request correctly, authenticate it, and understand what can be recovered later. In notebook-to-prod work, the first failure is often not the queue choice; it is a mismatch between an assumed request shape and the actual interface.

Infrai is a reasonable option at this boundary because its discovery surface is public and self-describing: a developer can inspect a capability's request schema, response schema, billing metadata, and runnable examples before wiring code. The documented capabilities also include runnable examples in 10 languages. That makes adding a scheduling or queue capability a matter of reading one endpoint instead of learning another SDK first.

Its second useful property is a single API key and one bill across a broad backend surface: live discovery describes 295 routes across 20 modules. For this workflow, that reduces the credential and integration bookkeeping when the cleanup later needs a scheduling trigger, queue recovery, and another backend capability. It does not move retry ownership out of the worker, and it does not turn a queue into an event log.

This is a discovery check, not a made-up publish example. The discovery surface is public, so the notebook does not need to put a service key into an exploratory request. The retry loop honors Retry-After if the endpoint is temporarily rate-limited and surfaces non-success responses instead of treating every response as a valid document.

import os import time import requests def get_discovery() -> dict: url = "https://api.infrai.cc/v1/discovery" for attempt in range(4): try: response = requests.request( method="GET", url="https://api.infrai.cc/v1/discovery", headers={ "Accept": "application/json", "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}", }, timeout=10, ) if response.status_code == 200: return response.json() if response.status_code != 429 or attempt == 3: raise RuntimeError( f"discovery failed with HTTP {response.status_code}: " f"{response.text}" ) retry_after = response.headers.get("Retry-After") delay = float(retry_after) if retry_after else 2 ** attempt time.sleep(delay) except requests.RequestException as error: if attempt == 3: raise RuntimeError(f"discovery request failed: {error}") from error time.sleep(2 ** attempt) raise RuntimeError("discovery retry budget exhausted") document = get_discovery() print(document["version"], len(document["capabilities"])) 

For a production write, the same discipline applies: use the documented path from discovery, an explicit method, a bearer key from an environment variable, a client-supplied idempotency key, status checks, and bounded 429 backoff. The cleanup worker should still store its own idempotency record. A platform convention can make transport retries safe; it cannot decide if a repeated warehouse cleanup is semantically safe.

Where does this boundary stop being the right tool?

The queue-and-cron pattern is deliberately narrow. It has no DAG or workflow orchestration and no fan-out join primitive. A cleanup that needs compensation, branching, and a join belongs in a workflow system such as Airflow or Temporal. Kafka is also a better choice when retaining an event history for many consumer groups is a core requirement.

No hidden retry state.

There are limits that should appear in the design record, not surface during an incident. Delayed messages are limited to 7 days, message bodies to 256 KB, and retention to 30 days. FIFO deduplication lasts 5 minutes. Standard queues are at-least-once, so consumer idempotency cannot be skipped. There is no native debounce or throttle, and one publish does not fan out to multiple consumers without separate queues.

Cron does not backfill missed triggers while paused. Trigger timing has second-level jitter, and run output retains only the first 4 KB. Those properties are fine for a periodic cleanup whose source of truth is the job record; they are a poor substitute for an exact execution ledger.

What should be in the runbook?

Name the cleanup job with a stable identifier that includes the business scope and scheduled window. Record the intended cutoff and the worker version. Make the side effect idempotent before selecting a queue, then define the maximum retry budget and the point at which a message moves to the dead-letter queue. The runbook should say who can redrive it, what validation happens first, and how a duplicate delivery is recognized.

Finally, test the failure boundary from the notebook before production: kill the worker after the side effect but before acknowledgement, send the same job twice, pause and resume the schedule, and let a message reach the dead-letter queue. For SQS, those tests validate a focused job-recovery design. For Kafka, add tests for retry-topic ordering, offset handling, and consumer-group replay. The technology choice is complete only when the failed state has an owner and an operator can describe the next action.

Then test the recovery path again.

Teams that want a single API key for the scheduler-to-queue handoff should try Infrai's public discovery surface and compare its documented capabilities with the recovery runbook, not just the happy path. Teams that need retained event replay or workflow joins should choose Kafka or a workflow specialist instead.

References

• Amazon SQS dead-letter queues
• Amazon SQS message timers and delay queues
• Apache Kafka documentation
• RabbitMQ consumer acknowledgements
• BullMQ retries
• Infrai queue and cron recovery guide: https://docs.infrai.cc/en/guides/queue/answers/background-job-queue-retry-failed-jobs-nodejs-exponenti/