Imagine you have an endpoint which processes large CSV files. How it would behave under peak concurrency and why?

The Problem

Processing a CSV file can be a time + memory consuming task. Which means at peak concurrency your server will be overwhelmed with resource usages

Because by the time your server completes the first request, the second request will arrive and your server needs to start processing that as well concurrently. For which your server needs to spin up new threads or child processes. Since CSV processing is CPU bound task, a single threaded concurrent processing would not be doable in this case.

Thus more thread/processes will consume more resource and eventually your server will be most likely to crash

Here we can see results of a csv processing workload under stress. The CPU usage is higher, memory usage is around 256mb, the request rate is also lower with significantly high inflight requests

Then What Exactly Is The Solution?

The typical solution is to offload the workload as jobs in a queue and let async workers do the expensive work

In this approach we do not process all the processing concurrently but we make sure that the pending requests are not lost but persists in the queue waiting to be processed

This approach significantly reduces the CPU and memory usage of the server leading to less chances of crash with much higher request rate and no inflight requests.

But Why QUEUE? Why not DATABASE itself?

However there can still be a question that, Why exactly are we using a queue service like RMQ or SQS for persisting the tasks to be processed. Why not just store them in database and let the workers fetch the pending tasks from the database independently?

Although in outbox pattern we do use database for storing async tasks but the catch is databases do not guarantee to store the data in order during insertion and do not retrieve the data from the very top during data fetch. It does not follow the FIFO order pattern like RMQ or SQS and thus both read and write are more expensive in databases compared to queues

Queues use append only log structured data structures instead of tree or heap like data structures in database which makes it more efficient for storing and retrieving data in FIFO order compared to databases

So When Exactly We Should Use Async Processing Architecture?

We should offload the workload to async workers when the workload is expected to be time consuming or resource consuming.

It can be calling third party api where we are not sure how much time it will take for them to respond, and also whether or not if we will receive any response

Or it can also be a native workload but the amount of resource consumed is expected to be higher. For example processing an image or CSV file

The goal is to save our client facing server from being overwhelmed and withstand peak concurrent load

Hope that helps !!!