When a parcel leaves one of our warehouses, the warehouse system puts a despatch message on an SQS queue with the order number, the carrier and the tracking number. Twice every evening, at half past five and half past nine, a job reads the queue, sends the tracking numbers to our largest marketplace in one feed, and deletes the messages it sent. The marketplace counts an order as despatched late if we confirm it after its despatch date, and it warns sellers whose late rate passes four percent.

In September we rewrote the job, from an old Java service into a short Python script using boto3. It loops on receive_message with MaxNumberOfMessages set to ten, and stops when a call returns no messages. In October our late despatch rate went from under one percent to six, and the marketplace sent us its warning.

Every one of those late orders had left the warehouse on time. Its message had still been in the queue after the half past nine run, and it went out in the next day's feed.

SQS keeps a queue's messages across many servers. A receive with no wait time is a short poll, and the documentation says what that means. SQS queries only a subset of its servers, chosen at random, and answers at once. With fewer than a thousand messages in the queue you may get fewer messages than you asked for, or none at all, and you should simply ask again. Our queue holds a few hundred messages at half past nine. An empty answer meant that the servers SQS had asked that time had nothing for us, and our loop took it as the end of the queue. On a typical night it left between thirty and ninety messages behind.

The Java service had set WaitTimeSeconds to twenty on every receive, which is long polling. A long poll asks all the servers, and returns empty only when no message has arrived by the end of the wait. boto3 sends no wait time unless you give it one, and then the queue's own ReceiveMessageWaitTimeSeconds applies, which is zero by default. Nobody had copied the setting across, because nobody knew it was doing anything.

The queue's default wait time is now twenty seconds, so any client that forgets still gets long polling, and the script sets it as well. The job stops after two empty long polls in a row, not after one. At eleven, an hour before the cut off, a check compares the parcels the warehouse system despatched that day with the orders confirmed to the marketplace and sends any difference straight away. And an alarm on the queue's ApproximateAgeOfOldestMessage fires when it passes three hours.

An empty answer from a short poll was a statement about the servers SQS had asked. We had read it as a statement about the queue.

– Sergey Shinder