Every morning between six and eight our tracking poller asks four carriers about the status of every parcel in transit, around two hundred thousand of them. It is a Python service on a single large EC2 instance, with many workers, and until October it opened a new HTTPS connection for every request, which meant a new DNS lookup every time.

In October volumes went up and about one poll in fifty began failing with socket.gaierror, Temporary failure in name resolution. Our dashboards put the failures under carrier errors, and for two weeks we raised tickets with carriers whose APIs were fine. The resolver logs we could see showed nothing, and a lookup run by hand on the instance always worked.

The Amazon DNS resolver in a VPC accepts at most 1024 packets per second from each network interface. Anything above that is dropped before it reaches the resolver, without an error, so the client waits and eventually gives up. Each of our lookups was at least two packets, an A query and an AAAA query, and at the peak the poller was making well over five hundred lookups a second. The instance had one network interface.

The evidence was on the instance all along. The ENA network driver keeps a counter called linklocal_allowance_exceeded, which counts packets dropped because traffic to local services such as DNS, the instance metadata service and the time sync service went over that limit. ethtool -S on the interface showed it at several million and climbing every morning.

We fixed the cause first. The poller now keeps one HTTP session per carrier and reuses connections, which cut lookups by more than ninety percent on its own. The instance also runs a local caching resolver, dnsmasq, listening on the loopback address, and the poller uses it, so a hostname is asked of the VPC resolver once per TTL rather than once per request. Node exporter's ethtool collector is enabled on all our instances, and we alert on any increase in that counter. Resolution failures are now their own category on the dashboard, separate from anything a carrier did.

We had a limit on the instance's most basic dependency and no way to see it from the application. The packets were dropped politely, and politely looked exactly like a slow carrier.

– Sergey Shinder