Our drivers' app downloads the day's route manifest from our API at the start of each shift, a JSON document of about two megabytes. nginx sits in front of the API, and our latency dashboard and alert were built from its access log, using $request_time. The alert fired when the ninety fifth percentile went above three seconds for ten minutes.
From the day the bigger manifests went live in September it fired every weekday between six and seven. We spent three weeks on the API. We added an index, cached the depot lookup, and doubled the replicas. The alert kept firing at the same time with about the same number.
$request_time is the time from nginx reading the first bytes of the request from the client until it has written the last byte of the response and logs the line. It includes everything the client contributes. With proxy buffering on, which is the default, nginx takes the whole response from the API into its buffers quickly and then feeds it to the client at whatever speed the client manages. A driver in a yard on 4G took six or eight seconds to receive two megabytes. The API had finished long before.
The number we wanted was $upstream_response_time, the time nginx spent receiving the response from the API. It was not in our log format. When we added it, the API's ninety fifth percentile at seven in the morning was eighty milliseconds, and had been all along. One detail matters for parsing: if nginx tries more than one upstream server for a request, the variable holds several values separated by commas, so our parser now sums them rather than reading the first.
The latency alert now uses $upstream_response_time. $request_time stays on a separate panel labelled as delivery time to the client, because it is a real cost to drivers, just not one replicas can fix. And there was a fix for it. The manifests were going out uncompressed, because nginx's gzip only compresses text/html unless you list other types in gzip_types. We added application/json, and the manifest went from two megabytes to about two hundred kilobytes. Delivery time at the yards fell below a second.
The replicas went back to what they were. The alert had been right that something was slow. It just could not tell us whose side of the connection it was on.
– Sergey Shinder
