Our storefront moved behind CloudFront in the spring. Product pages and the basket go through cache behaviours with the CachingDisabled policy, so every request reaches the application. Everything else falls through to the default behaviour, built for static assets, which uses the CachingOptimized policy: it follows the origin's caching headers and, when the origin sends none, keeps a response for a day.
Our external uptime check requests /health every minute from three regions and pages if two of them fail. The health endpoint came from a framework template. It returns 200 with a short JSON body, it sends no Cache-Control header at all, and its path matched none of our page behaviours.
On a Sunday in October a change to our load balancer's listener sent every request to an empty target group, and the storefront returned 503 for forty two minutes. We found out from a customer on the phone. The uptime check stayed green the whole time.
CloudFront had cached /health. With no instruction from the origin it used the policy's default TTL of twenty four hours, so each edge location our monitor reached was holding a 200 fetched the day before and served it without asking the origin anything. The responses carried an Age header of about seventy thousand seconds and X-Cache: Hit from cloudfront. Our monitor recorded the status code and the response time, and both were excellent.
For eight months the check had measured whether three CloudFront edges could serve a small file from memory. It had never failed, and we had taken that as a good sign.
/health now sends Cache-Control: no-store and has its own cache behaviour with the CachingDisabled policy. The body includes the instance name and the server's clock, and the monitor fails if that time is more than a minute old or if the response carries an Age header. A second check fetches a real product page, which has to touch the application and the database. We then looked for other paths that reached the default behaviour without caching headers and found two: a stock lookup endpoint and the page showing our delivery cut off times. Both had been serving answers up to a day old.
A check that goes through a cache tests the cache. If it is meant to test the origin, it has to prove it got there.
– Sergey Shinder
