In October we moved our stock service behind a new internal name, stock.internal, in a Route 53 private hosted zone. The cutover plan had two parts. The platform team's Terraform would create the record pointing at the new load balancer, and the client teams would deploy configuration that switched their services to the new name. The change window was ten o'clock.
The client deploys went out at five to ten. Terraform created the record at one minute past. For the next fifteen minutes, eleven of our nineteen client services, in two of our three VPCs, failed every call to stock with name not found, while the other eight worked perfectly against the same record.
Each client deploy had run a readiness check that resolved stock.internal before the record existed. The VPC resolver answered that the name did not exist and, as resolvers are supposed to, it cached that answer. How long a resolver keeps a negative answer is set by the zone's SOA record: the smaller of the SOA record's own TTL and its minimum field. In Route 53's default SOA the TTL is nine hundred seconds, so the resolver kept saying the name did not exist for fifteen minutes after it did. The eight that worked ran in a VPC where nothing had asked for the name before it existed, so its resolver had nothing to remember.
Nothing was broken, and every resolver in the path was doing what the standard tells it to do. Our plan had simply never considered that asking a question early is also an answer that gets remembered.
Records for a cutover are now created at least a day ahead, pointing at the old target, and the cutover itself changes the value of a record that already exists. A positive answer is cached for the record's own TTL, which we lower to sixty seconds the day before, so the switch lands in a minute. We also lowered the TTL on our private zones' SOA records to sixty seconds, so a negative answer that slips through costs a minute rather than fifteen. The cutover checklist begins with a lookup of every new name from inside the network, and it must succeed before any client deploy starts.
The plan asked whether the record would exist by ten. It never asked who would look for it before then.
– Sergey Shinder