Cloudflare is up to its optimization shenanigans again. Not too long after finding 100 TB of memory savings between the DNS cache's couch cushions, and another 100 TB of saved RAM by improving its task-server routing, the company's engineers are back with the hat trick. This time, they managed to halve DNS zone rebuild times and cut that procedure's memory usage by 85% across its network.
A "DNS zone" is the collection of records, settings, and metadata for a given domain like tomshardware.com. The records are the main part, containing IP addresses, hostnames, or text lines including website and mail server IPs, subdomains, domain redirections, and spam-control or certificate info. If you're a Cloudflare customer, when you update a record pointing your website to a new IP, or change one of the domain's settings, the service pushes this information onto its many thousands of DNS servers.
This sounds easy enough until you realize that Cloudflare's DNS fleet is huge, and that the company now gets 800 zone changes per second, up from 250 four years ago. All those changes add up to enormous amounts of RAM and network traffic. Consistency across the entire fleet must be guaranteed, too, as you can't have one DNS server saying your website is at one place, while another server says it's elsewhere — except during actual transitions, of course. Furthermore, some domains hosted at Cloudflare have over 20 million records, making them exceedingly difficult to build in memory.
The easy solution, then, is to only update the data that's needed, and that's precisely what the company did for domain records back in 2022. Updating records is simple: you just write the new value in and propagate it. Altering settings, though, is way trickier. For example, toggling Universal SSL for a domain can trigger a cascade of record updates.
Some metadata keys are shared across multiple zones, necessitating an update across all of them. Change events can come in from many Cloudflare services, too, and often didn't have a reason attached, so it was hard to know if other subsequent changes were necessary. Consequentially, setting updates required a full build for safety.
The engineers then carefully laid out a plan to optimize this process. First, they tweaked their internal software so that zone change events include a reason and additional information, so that said event can be properly classified. With that info in hand, specific setting changes can use smaller and shorter builds among given types.
For example, apex and metadata changes (like UniversalSSL) comprise one. Likewise, changing a domain's nameserver or wildcards also triggers a smaller procedure. Full builds are only used in cases where there is no partial build mechanism. Finally, domains with a large number of records are now updated piecemeal. Cloudflare saw a 85% drop in RAM usage for Zone Builder servers, from 50-79 GB to down to close to 7 GB, mostly due to these changes alone.
The second optimization pass was combining two formerly separate planning and implementation schedulers into just the one, and improving how it operates. The new scheduler detects if several changes apply to the same zone and combines them. Cloudflare uses Apache Kafka for event messaging, and the team changed how the scheduler acts when a message triggers a slow build.
Since Kafka only lets messages advance in sequence, the scheduler waited for that job to finish and didn't process requests that arrived in the meantime. Now, it'll acknowledge and run incoming smaller jobs arriving in the meantime. Before, there was always a steady long queue of requests waiting to be processed, as requests piled up and were completed in bursts.
Now, they're fulfilled very shortly after being issued, and in a consistent cadence. This significantly improved concurrency capacity, so the team chose to tighten the queue processing intervals. With all the improvements, the median time for a build dropped from 7.7 seconds to 3.3 seconds, too.
