Distributed systems are hard to test because partial failures overlap with live traffic and the same fault can produce different execution traces on different runs. Unmesh Joshi describes how deterministic simulation testing addresses this by replacing real clocks, networks, and storage with simulated components driven by a single-threaded execution loop. This allows simulating both hard failures and routine slowdowns—such as network and disk latency spikes—and more importantly makes every execution reproducible. This article explains the technique using a quorum key-value store as a running example, with code from Tickloom, a Java implementation of the approach.

more…