Coordinated Omission in Go: Why Your Load Tests Underestimate Latency

Your load test reports a p99 latency of 120 ms.

Everything looks healthy.

You increase traffic, run the test again, and the results still look reasonable.

But after deploying the same service to production, users start reporting requests that take several seconds.

What went wrong?

One possibility is that your load test never measured the worst part of the system's behavior.

This problem is known as coordinated omission.

It happens when a load generator unintentionally reduces its request rate while the system under test is slowing down. The slowest periods receive fewer measurements precisely because the generator is waiting for previous requests to finish.

The result is a misleading latency distribution.

This is particularly important when evaluating backend services where occasional pauses, dependency slowdowns, garbage collection, and traffic bursts can significantly affect tail latency.

In this article, we'll explore:

• What coordinated omission is and why it happens
• Closed-loop versus open-loop load testing
• Why average latency and percentiles can be misleading
• How measurement intervals affect latency distributions
• Building a simple open-loop load generator in Go
• Handling generator saturation and dropped requests
• Using HDR Histogram and latency correction
• Designing more reliable performance tests

The objective is simple: measure the latency users would actually experience, not just the latency your load generator happens to observe.

1. What Is Coordinated Omission?

Imagine a service normally processes requests in 20 milliseconds.

Your load generator sends one request, waits for the response, and then sends another.

Everything looks fine.

Now imagine the service experiences a temporary 2-second pause.

During those two seconds, the load generator is also waiting for the previous request to finish.

It doesn't send new requests.

When the service recovers, the load generator resumes sending requests.

Your test may record only one extremely slow request, even though a real production system could have received hundreds or thousands of requests during that same period.

This is coordinated omission.

The load generator and the system under test are effectively slowing down together.

Consider this example:

Normal operation: Request 1 -> 20 ms Request 2 -> 20 ms Request 3 -> 20 ms Request 4 -> 20 ms Service slowdown: Request 5 -> 2,000 ms After recovery: Request 6 -> 20 ms Request 7 -> 20 ms Request 8 -> 20 ms 

A closed-loop load generator waits for Request 5 to finish before sending Request 6.

But in production, incoming traffic doesn't necessarily stop just because one request is taking longer.

During the 2-second pause, new requests may continue arriving and accumulate.

The load test therefore misses many of the requests that would have experienced the slowdown.

The problem isn't necessarily that the measured request duration is wrong. The problem is that the test failed to generate and measure requests that should have existed.

2. Closed-Loop vs. Open-Loop Load Testing

Understanding coordinated omission starts with understanding how load is generated.

There are two common approaches.

Closed-loop load generation

In a closed-loop model, the client waits for a response before sending its next request.

Client | v Send Request | v Wait for Response | v Send Next Request | v Wait for Response 

For example:

for { start := time.Now() err := makeRequest(ctx) duration := time.Since(start) recordLatency(duration) if err != nil { recordError(err) } } 

This is simple and useful for certain workloads.

However, the request rate depends on response time.

If each request takes 20 ms, one client can generate approximately 50 requests per second.

If the service slows down to 200 ms, that same client generates only approximately 5 requests per second.

The offered load decreases as the service becomes slower.

That's the core problem.

Closed-loop testing is not inherently incorrect. It models workloads where clients genuinely wait for one operation to finish before starting another.

But it can be misleading when used to represent independently arriving production traffic.

Open-loop load generation

In an open-loop model, requests are scheduled independently of previous responses.

For example:

Scheduled arrivals: 0 ms -> Request 1 10 ms -> Request 2 20 ms -> Request 3 30 ms -> Request 4 40 ms -> Request 5 50 ms -> Request 6 

The arrival schedule continues even when earlier requests haven't completed.

Load Generator | +---- Request 1 -----------+ | | +---- Request 2 ------+ | | | | +---- Request 3 --+ | | | | | | v v v v Service Under Load 

This better represents systems where incoming traffic is independent of response completion.

For example:

• Public HTTP APIs
• Event ingestion services
• Message consumers
• Webhook receivers
• Market data ingestion
• Background processing pipelines

The important difference is that the arrival rate remains independent of the system's response time.

This makes open-loop testing particularly useful for identifying saturation behavior and tail-latency problems.

3. A Simple Example of Measurement Bias

Let's compare two load generators.

Assume:

• Normal response time: 20 ms
• Temporary service pause: 2 seconds
• Target arrival rate: 100 requests/second

A closed-loop generator with one outstanding request behaves differently from an open-loop generator.

Closed-loop behavior

The generator sends a request and waits.

When the service pauses, the next request cannot be sent until the previous one completes.

The generator may observe:

Request 1: 20 ms Request 2: 20 ms Request 3: 20 ms Request 4: 2,000 ms Request 5: 20 ms Request 6: 20 ms 

Only a small number of requests represent the slowdown.

Open-loop behavior

At 100 requests per second, new requests are scheduled every 10 milliseconds.

During a 2-second pause, approximately 200 requests would arrive.

Those requests may experience substantial waiting time, depending on the system's queueing and processing behavior.

A simplified view:

Scheduled Request Completion -------------------------------- 0 ms 2,000 ms 10 ms 2,010 ms 20 ms 2,020 ms 30 ms 2,030 ms ... 1,990 ms 3,990 ms 

The actual completion pattern depends on service capacity and scheduling, but the key point is that many requests are affected.

A closed-loop test may report one slow request.

An open-loop test can reveal a much larger group of requests experiencing elevated latency.

This is why the two approaches can produce dramatically different p99 measurements.

4. Why Percentiles Can Be Misleading

Performance reports often focus on:

• Average latency
• p50
• p95
• p99
• p99.9

These metrics are useful, but only if the underlying sample represents the workload being evaluated.

Consider this hypothetical result:

The numbers are illustrative, not a universal outcome.

Notice something interesting: p99 might look relatively similar, even though the average and p95 are dramatically different.

Why?

Because the open-loop test captures more requests affected by the slowdown.

The important lesson is that a percentile is not meaningful independently of its sampling process.

A p99 calculated from a biased sample is still a biased p99.

The importance of the measurement window

Suppose your service experiences a 2-second pause once every minute.

If your load generator stops sending requests during that pause, your measurement window may contain very few samples from the affected period.

Even a large total request count doesn't guarantee representative sampling.

Always ask:

• How were requests scheduled?
• Did the generator wait for previous responses?
• Were requests skipped when the system slowed down?
• Was latency measured from scheduled arrival or actual transmission?
• Did the load generator itself become saturated?
• Were errors and dropped requests included in the report?

These questions are just as important as the latency numbers themselves.

5. Scheduled Time vs. Actual Start Time

One of the most important details in an open-loop load test is the starting point of latency measurement.

Consider a request scheduled for 100 ms.

But because the load generator is busy, it actually starts sending the request at 180 ms.

There are now two possible measurements.

Actual request duration:

Response time - Actual start time 

This measures the time spent after the request begins transmission.

Scheduled end-to-end latency:

Response time - Scheduled arrival time 

This includes the delay between the intended arrival and the actual request start.

For example:

Scheduled arrival: 100 ms Actual start: 180 ms Response: 250 ms Actual request duration: 70 ms Scheduled end-to-end latency: 150 ms 

If we record only 70 ms, we hide 80 ms of delay caused by the load generator.

If the goal is to evaluate the system from the perspective of an independently arriving request, the scheduled arrival time is the more meaningful reference point.

However, there is an important distinction:

• If the generator is intentionally queueing requests before transmission, scheduled-to-response latency captures that queueing delay.
• If the generator cannot schedule or execute requests because it is saturated, the test may no longer represent the intended workload.

You should record generator delay separately instead of hiding it inside one latency number.

A useful set of measurements includes:

Scheduled arrival Actual request start Response completion 

From these timestamps, we can calculate:

Generator scheduling delay = Actual start - Scheduled arrival Request duration = Response completion - Actual start Scheduled end-to-end latency = Response completion - Scheduled arrival 

This makes it easier to distinguish system performance from load-generator limitations.

6. Building an Open-Loop Load Generator in Go

Let's implement a small open-loop load generator.

The objective is to schedule requests at a fixed interval without waiting for previous requests to complete.

We'll also enforce a maximum number of concurrent requests.

Most importantly, when the generator cannot launch a scheduled request, we'll count it as dropped instead of silently skipping it.

package main import ( "context" "errors" "sync" "sync/atomic" "time" ) var ErrConcurrencyLimit = errors.New("concurrency limit reached") type RequestFunc func(context.Context) error type Stats struct { Scheduled atomic.Int64 Started atomic.Int64 Completed atomic.Int64 Dropped atomic.Int64 Failed atomic.Int64 } type Result struct { ScheduledAt time.Time StartedAt time.Time FinishedAt time.Time Err error } type LoadGenerator struct { interval time.Duration maxInFlight int request RequestFunc stats Stats results chan Result } func NewLoadGenerator( interval time.Duration, maxInFlight int, request RequestFunc, ) (*LoadGenerator, error) { if interval <= 0 { return nil, errors.New("interval must be greater than zero") } if maxInFlight <= 0 { return nil, errors.New("maxInFlight must be greater than zero") } if request == nil { return nil, errors.New("request function cannot be nil") } return &LoadGenerator{ interval: interval, maxInFlight: maxInFlight, request: request, results: make(chan Result, maxInFlight), }, nil } func (g *LoadGenerator) Run( ctx context.Context, duration time.Duration, ) error { if duration <= 0 { return errors.New("duration must be greater than zero") } startedAt := time.Now() deadline := startedAt.Add(duration) semaphore := make(chan struct{}, g.maxInFlight) var wg sync.WaitGroup defer func() { wg.Wait() close(g.results) }() for scheduledAt := startedAt; scheduledAt.Before(deadline); scheduledAt = scheduledAt.Add(g.interval) { wait := time.Until(scheduledAt) if wait > 0 { timer := time.NewTimer(wait) select { case <-ctx.Done(): if !timer.Stop() { <-timer.C } return ctx.Err() case <-timer.C: } } if err := ctx.Err(); err != nil { return err } g.stats.Scheduled.Add(1) select { case semaphore <- struct{}{}: wg.Add(1) go func(scheduled time.Time) { defer wg.Done() defer func() { <-semaphore }() actualStart := time.Now() g.stats.Started.Add(1) err := g.request(ctx) finished := time.Now() if err != nil { g.stats.Failed.Add(1) } g.stats.Completed.Add(1) g.results <- Result{ ScheduledAt: scheduled, StartedAt: actualStart, FinishedAt: finished, Err: err, } }(scheduledAt) default: g.stats.Dropped.Add(1) } } return nil } 

This example demonstrates several important ideas:

• Requests are scheduled independently of previous responses.
• The generator uses a fixed arrival interval.
• Maximum in-flight concurrency is bounded.
• Requests that cannot be started are counted as dropped.
• Scheduled, actual start, and completion timestamps are preserved.
• Request execution is measured separately from scheduling delay.

The implementation is intentionally a small demonstration rather than a complete benchmarking framework.

Understanding the request interval

Suppose we want to generate 100 requests per second.

The interval is:

interval := 10 * time.Millisecond 

Because:

1 second / 100 requests = 10 milliseconds 

We can initialize the generator:

generator, err := NewLoadGenerator( 10*time.Millisecond, 200, func(ctx context.Context) error { return makeRequest(ctx) }, ) if err != nil { panic(err) } ctx, cancel := context.WithTimeout( context.Background(), 30*time.Second, ) defer cancel() if err := generator.Run(ctx, 30*time.Second); err != nil { // Handle cancellation or test failure. } 

With this configuration:

• Target arrival rate: 100 requests/second
• Maximum in-flight requests: 200
• Test duration: 30 seconds

The maximum concurrency is a safety boundary for the generator.

It is not a guarantee that the target arrival rate will be achieved.

If requests take too long, the concurrency limit may be reached and scheduled requests will be dropped.

That's an important result, not something to hide.

A limitation of this example

The example uses one scheduling goroutine and launches a goroutine for each accepted request.

It is suitable for demonstrating open-loop scheduling, but a production-grade load generator needs additional protections:

• A dedicated result collector
• A non-blocking or carefully sized result pipeline
• Explicit generator CPU and memory monitoring
• Better cancellation and shutdown behavior
• Histograms for latency measurements
• Accurate reporting of missed schedules
• Protection against excessive goroutine creation
• Separate reporting of generator delay and service latency

In particular, if the result channel fills and no consumer is reading from it, request goroutines can block while reporting results.

A real implementation should run a dedicated collector concurrently with request generation.

The test harness itself must never become the hidden bottleneck.

7. Coordinated Omission Correction

Open-loop load generation is one way to reduce coordinated omission.

Another approach is to correct latency measurements for the expected arrival interval.

This technique is commonly associated with HDR Histogram.

Imagine:

Observed request latency: 1,000 ms Expected arrival interval: 10 ms 

If the generator was waiting for a previous response, many expected requests may not have been recorded.

A coordinated-omission-aware histogram can account for those missing observations by recording additional latency values at intervals related to the expected request cadence.

Conceptually:

Observed latency: 1,000 ms Expected interval: 10 ms Additional observations represent: 990 ms 980 ms 970 ms ... 10 ms 

The exact recording behavior depends on the histogram API and correction method.

The important idea is that a long pause should not be represented by only one sample if the test model implies that many requests would have arrived during that period.

HDR Histogram in Go

HDR Histogram is useful for recording latency distributions with a wide dynamic range.

A Go implementation is available through:

HDR Histogram Go

A basic example:

package main import ( "fmt" "time" hdrhistogram "github.com/HdrHistogram/hdrhistogram-go" ) func main() { histogram := hdrhistogram.New( 1, 60_000_000, 3, ) latencies := []time.Duration{ 10 * time.Millisecond, 20 * time.Millisecond, 30 * time.Millisecond, 100 * time.Millisecond, 2 * time.Second, } for _, latency := range latencies { histogram.RecordValue( latency.Microseconds(), ) } fmt.Printf( "p50: %d µs\n", histogram.ValueAtQuantile(50), ) fmt.Printf( "p95: %d µs\n", histogram.ValueAtQuantile(95), ) fmt.Printf( "p99: %d µs\n", histogram.ValueAtQuantile(99), ) } 

This example records observed values.

It does not automatically correct coordinated omission.

Correction requires the expected arrival interval and an appropriate recording strategy.

For example, when using a histogram API that supports it, a corrected recording operation can account for the expected interval:

histogram.RecordCorrectedValue( latency.Microseconds(), expectedInterval.Microseconds(), ) 

Always check the library version and API you're using before integrating this into a test harness.

Correction is not a replacement for a good load model

There is an important limitation.

Coordinated omission correction estimates observations that were not recorded based on an assumed arrival interval.

It cannot reconstruct every detail of a real production traffic pattern.

If production traffic is bursty, user-driven, or has complex dependencies, a constant-rate correction may not accurately represent it.

Use correction as a measurement technique, not as a substitute for realistic workload generation.

8. The Load Generator Can Become the Bottleneck

There's another problem worth discussing.

An open-loop generator is only useful if it can actually generate the intended workload.

Imagine the target is 50,000 requests per second.

But the load generator runs out of CPU, network bandwidth, file descriptors, or available connections.

Your test may report low throughput and high latency.

Is the service saturated?

Or is the generator saturated?

Without separate measurements, it's difficult to know.

A load generator should monitor:

For distributed load testing, multiple generator instances may be required.

But adding more generators introduces another consideration: their clocks and scheduling behavior should be sufficiently consistent for the measurements you're trying to make.

Don't assume that a test generating 50,000 requests per second is actually doing so just because that was the configured target.

Measure the achieved arrival rate.

9. Designing a Reliable Load Test

A good load test should answer a specific question.

Instead of simply asking:

"How many requests per second can this service handle?"

Ask:

"What happens to latency and error rates when the service receives 2,000 independently arriving requests per second?"

A practical testing strategy can include several phases.

Phase 1: Baseline

Run the service under low traffic.

Measure:

• Average latency
• p50, p95, p99
• Throughput
• Error rate
• Resource utilization

This establishes a baseline.

Phase 2: Gradual ramp-up

Increase the arrival rate gradually.

For example:

500 req/s 1,000 req/s 1,500 req/s 2,000 req/s 2,500 req/s 

Observe where latency begins increasing disproportionately.

Phase 3: Sustained load

Hold the target arrival rate for a meaningful period.

This helps identify:

• Queue accumulation
• Memory growth
• Connection pool saturation
• Garbage collection effects
• Resource exhaustion

Phase 4: Burst traffic

Send short bursts above the normal arrival rate.

Observe how quickly the system recovers.

Phase 5: Dependency slowdown

Introduce latency into a downstream dependency.

Measure whether the application maintains predictable behavior.

Phase 6: Recovery

Return the system to normal traffic.

Check whether queue depth, latency, and resource usage return to baseline.

The recovery phase is important.

A service that survives a traffic spike but takes several minutes to recover may still have a serious resilience problem.

10. A Practical Checklist

Before trusting a performance test, ask:

• Does the load model represent the real production traffic pattern?
• Is the arrival rate independent of response completion where appropriate?
• Are requests scheduled at known intervals?
• Is latency measured from scheduled arrival time when required?
• Are generator scheduling delays recorded separately?
• Are dropped or missed requests visible?
• Is the achieved arrival rate close to the configured target?
• Is the load generator itself monitored?
• Are p95, p99, and p99.9 calculated from representative samples?
• Is coordinated omission correction appropriate for the workload?
• Have we tested both steady-state and burst traffic?
• Do we understand how the system behaves during recovery?

If you cannot answer these questions, your latency report may be less reliable than it looks.

Conclusion

Performance testing isn't just about sending requests and measuring how long they take.

It's about generating the right workload and ensuring that the measurement process doesn't hide the behavior we're trying to understand.

Coordinated omission is a good example of how a seemingly reasonable test can produce misleading results.

A closed-loop load generator may reduce its request rate when the service slows down, making the system appear healthier than it would under independently arriving production traffic.

Open-loop scheduling, accurate timestamps, generator monitoring, and appropriate histogram techniques help address this problem.

For Go engineers, the implementation details matter:

• Schedule arrivals independently of previous responses.
• Measure scheduled time and actual start time separately.
• Bound concurrency without silently hiding dropped work.
• Monitor the load generator as carefully as the service.
• Use latency histograms that reflect the workload being tested.

A performance test should reveal the system's limitations, not adapt to them.

If your load generator slows down whenever your service slows down, you may be measuring the generator's patience rather than your system's performance.

References

• Gil Tene — How NOT to Measure Latency
• HDR Histogram — Coordinated Omission
• HDR Histogram Go
• wrk2 — A Constant Throughput, Correct Latency Recording Fork of wrk
• Go Documentation — time Package
• Go Documentation — Context Package