Most retry policies are tested in one of two ways: not at all, or with a mocked HttpMessageHandler that throws on cue. The mock proves that Polly calls your delegate again. It doesn't show what your app does when a real server is slow, answers 503 twice, then takes three seconds to say yes.

Here's a way to see that in about five minutes, with no change to your app except the URL.

The setup: a server that fails on purpose

I built Curler for this. You create a bin, give it a script of responses, and point your app at the bin's URL. Every call your app makes shows up with the response Curler sent, when it was sent and how long it took.

The script for this post:

{ "steps": [ { "status": 503, "times": 2 }, { "status": 200, "delayMs": 3000, "contentType": "application/json", "body": "{\"paid\":true}" } ] }

Two 503s, then a 200 that takes three seconds. The last step repeats, so every call after the second gets the slow success. This is a payment provider on a bad day.

To use it: open curler.dev, click Create a bin, click Edit in the Responses bar, paste the script and save. Copy the capture URL from the top bar.

The app

A console app with Polly v8. It retries server errors and timeouts three times, with exponential backoff, and gives each attempt 2 seconds:

using System.Diagnostics; using Polly; using Polly.Retry; using Polly.Timeout; var url = args[0]; var attemptTimeout = TimeSpan.FromSeconds(double.Parse(args[1])); var clock = Stopwatch.StartNew(); var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>() .AddRetry(new RetryStrategyOptions<HttpResponseMessage> { ShouldHandle = new PredicateBuilder<HttpResponseMessage>() .Handle<HttpRequestException>() .Handle<TimeoutRejectedException>() .HandleResult(response => (int)response.StatusCode >= 500), MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential, Delay = TimeSpan.FromSeconds(1), OnRetry = retry => { var why = retry.Outcome.Result?.StatusCode.ToString() ?? retry.Outcome.Exception?.GetType().Name; Console.WriteLine($"{clock.Elapsed.TotalSeconds,5:0.0}s attempt {retry.AttemptNumber + 1} failed ({why}), retrying in {retry.RetryDelay.TotalSeconds:0.0}s"); return default; }, }) .AddTimeout(attemptTimeout) // added after the retry, so it applies to each attempt .Build(); using var http = new HttpClient(); try { var response = await pipeline.ExecuteAsync( async token => await http.PostAsync(url, new StringContent("{\"orderId\":42}"), token)); Console.WriteLine($"{clock.Elapsed.TotalSeconds,5:0.0}s done: {(int)response.StatusCode} {await response.Content.ReadAsStringAsync()}"); } catch (Exception exception) { Console.WriteLine($"{clock.Elapsed.TotalSeconds,5:0.0}s gave up: {exception.GetType().Name}"); }

The order matters. In Polly v8 the first strategy you add is the outermost, so the retry wraps the timeout, and the timeout applies to each attempt rather than to the whole call.

Run 1: a 2-second timeout per attempt 

$ dotnet run -- https://api.curler.dev/h/<your-bin> 2 0.8s attempt 1 failed (ServiceUnavailable), retrying in 1.0s 1.9s attempt 2 failed (ServiceUnavailable), retrying in 2.0s 5.9s attempt 3 failed (TimeoutRejectedException), retrying in 4.0s 11.9s gave up: TimeoutRejectedException

The server was ready to answer 200 from the third call on. The app never saw it: each attempt gave up after 2 seconds, while the answer needed 3. After 12 seconds and four calls, the order failed.

Curler's bin shows the same story from the server's side. The two slow successes are marked Abandoned: your app hung up before Curler could answer.

Curler's bin: two 503s, two abandoned 200s, then the second run's 503s and the 200 that completed

This is the bug a mock never finds: the policy works, but its numbers don't fit the service.

Run 2: a 5-second timeout per attempt

Click Reset counter in Curler, so the script starts again from step 1, then run with 5 seconds:

$ dotnet run -- https://api.curler.dev/h/<your-bin> 5 0.3s attempt 1 failed (ServiceUnavailable), retrying in 1.0s 1.4s attempt 2 failed (ServiceUnavailable), retrying in 2.0s 6.5s done: 200 {"paid":true}

Two quick failures, a wait, and the slow success gets through: 1.4 s + 2.0 s of backoff + 3 s for the answer = 6.4 s.

What to try next

Change the script and watch what your policy does:

  • A 429 with Retry-After: 5. Does your retry wait the 5 seconds the server asked for, or its own backoff?
  • A 200 after 10 seconds. With AddStandardResilienceHandler from Microsoft.Extensions.Http.Resilience, each attempt times out after 10 seconds by default.
  • A redirect. Does HttpClient follow it, and does the retry see the first response or the last?

Good to know

  • Curler is free: a bin lives 24 hours and keeps its newest 100 requests.
  • Curler's server sleeps when nobody uses it, and the first call after a quiet spell can take about 10 seconds. Open your bin's page before you run the app: it wakes the server, so your first attempt isn't the one that pays for the start.
  • The script accepts JSON and plain-text bodies only, and redirects stay inside Curler, so a bin can't be used to host a phishing page.

I'm the developer of Curler, and I'd like to hear what you'd test with it: feedback@curler.dev.