If you've spent enough time on incident calls, you've probably seen this one. A dependency slows down. It isn't down, just slow. p99 goes from under 100ms to a couple of seconds, and around minute five someone says "looks like it's recovering." It isn't.
What's happening underneath is almost boring. Callers start timing out, and because every sensible service has a retry policy, they send the same requests again. The slow service is now handling its normal traffic plus a second copy of a good chunk of it. That makes it slower, which causes more timeouts, which causes more retries. By the time the original cause has gone away (a long GC pause, a heavy query, a bad deploy that got rolled back), the service is buried under load that only exists because it was slow for a minute.
Retries are one of the first things we add to make a system more reliable. In my experience they're also one of the most common reasons a small problem turns into a long one.
The retry doesn't know why the call failed
Retries exist for transient failures, and they handle those well. A connection gets reset, a pod is mid-restart, a packet goes missing. Try again 200ms later and it works. Nobody notices.
The problem is that from the caller's side, a transient failure and an overloaded dependency often look the same. You get a timeout or a 503 either way. In the first case the retry costs almost nothing. In the second, you're sending more work to a service that is struggling because it already has too much, and every other caller is doing the same thing at roughly the same moment.
You can make a retry policy smarter about error types, and you should. But the caller still rarely has enough information to know whether another attempt will help or make things worse.
Retries multiply
This is the part that surprises people, mostly because nobody sees it in code review.
Take a fairly ordinary chain. An API service calls an order service, which calls a payment service, which calls an external payment gateway. Each team has done the reasonable thing and configured three retries, so up to four attempts per call.
Then the gateway gets slow. The payment service tries it up to four times. The order service sees the payment service time out and retries that up to four times, and each of those can turn into four gateway calls. The API service does the same one level up. Four times four times four: a single checkout can become as many as 64 requests at the gateway.
No one decided on 64. Each "3 retries" looked fine in its own pull request. You only find out the real number when the gateway has a bad afternoon, which is why retry settings need to be looked at across the whole request path rather than one service at a time.
Why the system stays down
A spike on its own is usually survivable. What drags these incidents out is that the system doesn't bounce back once the original cause is gone.
When retries push load past what the service can handle, latency goes up, timeouts go up, retries go up, and load stays above capacity. The loop keeps itself running. There's a name for this, metastable failure: a system with a healthy state and an overloaded state that can get stuck in the second one after a fairly small nudge.
Getting out usually means someone does something drastic. Shed traffic at the edge, scale the callers down, turn off a feature, or switch off retries with a flag, if you were lucky enough to have one. That alone is a good reason to build the flag before you need it.
Where retry configs usually go wrong
Looking back at incidents like this, the causes tend to be the same handful of things.
Retries with no delay, so the second attempt lands while nothing has changed. Retries configured at every layer, which is the multiplication problem above. Fixed backoff, where every client waits exactly one second and then everyone hits the service together, in waves.
Then there are retries that were never going to work. A 400 validation error won't succeed on attempt three, and neither will a 401 or 403. Same for business-rule rejections. And retries that start after the original caller has already given up are doing work nobody will read.
The one that actually costs money is retrying calls that aren't idempotent. A timeout doesn't tell you the request failed. It tells you that you don't know. If that request was a card charge, retrying it blind is how a customer ends up paying twice.
What I'd do instead
Retry in one place
Pick the layer closest to the failing dependency and retry there. Everything above it should fail fast and hand the error back up. If your team owns the whole chain, this alone removes most of the amplification, and it's usually a config change rather than a rewrite.
Back off, add jitter, and be picky about what you retry
Space retries out, randomize the wait so clients don't line up, and write down which failures are actually worth a second attempt. With Resilience4j it looks something like this:
@Configuration
public class PaymentGatewayResilienceConfig {
@Bean
public Retry paymentGatewayRetry() {
RetryConfig config = RetryConfig.custom()
.maxAttempts(3) // first call + 2 retries
.intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
Duration.ofMillis(200), 2.0, 0.5))
.retryOnException(PaymentGatewayResilienceConfig::isRetryable)
.build();
return Retry.of("paymentGateway", config);
}
static boolean isRetryable(Throwable t) {
Throwable cause = (t instanceof ResourceAccessException && t.getCause() != null)
? t.getCause() : t;
if (cause instanceof ConnectException) return true;
if (cause instanceof SocketTimeoutException) return true;
if (cause instanceof HttpStatusCodeException e) {
int status = e.getStatusCode().value();
return status == 429 || status == 502 || status == 503 || status == 504;
}
return false;
}
}
Two things worth pointing out. RestTemplate wraps I/O errors in ResourceAccessException, so if you check for SocketTimeoutException directly, you'll never match it. I'd bet that bug is sitting quietly in more than a few codebases. (The exact exception types depend on your HTTP client, so check what yours actually throws.) And retrying on a socket timeout is only safe if the downstream call is idempotent. For payments, that means an idempotency key, which is what the next post is about.
Put a circuit breaker in front of it
Backoff slows retries down. A circuit breaker stops them. Once enough calls fail or run slow, the breaker opens and further calls fail immediately without touching the dependency, which gives it room to recover.
@Bean
public CircuitBreaker paymentGatewayBreaker() {
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
.slidingWindowType(SlidingWindowType.COUNT_BASED)
.slidingWindowSize(50)
.minimumNumberOfCalls(20)
.failureRateThreshold(50)
.slowCallDurationThreshold(Duration.ofSeconds(2))
.slowCallRateThreshold(60)
.waitDurationInOpenState(Duration.ofSeconds(30))
.permittedNumberOfCallsInHalfOpenState(5)
.build();
return CircuitBreaker.of("paymentGateway", config);
}
// Retry on the outside, breaker on the inside
Supplier<ChargeResponse> call = () -> gatewayClient.charge(request, idempotencyKey);
Supplier<ChargeResponse> guarded = Retry.decorateSupplier(retry,
CircuitBreaker.decorateSupplier(breaker, call));
The setting I'd pay the most attention to is the slow-call threshold. Overloaded services often don't return errors; they just take longer and longer. A breaker that only counts failures can sit there, closed, through the whole incident.
The ordering matters too. Because the retry wraps the breaker, once the breaker opens the retry gets a CallNotPermittedException. That isn't in the retryable list, so it gives up straight away instead of hammering an open circuit.
Cap retries as a share of traffic
Limiting retries per request doesn't limit retries overall. A retry budget does. The idea is simple: allow retries only while they're a small share of total traffic, something like 10%. At 10,000 requests a second, that's roughly 1,000 retries a second at most. On a normal day that's plenty for the odd failed call. During an overload the budget runs out quickly, and retries stop being an open-ended source of extra load. The right percentage depends on the dependency and your traffic, so treat 10% as a starting point rather than a rule.
Resilience4j doesn't have this built in, but a sliding-window count of requests versus retries in your client wrapper does the job. If you're on gRPC or Envoy, both already have a version of it: retry throttling in gRPC, retry budgets in Envoy.
Stop when the caller has stopped
Pass a deadline down with the request, and don't start a retry that can't finish before it. Say the client gives you 2 seconds. The first attempt takes 700ms, you back off for 300ms, and the second attempt takes another 700ms. You have 300ms left, and another 700ms attempt won't make it. Starting it anyway just produces work the caller will never see.
And when a server sends Retry-After with a 429 or 503, use it. It's the only information you'll get about when the other side can take more.
Questions I ask before adding a retry
- Is this failure likely to be transient, or could the dependency be overloaded?
- Is the call idempotent, or are we sending an idempotency key?
- Is anything above or below this already retrying?
- Is there backoff, and is it jittered?
- What stops retry volume from growing when everything is failing?
- Will the retry finish before the caller times out?
- Can we turn retries down or off during an incident without a deploy?
The last three are the ones that tend to get skipped, and they're the ones that matter once you're on the incident call.
One last thought
Retries are a trade: a bit more load and latency in exchange for fewer failed requests. Most days it's an easy trade to make.
On the day a dependency is struggling, that same trade works against you, and every extra attempt makes things worse for everyone. I'm not arguing for removing retries. A well-configured retry backs off when the system is under pressure; a badly configured one piles on. Most teams find out which kind they have during an incident.