> ## Content Index
> Fetch the complete content index at: https://debugly.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# Reviewing Retry Logic Before It Becomes an Outage
- URL: https://debugly.dev/reviewing-retry-and-backoff-logic/
- Published: 2026-09-09T03:30:00.000Z
- Updated: 2026-10-23T09:13:00.000Z
- Description: A retry looks like resilience and behaves like a load generator. Every retry needs a budget, jitter and an answer to what happens if the first attempt…
- Author: Rohit Bhadani
- Tags: Code Review, DevOps, Networking

Retries are the most common well intentioned defect I review. The author adds them to make the system more resilient, and under normal load they do exactly that. Then one dependency slows down, the retries multiply the load, and the resilience feature becomes the outage. A retry storm is not a rare event. It is the default behaviour of unbounded retrying under partial failure.

So I review retry logic with more suspicion than almost anything else, because the failure mode is not visible in the happy path and is catastrophic in the unhappy one.

This is the checklist I run, with the reasoning behind each item.

## 1\. Is there a budget, and is it enforced

A retry without a maximum is a loop with a hope. The code must answer: how many attempts in total, or how much wall time in total, before we give up. Either bound is fine. No bound is a defect.

The budget should be reasoned about from the caller's timeout, not picked at random. If the client gives up after five seconds, a retry policy that keeps trying for thirty seconds is doing work nobody will ever see, which is the same mismatch as [every timeout should be shorter than the one above it](https://debugly.dev/reviewing-timeouts-and-cancellation/).

What I look for in the diff is a named constant for the budget and a comment tying it to the upstream timeout. If the number is a bare literal inside a loop condition, I ask.

## 2\. Is there backoff, and is the backoff exponential

Retrying immediately is usually wrong, because the most common cause of a failed request is a resource that is briefly saturated, and an immediate retry lands in the same saturation. The first retry should wait, and each subsequent retry should wait longer, so that the aggregate retry load decays instead of persisting.

Exponential backoff is the standard shape: one second, two, four, capped. The cap matters. Uncapped exponential backoff can wait longer than the caller's timeout, which means the retry never gets observed, which is the budget problem again.

## 3\. Is there jitter, and is it full jitter

This is the item most often missing and it is the one that prevents lockstep. If a thousand clients all fail at the same moment and all back off by the same schedule, they will all retry at the same moments, producing periodic spikes that re saturate the dependency exactly when it is trying to recover.

Jitter randomises the wait. Full jitter, where the wait is a uniform random value up to the exponential cap, is the best simple option. The review question is: if every client in the fleet executed this retry at once, would the retries arrive spread out or in waves. If the code has no randomness, the answer is waves.

## 4\. Is the operation idempotent, and what proves it

The dangerous retry is one where the first attempt may have succeeded but the response was lost. Retrying then performs the operation twice. For a read this is fine. For a charge, a transfer, an enqueue, it is a duplicate.

The review must establish what happens on the second attempt. The acceptable answers are: the operation is naturally idempotent, or there is an idempotency key the server de duplicates on, or the retry is only issued when the request provably never reached the server, such as a connection refused before any bytes were sent.

"If it failed, it did not happen" is not an acceptable answer, because the most interesting failures are the ones where it happened but you did not get the receipt. This is the core of [webhook idempotency and duplicate delivery](https://debugly.dev/webhook-idempotency-duplicate-delivery/).

## 5\. Which errors are retried, and which are final

Retrying a 404 is usually pointless. Retrying a 429 without honouring the rate limit's own guidance makes you the abuser. Retrying a validation error forever is a busy loop.

The retryable set should be narrow and explicit: connection failures, timeouts, and the 5xx and 429 classes, with 429 honouring any Retry-After header. Everything else should fail fast. A catch all retry that retries every exception is the worst shape, because it retries your own bugs too, turning a logic error into repeated load.

## 6\. What does the retry do to the thing it retries against

Finally, review the retry from the dependency's perspective. If every client retries three times, a dependency at 90 percent failure becomes a request rate several times higher. Ask whether the dependency has its own limits, and whether your retries respect them. This is the system level view, and it is why [rate limiting on both sides](https://debugly.dev/429-rate-limiting-both-sides/) matters.

## The shape I want to see

```javascript
const attempt = withRetry({
  maxAttempts: 3,
  baseMs: 500,
  capMs: 8000,
  jitter: "full",
  retryOn: isTransient,         // explicit, narrow
  idempotencyKey: op.key,       // or proof the op is safe to repeat
}, doWork);

```

Every knob is named, the retryable predicate is a separate function I can read, and the idempotency story is in the call. When retries look like this, they are a reviewed decision rather than a loop somebody hoped would help.

## The rule

A retry is a contract with the future load of your whole fleet, not a local convenience. Review it by asking what a thousand simultaneous executions would do, and if the answer is "arrive together and multiply", the resilience feature is the vulnerability.

The same discipline applies to client backoff everywhere, including the canary and health check logic in [outage caused by a successful health check](https://debugly.dev/outage-caused-by-successful-health-check/), where a retrying prober can itself be the load that keeps a sick system sick.

## The one retry that is fine

To be precise about what I am not arguing: a single immediate retry on a connection level failure, before any bytes reached the server, is cheap and safe, and most HTTP clients already do it. The danger begins with application level retries that repeat work of unknown effect, multiply under fleet wide failure, and lack a budget. Review the second kind with suspicion and let the first kind pass.

---

Related reading: [Reviewing a Terraform Diff for the Resource It Will Replace](https://debugly.dev/reviewing-terraform-force-replacement/) explores a connected angle.

Related reading: [lsof and ss Will Tell You Who Owns That Port](https://debugly.dev/lsof-and-ss-for-open-sockets/) explores a connected angle.

Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)