Reviewing the Timeout That Was Longer Than the Caller's

Between the slow and the incident sit 6 quiet assumptions, and the incident is the story of one of them failing.

I have hit this enough times that I now check for it before I check for anything clever.

The inverted timeout has a single weak property, and here it is: the downstream keeps working after the upstream has stopped waiting, so every retry adds work that nobody is waiting for, and the added work makes the next attempt slower.

The case was a service calling a payment provider; the mechanics generalise to any chain of calls where each layer sets its own timeout independently.

What the review saw

The review saw the timeout, and the timeout was the ten seconds, and the ten seconds was the configured, and the configured was the deliberate, and the deliberate was the pass, because the timeout was the present, and the present was the good practice, and the practice was the review's criterion, and the criterion was the met.

The review did not ask what the caller's timeout was, and the ask is the question, and the question is the five, and the five is the less than the ten, and the less is the inversion, and the inversion is the wasted work, and the wasted work is the incident, and the incident is the review's question that nobody asked, because the timeout is the local setting and the inversion is the system's property.

The timeout is the correct in isolation, and the isolation is the review's view, and the view is the file, and the file is the not the chain, and the not chain is the blind spot, and the blind spot is the incident, and the incident is the reason the timeout should be reviewed against the caller's.

Why the inversion amplifies the load

The caller waits the five, and the five is the give up, and the give up is the retry, and the retry is the second request, and the second request is the second work, and the second work is the first's continuation, and the continuation is the not cancelled, and the not cancelled is the doubled, and the doubled is the slower, and the slower is the more timeouts, and the more timeouts is the more retries.

The loop is the positive feedback, and the feedback is the load's growth, and the growth is the saturation, and the saturation is the outage, and the outage is the incident, and the incident is the reason the downstream's timeout should be shorter than the upstream's, so the downstream fails first and the upstream sees the real error rather than the silence.

The cancelled work is the absent, and the absent is the context, and the context is the propagation, and the propagation is the cancellation's reach, and the reach is the fix, and the fix is the deadline, and the deadline is the chain's, and the chain's is the discipline.

This is the same lifetime problem as the async call that nobody could cancel, and the shared property is the one worth fixing.

The questions that catch it

What is the caller's timeout? The answer is the number, and the number is the comparison, and the comparison is the inversion, and the inversion is the fix, and the fix is the shorter downstream, and the question is the review's first, because the local timeout is meaningless without the caller's.

Does the deadline propagate down the chain? The answer should be the yes, and the yes is the cancellation, and the cancellation is the waste's absence, and the absence is the fix, and the fix is the context, and the question is the review's second.

What happens to the work when the caller gives up? The answer should be the cancelled, and the cancelled is the resource's release, and the release is the fix, and the fix is the propagation, and the question is the review's third.

Is the retry bounded and backed off? The answer is the budget, and the budget is the amplification's limit, and the limit is the fix, and the fix is the cap, and the question is the review's fourth, because the unbounded retry is the loop's fuel.

The fix

Set the downstream timeout shorter than the upstream's. The ten became the four, and the four was the inside the caller's five, and the inside was the downstream's failure first, and the first was the real error, and the error was the fix, and the fix was the number, and the number was the discipline.

Propagated the deadline through the call chain. The context carried the remaining time, and the remaining was the every layer's budget, and the budget was the cancellation, and the cancellation was the fix, and the fix was the context, and the context was the discipline.

Bounded the retries and backed off with jitter. The budget was the two, and the two was the cap, and the cap was the amplification's limit, and the limit was the fix, and the fix was the exponential with the jitter, and the jitter was the discipline.

Documented the timeout budget for the service. The budget was the contract, and the contract was the caller's knowledge, and the knowledge was the configuration, and the configuration was the fix, and the fix was the document, and the document was the discipline, because the undocumented timeout is the guessed timeout.

The rule

A downstream timeout longer than the caller's means the downstream keeps working after the caller has given up, so retries pile up and the load amplifies. Set the downstream timeout shorter than the upstream's, propagate the deadline through the chain, bound the retries with backoff and jitter, and document the timeout budget.

The service waited ten seconds for a dependency while its caller gave up after five and retried, so the work was done twice for an audience that had already left, and the doubled load made the next attempt slower. Get that property wrong and every layer above it lies to you politely.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud