Networking

Your Timeouts Are Wrong and Here Is How to Set Them

A missing timeout turns a slow dependency into an outage. A badly chosen one turns a blip into a retry storm. Both are common.

The short answer

Every network call needs a timeout. Most defaults are either infinite or wildly too long.

Client	Default connect timeout	Default read timeout
Python `requests`	none	none
Node `fetch`	~300s (OS)	none
Go `http.Client`	none	none
Java `HttpClient`	none	none
`curl`	300s	none

Read that table again. The default for most HTTP clients is to wait forever.

A reasonable starting point for an internal service call: 1 second to connect, 5 seconds total, 2 retries with jittered backoff, on idempotent requests only.

Why a missing timeout is an outage

A dependency gets slow. Not down, slow. Requests that took 40ms now take 30 seconds.

Without a timeout, your request handler waits. Each waiting request holds a connection from your pool, a worker thread or a task, and memory. Within a minute every worker is blocked on the slow dependency and your service stops responding to requests that do not touch it at all.

Your service is now down because something you call is slow. You have propagated their degradation to your users, and to everything that calls you.

This is the single most common way a partial failure becomes a total one, and the fix is a number in a config file.

The timeouts you actually have

"Timeout" is not one setting. A single HTTP request has several phases and most clients let you bound them separately.

DNS resolution. Frequently not configurable in the client, and it can take five seconds when a UDP packet is lost. Worth knowing that your 3 second total timeout can be consumed entirely by name resolution.

TCP connect. Should be fast. Within a datacentre, tens of milliseconds. Across the internet, a few hundred. Set this to 1 second internally, 3 externally. A connect that is slow means the host is unreachable or overloaded, and waiting longer rarely helps.

TLS handshake. Another round trip or two. Usually bundled with connect.

Time to first byte. How long the server takes to start responding. This is where server side processing shows up.

Total request time. Everything including body transfer. The one that matters for a streaming or large response.

Idle timeout on a pooled connection. How long a connection sits unused before being closed. Set this lower than the server's keepalive timeout, or you will occasionally send a request on a connection the server has already closed and get a confusing connection reset.

That last one causes intermittent errors that look random and correlate with traffic troughs. If you see sporadic ECONNRESET on a service that is otherwise healthy, check the two idle timeouts against each other.

Choosing the number

Not by intuition. From your own latency data.

Start from p99, not the average. If p99 for this call is 200ms, a 5 second timeout is generous and a 250ms timeout will fail a meaningful fraction of legitimate requests.

A workable rule: timeout = p99 times 3, rounded to something sensible, with a floor of a few hundred milliseconds.

Then check it against your own budget. If your endpoint has a 1 second latency objective and it makes three sequential downstream calls, each cannot have a 5 second timeout. The sum of the downstream timeouts must fit inside your own.

This is the constraint most people miss. Timeouts must be shorter as you go deeper. If the client gives up after 3 seconds and your downstream timeout is 10 seconds, that work continues after nobody is listening, consuming capacity for a result that will be discarded.

Propagate the deadline where you can. gRPC does this natively. In HTTP you can pass a remaining budget header and have each service bound its own calls by what is left:

const deadline = Number(req.headers["x-deadline-ms"]) || 3000;
const budget = deadline - elapsed();
if (budget < 50) throw new DeadlineExceeded();

await fetch(url, { signal: AbortSignal.timeout(Math.min(budget, 2000)) });

Setting them

Node fetch:

const res = await fetch(url, { signal: AbortSignal.timeout(5000) });

AbortSignal.timeout is the modern answer and it covers the whole request. For separate connect and read timeouts you need an agent with connectTimeout, or undici directly.

Python requests:

requests.get(url, timeout=(1, 5))   # (connect, read)

The tuple form is important. A single number sets both, and the read timeout is per socket read rather than for the whole response, so a slow trickle of bytes can exceed it indefinitely. For a hard total bound you need httpx with its Timeout object, or an outer mechanism.

Go:

client := &http.Client{
    Timeout: 5 * time.Second,           // total, including body read
    Transport: &http.Transport{
        DialContext: (&net.Dialer{Timeout: 1 * time.Second}).DialContext,
        TLSHandshakeTimeout:   2 * time.Second,
        ResponseHeaderTimeout: 3 * time.Second,
        IdleConnTimeout:       90 * time.Second,
    },
}

Prefer context.WithTimeout per request over the client level Timeout, because it composes with deadline propagation.

Database clients need them too, and they are separately configured:

SET statement_timeout = '5s';
SET lock_timeout = '2s';
SET idle_in_transaction_session_timeout = '30s';

Those three should be set on every production Postgres database. The third one in particular prevents a class of connection exhaustion.

Retries make it worse if you are careless

A timeout without a retry policy is half a design.

Only retry idempotent operations. GET, PUT, DELETE by design. POST only with an idempotency key. Retrying a non-idempotent POST after a timeout is how you charge a customer twice, because the timeout does not tell you whether the server processed the request.

Always use exponential backoff with jitter. Without jitter, every client that failed at the same moment retries at the same moment, and your recovering service is immediately knocked over again.

const delay = Math.min(cap, base * 2 ** attempt);
const jittered = Math.random() * delay;      // full jitter

Full jitter, meaning a random value between zero and the computed delay, spreads load better than adding a small random offset. This is well established and rarely implemented correctly.

Cap total attempts and total time. Three attempts, and abandon if the overall deadline has passed regardless of attempt count.

Do not retry at every layer. Three layers each retrying three times is 27 requests for one user action. Pick one layer, usually the outermost that can meaningfully recover, and make the others fail fast.

Add a circuit breaker for a dependency that is properly down. After a threshold of failures, stop calling for a period and fail immediately. This protects the dependency from your retries while it recovers, and it protects you from spending your latency budget on something certain to fail.

Testing that any of this works

Almost nobody tests timeout behaviour, and it is the code path that runs during your worst incidents.

Inject latency in a test:

server.use(
  http.get("/api/slow", async () => {
    await delay(30000);
    return HttpResponse.json({});
  })
);

await expect(fetchWithTimeout("/api/slow")).rejects.toThrow(/timeout/i);

Throttle in a staging environment with tc on Linux:

tc qdisc add dev eth0 root netem delay 2000ms

Verify the failure mode, not just that it times out. Does your service return a useful error? Does it fall back? Does it stay up? A timeout that produces a null and continues is worse than one that fails loudly.