The Client That Opened a New Connection for Every Request

I have chased this from the incident end and from the slow end, and the truth was in the hand-off, as it usually is.

Here is the shape of the connection pool thrown away per call, and the one property that makes it invisible: the code is correct and the response is correct, and the only evidence is the handshake's cost, which the server's timing does not include.

The case was a Python service calling an internal API; the mechanics generalise to any HTTP client library that pools connections per instance.

Why the pool was discarded

The client was the constructed, and the constructed was the per call, and the per call was the function's body, and the body was the new instance, and the instance was the new pool, and the new pool was the empty, and the empty was the connection's absence, and the absence was the handshake, and the handshake was the three round trips, and the three were the latency.

The pattern was the tidy, and the tidy was the with block, and the with block was the close, and the close was the connection's return, and the return was the pool's, and the pool's was the discarded, and the discarded was the next call's new handshake, and the new handshake was the repeated cost, and the repeated was the incident.

The construction looked correct, and the correct was the review's pass, and the pass was the miss, because the cost is the not the syntax, and the not syntax is the lifetime, and the lifetime is the review's question, and the question is the who owns this client, and the owner should be the process rather than the call.

Why the server's timing hid it

The server's log was the twenty milliseconds, and the twenty was the handler, and the handler was the not the handshake, and the not handshake was the invisible, and the invisible was the confusion, because the server said fast and the client said slow, and the slow was the difference, and the difference was the network and the setup, and the setup was the three round trips.

The client's timing was the three hundred, and the three hundred was the total, and the total was the handshake plus the transfer plus the handler, and the three were the parts, and the parts were the diagnosis, and the diagnosis was the handshake's dominance, and the dominance was the fix's target, and the target was the reuse.

This is the same setup cost as the TCP slow start that made the first kilobytes the slow ones, and the shared property is the one worth fixing.

The fixes

Made the client a module-level or injected singleton. The client was the one instance, and the instance was the process's life, and the life was the pool's persistence, and the persistence was the reuse, and the reuse was the fix, and the fix was the lifetime, and the lifetime was the discipline, because the pool is the instance's and the instance should outlive the call.

Configured the pool's size and the keep-alive. The size was the concurrency, and the concurrency was the parallel requests, and the parallel was the no waiting, and the waiting was the fix's target, and the target was the number, and the number was the discipline, because the default pool is the small and the small is the queue.

Used the connection's keep-alive and checked the server's timeout. The keep-alive was the persistent connection, and the persistent was the reuse, and the reuse was the fix, and the fix was the header, and the header was the default, and the default was the server's timeout, and the timeout was the idle close, and the close was the reconnect, and the reconnect was the discipline.

Measured the client's total against the server's handler time. The comparison was the gap, and the gap was the setup and the network, and the network was the diagnosis, and the diagnosis was the fix's target, and the target was the measurement, and the measurement was the discipline, because the server's log alone cannot show the client's cost.

Handled the stale connection's retry. The pooled connection may be the closed by the server, and the closed was the first request's failure, and the failure was the retry, and the retry was the fix, and the fix was the library's setting, and the setting was the discipline, because the naive retry of the non-idempotent request is the duplicate.

If it happens again

An HTTP client pools connections per instance, so constructing one per request discards the pool and pays the DNS, TCP and TLS handshake every time. Create one client for the process, size the pool to the concurrency, keep the connections alive, and compare the client's total time against the server's handler time.

The internal API call took three hundred milliseconds and the server logged twenty, because the client was constructed inside the function and thrown away at the end of it, so every call opened a fresh connection and paid the full handshake. Get that property wrong and every layer above it lies to you politely.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud