The tcpdump Capture That Showed Who Sent the Reset

Between the failed and the body size sit 6 quiet assumptions, and the incident is the story of one of them failing.

Here is the shape of the packet capture as the arbiter, and the one property that makes it decisive: the capture shows the packets that actually crossed the wire, so it settles the argument between the layers, and the argument is which one closed the connection.

This was an upload through a load balancer to an application server, and the mechanics are the same for any connection that terminates unexpectedly.

Why the logs could not answer

The application log was the empty, and the empty was the request's absence, and the absence was the application's innocence, and the innocence was the not the answer, because the innocence does not say who sent the reset, and the who is the question, and the question needs the wire, and the wire is the capture.

The client log was the reset, and the reset was the symptom, and the symptom was the not the source, and the source was the unknown, and the unknown was the argument, and the argument was the three suspects, and the three were the client and the balancer and the application, and the three was the capture's job, and the job was the settlement.

The layers each see their own view, and the view is the partial, and the partial is the disagreement, and the disagreement is the hour, and the hour is the speculation, and the speculation is the cost, and the cost is the reason the capture comes before the argument's third round.

What the capture showed

The capture was the packets, and the packets were the sequence, and the sequence was the upload, and the upload was the bytes, and the bytes were the count, and the count was the same every time, and the same was the threshold, and the threshold was the limit, and the limit was the reset, and the reset was the balancer's address, and the address was the answer.

The source address was the balancer, and the balancer was the not the application, and the not application was the innocence's confirmation, and the confirmation was the configuration, and the configuration was the body size, and the body size was the default, and the default was the megabyte, and the megabyte was the less than the upload, and the upload was the rejection, and the rejection was the reset.

The same byte count was the threshold's proof, and the proof was the deterministic, and the deterministic was the limit rather than the flake, and the flake would have been the random, and the random was the absent, and the absent was the configuration, and the configuration was the fix, and the fix was the one line.

This is the same principle as the curl command that reproduced the bug in four seconds, and the common root is the property, not the platform.

How to run the capture well

Filter to the connection. The filter is the host and the port, and the two are the noise's removal, and the removal is the readable, and the readable is the diagnosis, and the diagnosis is the fix, and the fix is the expression, and the expression is the discipline, because the unfiltered capture is the haystack.

Capture on both sides of the suspect. The balancer's in and the application's in are the two views, and the two views are the comparison, and the comparison is the layer's identification, and the identification is the fix, and the fix is the two captures, and the two are the discipline, because the one capture cannot show what was dropped in between.

Write to a file and read it in a viewer. The file is the Wireshark's input, and the input is the readable, and the readable is the sequence, and the sequence is the diagnosis, and the diagnosis is the fix, and the fix is the tool, and the tool is the better than the terminal for the long capture.

Note the byte count and the timing. The count is the threshold, and the threshold is the limit, and the limit is the configuration, and the configuration is the fix, and the timing is the pattern, and the pattern is the deterministic or the random, and the two are the different diagnoses.

Remember the capture holds the payload. The payload is the sensitive, and the sensitive is the handling, and the handling is the deletion, and the deletion is the discipline, and the discipline is the policy, because the capture is the evidence and the evidence is the liability.

What I now do

When a connection drops and the application log is empty, the packet capture is the only thing that shows who sent the reset. Filter to the connection, capture on both sides of the suspect, note whether the failure is at a fixed byte count, and delete the capture when the diagnosis is done.

The upload died at exactly the same byte count every time and the application never saw the request, and the capture showed the reset coming from the load balancer's address, which had a one megabyte body limit. That is the whole pattern, and it is the part worth remembering.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud