The 4 MB JSON Response That Compression Fixed

I started at the slow, because that is where the symptom landed, and walked it backwards link by link until the slow stopped looking innocent.

The compression allowlist that omits the type that matters has a single weak property, and here it is: the response is correct and the status is 200, and the only evidence is the absence of a Content-Encoding header that nobody checks.

This was an nginx proxy in front of an API, and the mechanics are the same for any reverse proxy with a MIME-type allowlist for compression.

Why the JSON was not compressed

The proxy compressed the listed types, and the listed was the html and the css and the javascript, and the three were the web's assets, and the assets were the assumption, and the assumption was the API's exclusion, and the exclusion was the application/json's absence, and the absence was the four megabytes, and the four megabytes was the mobile's cost.

The JSON is the highly compressible, and the compressible is the repeated keys, and the repeated keys are the structure, and the structure is the redundancy, and the redundancy is the gzip's target, and the target is the ninety percent's reduction, and the reduction is the four hundred kilobytes, and the four hundred is the nine percent, and the nine percent is the fix, and the fix is the one line.

The reduction is the not the CPU's free, and the free is the trade, and the trade is the compression's CPU against the transfer's time, and the time is the mobile's dominant, and the dominant is the win, and the win is the fix, and the fix is the configuration, and the configuration is the discipline.

Why the omission was invisible

The response was the 200, and the 200 was the correct, and the correct was the data, and the data was the complete, and the complete was the no error, and the no error was the invisible, and the invisible was the header's absence, and the absence was the Content-Encoding, and the Content-Encoding was the unchecked, and the unchecked was the four megabytes, and the four megabytes was the slow.

The API's own test was the local, and the local was the fast network, and the fast network was the compression's irrelevance, and the irrelevance was the pass, and the pass was the deploy, and the deploy was the mobile, and the mobile was the slow network, and the slow network was the transfer's dominance, and the dominance was the incident, and the incident is the reason the payload size should be measured on the constrained connection.

This is the same shape as the table that rendered ten thousand rows to show twenty, and the shared property is the one worth fixing.

The fixes

Added the JSON type to the compression allowlist. The configuration gained the application/json, and the gained was the compressed, and the compressed was the nine percent, and the nine percent was the fix, and the fix was the one line, and the line was the discipline, because the allowlist is the explicit and the explicit omits what nobody added.

Compressed at the application or the proxy, not both. The double compression is the waste, and the waste is the CPU, and the CPU is the fix's target, and the target is the one layer, and the layer is the proxy, and the proxy is the discipline, because the application's compression and the proxy's are the redundant.

Set the minimum length for compression. The threshold is the small response's skip, and the skip is the CPU's saving, and the saving is the fix, and the fix is the configuration, and the configuration is the discipline, because the tiny response's compression costs more than it saves.

Measured the payload size and the transfer time. The metric was the response's bytes, and the bytes were the alert, and the alert was the growth's detection, and the detection was the prevention, and the prevention was the fix, and the fix was the monitoring, and the monitoring was the discipline, because the unmeasured payload grows until the mobile notices.

Checked the Content-Encoding header in the test. The assertion was the gzip's presence, and the presence was the regression's guard, and the guard was the prevention, and the prevention was the fix, and the fix was the test, and the test was the discipline, because the header is the only evidence the compression happened.

Reduced the payload as well as compressing it. The fields were the requested, and the requested was the sparse response, and the sparse was the smaller, and the smaller was the fix, and the fix was the API's design, and the design was the discipline, because the compression is the mitigation and the reduction is the cure.

What I now do

A proxy with a MIME allowlist for compression skips any type not listed, and application/json is frequently missing, so API responses travel uncompressed. Add the JSON type, compress at one layer only, set a minimum length, assert the Content-Encoding header in a test, and reduce the payload rather than only compressing it.

The API took six seconds on mobile and the payload was four megabytes of JSON that compressed to about three hundred and sixty kilobytes, because the proxy's compression allowlist contained the web asset types and not application/json. The lesson I keep is the property, not the incident.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud