File Uploads Corrupted Above One Megabyte and Nothing Below

Support said uploads were "sometimes broken". The word sometimes was doing a lot of work. Once we got actual files, the pattern was binary: anything at or under one megabyte was fine, anything above was corrupted, and every corrupted file was corrupted in the same way, cut off at the same point.

A bug that flips at a precise size is not a data problem. It is a boundary problem, and boundaries in HTTP are almost always about buffering or chunking.

This was a Node 22.14 upload endpoint behind nginx 1.27, storing to object storage.

The symptom, precisely

A 900 KB image uploaded fine. A 1.4 MB image arrived as if someone had taken scissors to it at exactly 1,048,576 bytes. The stored object was the first megabyte plus a fragment, never the whole file. The client reported success, because from the client's point of view the bytes left the machine.

The one megabyte number is a strong hint. It is the default high water mark territory and it is also a common proxy buffer size.

The hypotheses that died

The client is sending it wrong

First suspect was the uploader. Maybe the browser or the SDK was slicing the file and dropping a chunk. I uploaded the same file with curl, bypassing all client code, and it still truncated at the same byte.

It died because the transport was innocent. The bytes arrived at the server intact. The loss happened after arrival.

nginx is cutting it off

Next I blamed the proxy. A client_max_body_size would reject, not truncate, and would return a 413, which we were not seeing, and that is the subject of 413 with two limits. A proxy buffer misconfiguration could in theory mangle a body, so I pointed curl straight at the Node port, skipping nginx. Still truncated.

It died because the truncation survived the removal of the proxy.

The disk or object storage is dropping bytes

Then I suspected the write side. Maybe the S3 multipart upload was misassembled. I logged the length of the buffer right before writing and the length reported by storage after. They matched, both truncated. So storage received exactly what we sent.

It died because the corruption was already present before the write. We were faithfully storing a wrong value.

The breakthrough

The handler looked like this:

app.post("/upload", (req, res) => {
  req.on("data", (chunk) => { process(chunk); });
  // ... but the response logic was wired to "end" of a different stream
});

The real defect was subtler and more common. The code consumed the request with req.read() style logic in a place that only pulled the first buffered chunk, then treated the request as done. Under one megabyte, the body fit in the initial buffer, so one read got everything. Above it, the body spanned multiple chunks, and the code only kept the first.

The smoking gun was that the truncation point matched the size of the first chunk the runtime delivered, which under load was one megabyte. The handler was, in effect, reading the stream the way you would read a file that fits in memory, and a stream is not that.

What I changed

The fix was to stop hand rolling stream consumption and let a well tested pipeline do it:

import { pipeline } from "node:stream/promises";

app.post("/upload", async (req, res) => {
  const target = createWriteStreamToStorage(key);
  await pipeline(req, target);   // drains the whole stream, backpressured
  res.json({ ok: true });
});

pipeline drains the request fully, applies backpressure so memory stays bounded, and rejects if the client aborts or the stream errors. It removes the entire class of "I read the first chunk and declared victory".

I also added an integrity check, because correctness matters more than speed here. The client sends a Content-Length, and we hash while streaming:

let bytes = 0;
const hash = crypto.createHash("sha256");
req.on("data", (c) => { bytes += c.length; hash.update(c); });

On completion we compare bytes to Content-Length and, where the client supplies it, to a checksum header. A mismatch becomes a 409 the client can retry, instead of a silently stored half file.

Finally, I added a test that uploads a body larger than one megabyte and asserts the stored byte length equals the sent length. It sounds almost insulting as a test, but it is precisely the regression that shipped, and it now fails in CI in under a second.

What I would do differently

I would have trusted the size threshold sooner. "Fails above X bytes" is a sentence with one common cause family: a boundary between buffered and streamed handling. I spent time on client and proxy before reading the one function that touched the bytes, and that function had the answer in it.

I would also have made the endpoint honest about partial writes from day one. A stored object whose length disagrees with the declared length is a lie, and systems that store lies get debugged at two in the morning. Length and hash verification at write time turns that lie into a loud, retryable error.

What I now do

A stream must be drained, not sampled. If your code reads a request body and the correctness of the result depends on the body being small enough to fit in one read, you do not have an upload handler, you have a handler with a size based coincidence.

The same "read once and stop" defect shows up in log readers, socket handlers and response accumulators, anywhere a stream is mistaken for a value. And the same honesty principle, verify what you stored against what was declared, is the difference between a bug you catch in CI and one a customer finds with a corrupted invoice.

If your uploads instead fail with a hard rejection at a size limit, that is the two limit problem in 413 request entity too large, which is a different signature and a different fix.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud