The Export That Always Returned Nine Hundred and Ninety Nine Rows
A finance person noticed that two exports of "all orders" did not agree with the database count. The export said 999 rows. A direct count said 4,180. The export was not broken in any visible way. It was simply, silently, a thousand-row window on a much larger truth, and it had been treated as the truth in a board deck.
The scariest property of a silent cap is that it produces a plausible number. An error would have been caught. A wrong but confident number gets forwarded, charted and decided upon.
This was a Node 22.14 service using a client library for its data source, and the cap lived in the library's default page size, not in any code we had written.
The symptom, precisely
Any export whose true size exceeded one thousand rows returned at most one thousand, usually 999 or 1000 depending on off by one handling, with no warning, no flag and no exception. Exports under the cap were correct, which is what made the bug intermittent from the team's point of view. The boundary moved as the data grew, so the export "worked" for months and then quietly began lying.
The hypotheses that died
The query has a LIMIT we forgot
First I grepped the export code for LIMIT and page size. Our SQL had none. The query, run directly against the database, returned all rows.
It died because our query was fine. The truncation happened after the query, inside the client.
The database is limiting results
Next I suspected a server side row limit, a safety setting. Running the same query through a raw driver returned all 4,180 rows.
It died because the raw driver had no cap. Only the high level client truncated.
The export stream is dropping rows
Then I assumed the CSV writer was losing rows, a backpressure or flush bug. I counted rows arriving at the writer and they were already 1000 before the writer ever ran.
It died because the loss was upstream of the writer. The reader was handing over a truncated set.
The breakthrough
I read the client library's pagination defaults. The high level list() or find() method fetches a single page with a default page size of one thousand and returns it as if it were the whole collection. To get everything you must paginate, and the library's doc buried that in a paragraph under "performance considerations".
The code we inherited called the convenient method and trusted the array it got. The array was one page. There was no error because fetching one page successfully is not an error. The library's API made the wrong thing easy and the right thing manual, which is the default shape of this entire bug class.
This is the same silent truncation family as the export that always returned 999 rows in every SaaS client, and the same "the boundary moves as data grows" dynamic as the query that was fast until the table grew.
Why the number is 999 and not 1000
The off by one is itself a diagnostic. A cap reported as 999 usually means the library fetched a page of one thousand, used one row as a look ahead to decide whether more pages exist, and returned the rest. The 1000 variant means it returned the page untouched. Either way the exact number tells you the page size and the look ahead strategy, which is how you confirm you are looking at a pagination default and not a bespoke limit.
When a future export shows a new suspicious ceiling, read the ceiling first. Caps are fingerprints. A ceiling at 5000 is a different default than one at 1000, and naming the ceiling names the layer that imposed it, which is the same move as reading the sixty second interval in ECONNRESET at exactly sixty seconds.
What I changed
Paginate explicitly and exhaustively. The export now loops pages until a page returns fewer rows than the page size, accumulating the total, and it does this with an explicit maximum iteration guard so a broken cursor cannot loop forever.
Assert the count. Where the source can report a total, the export compares the number of rows fetched against the reported total and fails loudly on mismatch. A finance export that cannot prove it is complete should not present itself as complete.
Surface the cap at the API boundary. The public export endpoint now refuses to run an unbounded export synchronously above a threshold and moves it to the async job path, returning a file when done. This removes the temptation to lean on a single convenience call for large data.
Added the regression test. Seed 1,500 rows, export, assert 1,500. It is the test that would have caught this on day one, and it now guards every change to the data access layer.
What I would do differently
I would have treated "all" as a claim to verify, not a parameter. Any function named all, list, find all or export should be assumed to be paginated under the hood until proven otherwise, and the proof is a count against a source of truth.
I would also have been more suspicious of convenience. Client libraries optimise for the demo, where one call returning an array is the whole pitch. The demo never has more than a page of data. Production does. The gap between the demo and production is exactly where the silent cap lives.
The rule
A silent cap is a lie with good manners. Every "fetch all" path in a system should be able to state how many rows it returned and how it knows that is all of them. If the answer is "the array's length", that is not knowledge, it is trust in a default, and defaults are where the cap hides.
The same lesson, that a convenient boundary hides a boundary, is the cache key story in the cache that returned another customer's data, where the convenience was a key built from too little.
