The Test That Only Failed When It Ran Second

An integration suite, green locally, green when re-run by hand, and failing in CI roughly one run in four. The classic profile of a flaky test, which is a phrase I have come to distrust because it usually means nobody has looked properly.

The failure was always the same assertion, and always only when the pipeline had run something else first.

What the symptom actually was

FAIL  tests/integration/import.spec.ts
  ● bulk import › creates 500 records

    expected 500, received 641

Six hundred and forty one. Not a timeout, not a connection error, not a race. A precise number that was 141 too high, and 141 was exactly the number of records the previous job in the pipeline created.

That single detail is the whole story, and it took me longer than it should have to look at it properly rather than re-running the job.

Tested on Node 22.14, Postgres 16.3, Linux 6.8.

The hypotheses that were wrong

Hypothesis one: the test does not clean up after itself

Reasonable, and the first thing anyone checks. If afterEach is not truncating, records accumulate within the run.

I read it. It truncated. I added a count assertion at the start of the test to prove the table was empty before the import, and in a passing run it was.

In a failing run it was not empty at the start, which meant the pollution predated the test file entirely. That was the moment the theory changed from "this suite leaks" to "something before this suite leaked".

Hypothesis two: parallel jobs sharing a database

Plausible. Two jobs against one Postgres instance will interfere, and CI parallelism makes it intermittent in exactly this way.

Ruled out by the connection string. Each job created its own database with a run scoped name. I verified in the logs that the failing run and its predecessor used different database names.

Different databases, and the data still crossed. So it was not the database.

Hypothesis three: a caching layer holding rows between jobs

We cache aggressively. If a cached count survived the job boundary, a stale number could surface.

The number was coming from a live SELECT count(*), not from cache. I confirmed by running the same query manually against the test database while the job was paused. The rows were genuinely in the table.

Real rows, in a fresh database, that nobody in this job had inserted.

The breakthrough

I stopped looking at the application and looked at the machine.

# added to the start of the CI job
echo "hostname: $(hostname)"
echo "uptime:   $(uptime -p)"
echo "docker:   $(docker ps -a --format '{{.Names}} {{.Status}}' | head)"
ls -la /tmp | head -20

The output on a failing run:

hostname: runner-07
uptime:   up 6 days, 4 hours
docker:   pg-testdb-a4f2  Up 41 minutes
          pg-testdb-91bc  Exited (0) 12 minutes ago
/tmp/pgdata-91bc/  ...

Six days of uptime. A leftover Postgres container from an earlier job still running. A /tmp directory with data from previous runs.

Our runners were long lived and shared. Each job created a database, but the container was reused when a previous job had left one healthy, because our setup script checked whether a Postgres was already listening on the port and reused it if so. That check was an optimisation somebody added to save thirty seconds of startup, and it was correct in isolation and catastrophic across jobs.

So the sequence was: job A creates its database in the shared container and does not drop it. Job B starts, finds a Postgres listening, reuses the container, and its "fresh" database is created inside a server whose data directory still holds A's tables. A specific query pattern that did not fully qualify the schema then picked up A's rows.

The test was correct. The isolation was imaginary.

What I changed

Made each job get its own machine rather than its own database. This is the actual fix, and everything else is a mitigation.

The pipeline now provisions a fresh microVM per job from a snapshot with the toolchain already installed, runs the job, and destroys it. On Krova Cloud that is a single API call and the machine boots in under a second, so the cost is negligible compared to the twenty minutes I had already spent on one flaky assertion.

krova cubes create ci-$CI_JOB_ID \
  --cpu 2 --ram 4 --disk 40 --snapshot ci-base
# run the job over SSH
krova cubes delete ci-$CI_JOB_ID

The property that matters is not speed, it is that there is no previous run. A machine created for this job has never executed anything else. Leftover containers, /tmp residue, stray processes, cached DNS, a modified /etc/hosts from a test that was killed halfway: none of it can exist, because the filesystem is a clone of a known good snapshot rather than the accumulated history of a shared box.

Removed the reuse optimisation. Even on a disposable machine, "reuse if already running" is a rule that will find a way to be wrong. Thirty seconds is not worth an intermittent correctness bug.

Made the runner state visible in every job. Hostname, uptime, running containers, and disk usage now print at the start of every run. It costs nothing and it turns a class of invisible problem into a line in the log.

What I would do differently

Print the environment before debugging the code. The application was innocent the whole time. Six days of uptime on a supposedly ephemeral runner was the answer, and it was one command away from the beginning.

Treat a precise wrong number as a gift. 641 was not noise. It was 500 plus exactly the previous job's 141, and that arithmetic pointed at a shared resource before I had formed a single hypothesis. Vague symptoms are hard; specific ones like this are the system telling you what happened, in the same way a failure at a regular interval means a configured timeout.

Stop accepting "flaky" as a diagnosis. A test that fails one run in four is failing deterministically under a condition you have not identified yet. In this case the condition was "ran after a job that used the same container", which is entirely predictable once you can see it. I have written about this before and still spent a morning ignoring my own advice.

Distrust optimisations that check for existing state. "Reuse it if it is already there" assumes the thing that put it there had the same intent you do. Across job boundaries that assumption is unfounded, and it fails silently rather than loudly.