Your Flaky Tests Are Not Flaky, They Are Telling You Something

Every team eventually adds this to their CI config:

retries: 3

It makes the red builds go away. It is also, most of the time, deleting a bug report.

I want to argue something slightly unpopular: a flaky test is usually correct. It is telling you that your system has a behaviour that depends on timing, ordering, or shared state. The test is not unreliable. The system is, and the test is the only thing that noticed.

The categories, and which ones are real bugs

Not all flakiness is equal. Sorting it is the first useful step.

1. A genuine race in the code under test

The valuable case, and the one retries destroy.

test("updates the counter", async () => {
  await Promise.all([increment(), increment()]);
  expect(await getCount()).toBe(2);
});

Fails occasionally because increment does a read-then-write with no atomicity. In production this is a lost update, happening silently, at a rate proportional to concurrency.

Your test found a data integrity bug. Retrying it hides the bug and keeps the failure rate.

The tell: the test fails more under load, on slower machines, or when the suite runs in parallel. If your CI is flakier than your laptop, that is a signal rather than an annoyance.

2. A race in the test itself

The test asserts before the thing has happened.

// flaky
fireEvent.click(button);
expect(screen.getByText("Saved")).toBeInTheDocument();

// correct
fireEvent.click(button);
expect(await screen.findByText("Saved")).toBeInTheDocument();

This is a real bug in the test and worth fixing properly. The fix is a proper wait condition, never a sleep.

await sleep(500) is the worst possible solution: it makes the suite slower for everyone, and it still fails when a machine is under load. If you have sleeps in your tests, each one is a future flake with a timer on it.

3. Shared state between tests

The classic, and the one that produces the maddening symptom of a test that passes alone and fails in the suite.

// test A leaves a user behind
test("creates user", async () => {
  await db.users.insert({ email: "[email protected]" });
});

// test B counts users and gets the wrong number
test("counts users", async () => {
  expect(await db.users.count()).toBe(0);
});

Order dependence means the suite passes locally where the order happens to work and fails in CI where the runner shards differently.

Find it by running the suite in random order deliberately:

vitest --sequence.shuffle
pytest -p no:randomly --randomly-seed=1234

If shuffling breaks your suite, you have order dependence, and it will bite you the day somebody adds a test in the middle.

4. Time

// fails at 23:59:59, and in December, and on the 31st
expect(formatDate(new Date())).toBe("2025-10-14");

// fails when the test runs slower than expected
expect(Date.now() - start).toBeLessThan(100);

Also: tests that fail only on the last day of a month, only during a daylight saving transition, or only when CI runs in UTC and you are in Asia/Kolkata.

Freeze time. vi.useFakeTimers(), freezegun, or dependency injection of a clock. Never assert on real elapsed time unless the test is specifically a performance test, and then give it generous bounds.

5. External dependencies

A test hitting a real network endpoint will eventually fail because the network exists. This is not flakiness in your code, it is a test design problem.

Mock at the network boundary with something like MSW or responses, or use a container for a real dependency you control. Do not call third party APIs in unit tests.

6. Resource limits in CI

Sometimes it genuinely is the environment. A CI runner with 2GB of memory running a suite that peaks at 1.9GB will fail unpredictably. Same for tests that open many file descriptors or ports.

The tell here is that failures are spread across unrelated tests rather than concentrated in one, and that the error messages are things like EMFILE, ENOMEM, or EADDRINUSE.

Finding which one you have

The instinct is to look at the failing run. As with any intermittent problem, the population tells you far more than the instance.

Record every failure. Most CI systems can export test results. Get them into something queryable and ask: which tests, how often, at what time, on which runner, in which shard, with which seed.

Then look for the correlation. A test that fails only on the 4-core runners is a resource or timing issue. A test that fails only when shard 3 runs it is order dependence. A test that fails at a constant low rate regardless of anything is probably a genuine race.

Reproduce deliberately. Once you have a suspect, try to make it fail on demand:

# run one test 200 times
for i in $(seq 200); do npx vitest run path/to/test.ts || break; done

# add CPU contention to widen timing windows
stress-ng --cpu 8 --timeout 60s &
npx vitest run path/to/test.ts

Adding load is the single most effective technique for timing dependent flakes. A race with a two millisecond window is nearly impossible to hit on an idle machine and easy to hit on a busy one.

Quarantine, do not retry

There is a real tension here. You cannot block every deploy on a test that fails one time in fifty, and hunting every flake immediately is not a good use of a team's time either.

The practice I would advocate:

Quarantine rather than retry. Move the flaky test out of the blocking suite into a separate job that runs and reports but does not gate the merge. The difference from a retry is that the failure stays visible. A retried test that passes on attempt two produces a green check and no record.

Cap the quarantine. A test in quarantine for more than two weeks gets fixed or deleted. Quarantine with no expiry is just a slower delete, and pretending otherwise lets the list grow forever.

Track the flake rate as a number. If you have twelve quarantined tests, that is twelve known behaviours in your system that nobody understands. That is a meaningful engineering health metric and it belongs somewhere visible.

Retries are acceptable for one category only: genuinely external dependencies you have decided not to mock, in an end to end suite, where the failure mode is a network blip. Even then, log the retry so you can see the rate.

Why this matters more with generated code

One more reason to take flakes seriously now.

Concurrency bugs are one of the categories coding agents produce most reliably, because concurrency is invisible in the text of a file. A read-then-write that is correct in a single threaded test and wrong under two simultaneous requests looks completely fine in review.

A flaky test is one of the few mechanisms that catches this automatically. If your CI retries it away, you have removed the detector for a defect class that is becoming more common, not less.

The reframe

Stop thinking of flaky tests as broken tests. Think of them as the only nondeterminism detector you have.

Production is concurrent, ordered unpredictably, and shares state. Your test suite is the one place that concurrency shows up as a red build instead of a support ticket. When a test fails one time in fifty, it has found something that happens one time in fifty in production too, except in production nobody is watching for it.

That is worth a day of investigation more often than teams assume.