hyperfine Will Tell You the Build Got Slower and By How Much

The complaint was that the build got slower, and the evidence was one run, timed by eye, on a cold machine, compared against a memory of how it used to feel. Every performance argument that starts that way ends in a stalemate, because a single measurement of a noisy system is not a measurement, it is an anecdote with a stopwatch. The fix is not a better stopwatch. It is statistics, and the shortest path to honest statistics about a command is hyperfine.

I have lost on-call weeks to this exact shape of slow burn, so the order below is the order I fix it in.

hyperfine runs a command many times, with warmup runs discarded, and reports the mean, the spread and the comparison between commands, with the outlier sensitivity removed. It turns the build argument into a number with error bars, which is the difference between a feeling and a fact.

This was hyperfine on Ubuntu 24.04, benchmarking builds, test suites and a couple of CLI tools, and the habits below are the ones that make the numbers trustworthy.

The warmup is the point

The first run of almost any command is unrepresentative, because it pays every cold cost the machine has: the page cache empty, the language runtime compiling, the dependency graph loading, the CPU governor asleep. A benchmark that includes the first run measures the cold start, and if the thing being argued about is the steady state, the cold start is noise that dominates the mean.

hyperfine's warmup flag discards those runs before measuring:

hyperfine --warmup 2 'npm run build'

The warmup count should be enough for the cache to settle, which for builds is usually two or three, and the discipline is to state, in the argument, whether the question is the cold start or the steady state, because they are two different questions and each needs its own measurement, the cold one with warmup zero and the warm one with warmup.

Repeats are the confidence

A single warm run is still one draw from a noisy distribution, because the machine is shared, the scheduler is busy and the network, if involved, is someone else's. Repeats turn the draw into a distribution, and hyperfine's default repeat count plus its statistical summary gives the mean and the standard deviation, and the comparison between two commands with a speedup and a spread.

The spread is the information. A five percent difference with a twenty percent spread is no difference, and the tool saying so, plainly, ends the argument that a naive stopwatch would have inflamed. The number to trust is the ratio against the spread, and hyperfine computes it for you.

For flaky or slow commands, cap the runs with a maximum and accept the wider interval, because a benchmark that takes an hour to be precise is a benchmark nobody runs, and the habit matters more than the decimal.

Comparing is the real use

The single command benchmark answers "how long", but the question that starts arguments is "did it change", which is a comparison. hyperfine takes several commands and reports the relative speedup, which is exactly the question:

hyperfine --warmup 2 'git checkout main && npm run build' 'npm run build'

More usefully, it compares the before and after of a change, by running the old and new command, or the same command at two commits, and the reported ratio is the claim you ship. A refactor that claims no behaviour change should report a build time ratio of one, within the spread, and the report is the evidence.

This is the A B discipline applied to the terminal, and it generalises beyond builds: the test suite before and after the runner change, the query before and after the index, the deploy script before and after the rewrite. Anywhere someone says slower, the comparison command is the reply.

The environment is half the measurement

A benchmark is a claim about a machine and a moment as much as about the command, and the habits that keep it honest are the boring ones.

Run on an idle machine, or accept that the spread will include the neighbour, and say so. A laptop running a video call is a different instrument from the same laptop at rest, and comparing across them compares the calls, not the commands.

Pin what you can. The same checkout, the same dependency state, the same data size, because a build benchmarked against a small checkout and then a large one measures the data, and the query benchmarked before and after the table grew measures the growth, which is a real measurement but a different claim than the code change.

And record the context with the number, the machine, the commit, the cache state, because a performance number without its context is a rumour, the same lesson as the backup in a backup you have never restored is a rumour, applied to latency.

The CI habit that prevents the argument

The durable use is a scheduled benchmark in CI that runs the warm build and the key queries against a pinned dataset, stores the mean and the spread, and charts them over time. The chart turns "the build got slower" from a complaint into a lookup, with the date the slope changed, which hands the bisect its range for free, the same narrowing as git bisect run, pointed at performance instead of correctness.

The chart's value is the date, because the date names the commit, and the commit names the decision, and the argument becomes a review instead of a feud.

If it happens again

One run is an anecdote, and the cold run is the wrong anecdote. Warm up, repeat, compare, and record the context, and let the spread decide whether the difference is real.

hyperfine is the shortest path from feeling to fact about a command, and the habit of statistical comparison is the difference between a team that argues about performance and one that looks it up. The query side of the same discipline, where the plan changes under the data, is the query that was fast until the table grew, same mechanism, different costume.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud