git bisect run Finds the Commit While You Get Coffee

Something broke between two releases, four hundred commits apart. Manual bisection would take nine careful checkouts, each with a build and a manual reproduction, an afternoon of attention with a real chance of a misjudged step. Automated bisection took ninety seconds of my time and twelve minutes of the machine's, because the reproduction was a script and the script could answer good or bad faster and more consistently than I can.

git bisect run is the most underused debugging tool I know. The concept is simple. You give bisect a script, and it drives the binary search, checking out each candidate and asking the script whether that commit is good or bad, until the first bad commit is isolated.

This was git 2.47. The exit code contract has been stable forever.

The exit code contract

The script's exit code is the whole interface.

The subtle one is 125. Some commits simply cannot be tested, a broken build, a missing dependency, a file the test needs that did not exist yet. Returning 125 tells bisect to skip that commit rather than misclassify it. Misclassifying an untestable commit as bad sends the search into the wrong half, which is the failure mode that makes people distrust automated bisection.

The script that works

A bisect script is a reproduction, and the discipline is the same as writing any good test: it must fail for the bug and pass without it, quickly and deterministically.

#!/usr/bin/env bash
set -u
# build the thing under test; if it cannot build, skip
make build 2>/dev/null || exit 125

# run the reproduction; its exit code is the verdict
./run_repro.sh

The reproduction is the part that takes thought. For a wrong output bug, it is a script that computes the output and compares to the expected. For a crash, it is the crashing invocation. For a performance regression, it is a timed run with a threshold, which works but is noisier, and needs a threshold loose enough to absorb jitter or you will get false verdicts and a flaky bisect.

Driving it

git bisect start
git bisect bad HEAD
git bisect good v2.4.0
git bisect run ./bisect.sh

Bisect then walks the range, and finishes by naming the first bad commit. When done, git bisect reset returns you to where you were.

The range choice matters. The good end should be a commit you have verified is good, not a guess, because a wrong good end bisects the wrong history. If you are unsure, verify the ends first by running the reproduction on each, which is two runs that buy the whole search's correctness.

Making the reproduction fast

The build is usually the slow part, and bisect rebuilds at every step. Two tricks cut it drastically.

Cache what is stable. If the bug is in one module, build the rest once and rebuild only the changed part, or use your build system's incremental mode so most steps are cheap.

Test at the lowest level that still shows the bug. A unit level reproduction that shows the defect bisects in seconds per step. A full end to end reproduction bisects in minutes per step. The commit found is the same. The cost is not. This is the same argument as writing the isolation test at the cache layer in the cache that returned another customer's data, where the narrow reproduction is the cheap and sharp one.

The traps

A nondeterministic reproduction bisects to the wrong commit. If the bug reproduces only sometimes, a bad commit can test good and halve the search wrongly. Make the reproduction deterministic, or loop it inside the script until it is confident, trading time for a trustworthy verdict.

The bug may predate your good end. If bisect reports the good commit itself as bad, or finds the first bad at the very start, your range is wrong and the regression is older than you thought. Widen the range rather than trusting a boundary result.

Multiple regressions in the range. If two commits broke related things, the first bisect finds one. Rerun bisect from that commit to the bad end to find the next. The git bisect log of the first run tells you where to resume.

When manual is still right

If the reproduction is a human judgement, a visual glitch, a subtle UX regression, automation cannot verdict it, and git bisect manual with a prepared checklist is fine. But even then, try to find a proxy the machine can judge, a rendered snapshot diff, an assertion on the DOM, because the proxy, imperfect as it is, removes the attention cost that makes manual bisection slow and error prone.

The rule

A regression between two known points is a binary search, and a machine should run it. Write the reproduction as a script that honours the exit code contract, skip untestable commits with 125, verify your range ends, and bisect at the narrowest level that shows the bug.

The ninety seconds you spend writing the script replaces an afternoon of careful checkouts, and the machine, unlike the human, does not misjudge step six because it is tired. Keep the bisect script in the repository next to the reproduction it encodes, because a bisect script is documentation of how the bug manifests, and the next person to regress that behaviour will thank you for a ready made verdict machine rather than a paragraph of vibes. For the case where the defect is not a regression but a mystery behaviour, the equivalent narrowing tool at runtime is strace for web developers.