Find the commit that broke your build with git bisect

You will find the commit that broke a build or a test with git bisect, first by hand and then automatically with a test script. The guide is for developers who know everyday Git and want a repeatable way to find regressions.

Prerequisites#

• Git 2.56.0, the current release. Git 2.29 or newer runs everything here except --reset-when-found, which is new in 2.56.0. The output in this guide comes from 2.56.0, and older versions word some messages differently.
• Git user.name and user.email set, because the demo script makes commits.
• Python 3 from python.org (opens in a new tab) for the worked example. It uses only the standard library.
• A clean working tree when you bisect your own projects. Bisect checks out other commits, so commit or stash your changes first.

Check your Git version:

git --version

How bisect finds the commit#

You give Git one commit that has the bug (bad) and one that does not (good). Git checks out a commit halfway between them. You test it and report the result. Git drops the half of the range that cannot contain the culprit and picks a new midpoint.

Each answer halves the remaining commits, so 1,000 commits need about 10 tests.

1. Build the example repository#

The example is a small invoicing module. At tag v1.0, invoice_total rounds 2.675 half up to 2.68. Forty commits later the test fails, and one of those commits is the culprit. A WIP commit also leaves a syntax error in cli.py that lasts three commits, so the build fails on those.

Create the repository#

Make an empty working directory and save this as make-demo.sh inside it:

1set -e
2mkdir bisect-demo && cd bisect-demo
3git init -q -b main
4printf '__pycache__/\n' > .gitignore
5
6cat > pricing.py <<'EOF'
7from decimal import Decimal, ROUND_HALF_UP
8
9
10def invoice_total(lines, tax_rate):
11    subtotal = sum((Decimal(price) * qty for price, qty in lines), Decimal("0"))
12    total = subtotal * (1 + Decimal(tax_rate))
13    return total.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
14EOF
15
16cat > test_pricing.py <<'EOF'
17import unittest
18from decimal import Decimal
19
20from pricing import invoice_total
21
22
23class InvoiceTotalTest(unittest.TestCase):
24    def test_rounds_half_up(self):
25        self.assertEqual(invoice_total([("2.50", 1)], "0.07"), Decimal("2.68"))
26
27
28if __name__ == "__main__":
29    unittest.main()
30EOF
31
32git add . && git commit -qm "Add invoice_total with tests" && git tag v1.0
33
34for i in $(seq 1 40); do
35  case "$i" in
36    19) echo 'def main(:' > cli.py; git add cli.py
37        git commit -qm "WIP: start CLI" ;;
38    22) printf 'def main():\n    pass\n' > cli.py
39        git commit -qam "Fix CLI syntax" ;;
40    27) sed -i.bak 's/return total.quantize.*/return Decimal(str(round(float(total), 2)))/' pricing.py
41        rm pricing.py.bak
42        git commit -qam "Simplify rounding in invoice_total" ;;
43    *)  echo "- change $i" >> CHANGELOG.md; git add CHANGELOG.md
44        git commit -qm "Update changelog ($i)" ;;
45  esac
46done

Run it:

bash make-demo.sh

Write the test script#

Save this as bisect-test.sh next to bisect-demo/, not inside it. The git-bisect docs (opens in a new tab) say test scripts are safer outside the repository. If the script lived in the repository, old commits would lack it or carry a different version of it.

#!/bin/sh
# Exit 125: this commit cannot be built, so bisect skips it.
# -f recompiles every file, so stale bytecode cannot hide a syntax error.
python3 -m compileall -f -q . >/dev/null 2>&1 || exit 125

# Exit 0 if the test passes (good), 1 if it fails (bad).
python3 -m unittest -q test_pricing

Make it executable and confirm that HEAD fails:

chmod +x bisect-test.sh
cd bisect-demo
../bisect-test.sh; echo "exit $?"

The test reports AssertionError: Decimal('2.67') != Decimal('2.68') and the script exits with 1.

2. Bisect by hand#

Start the session#

Run these from the root of bisect-demo:

git bisect start
git bisect bad
git bisect good v1.0

git bisect bad with no argument marks HEAD. The short form is git bisect start HEAD v1.0, with the bad commit first. Git checks out the first midpoint:

# Bisecting: 19 revisions left to test after this (roughly 4 steps)
# [6501d8808c4baf9bb271b053e76bdfc6aa4acc8d] Update changelog (20)

Your hashes will differ from the ones in this guide.

Mark each commit#

Test the checked-out commit, then report one of three results:

../bisect-test.sh
git bisect good    # the test passed
git bisect bad     # the test failed
git bisect skip    # this commit cannot be tested

The first midpoint has the broken cli.py. The syntax error has nothing to do with the rounding bug, so run git bisect skip and Git picks a nearby commit. The next pick is WIP: start CLI, which is broken too, so skip it as well.

To skip a whole range, use range notation. For example, git bisect skip v2.5..v2.6 skips every commit after v2.5, up to and including v2.6.

If you skip the commits next to the culprit, Git cannot single it out. It prints The first 'bad' commit could be any of: followed by a list of hashes.

Read the result#

After the last mark, Git names the culprit and shows the commit:

# a2658dd2e205f23b280a3722323d5c1305c00f58 is the first 'bad' commit
# commit a2658dd2e205f23b280a3722323d5c1305c00f58
# ...
#     Simplify rounding in invoice_total
#
#  pricing.py | 2 +-

The ref refs/bisect/bad points at that commit until you end the session.

End the session#

git bisect reset

This returns you to the commit you started from. git bisect reset bisect/bad checks out the culprit instead. Reset before each new session, so that HEAD in the next start refers to the commit you expect.

3. Fix a wrong mark with log and replay#

One wrong good or bad mark sends bisect into the wrong half. You do not need to start over. Save the session log:

git bisect log > ../bisect.log

The log is a script that Git can replay. Each mark appears as a comment line followed by a command. Say you marked Update changelog (28) good by mistake. The end of the log looks like this:

# skip: [1a7ec1eb0caa60266a6feb400088590bcc0645db] WIP: start CLI
git bisect skip 1a7ec1eb0caa60266a6feb400088590bcc0645db
# good: [71a920af0ea619cafca47c7c6a9f47dd211272e4] Update changelog (28)
git bisect good 71a920af0ea619cafca47c7c6a9f47dd211272e4

Delete the two lines for the wrong mark. Then reset and replay:

git bisect reset
git bisect replay ../bisect.log

Git reapplies the marks you kept and checks out the next commit to test. In this example that is Update changelog (28) again. To list the remaining suspects in the terminal, run:

git bisect visualize --oneline

Without arguments, git bisect visualize opens gitk when it detects a desktop session and gitk is installed. Otherwise it runs git log.

4. Automate with git bisect run#

git bisect run runs your command on each commit it checks out and marks that commit based on the exit code:

Start the session and pass the script to git bisect run:

git bisect reset
git bisect start HEAD v1.0
git bisect run ../bisect-test.sh

Git prints running '../bisect-test.sh' before each test. It ends with:

# a2658dd2e205f23b280a3722323d5c1305c00f58 is the first 'bad' commit
# ...
# bisect found first 'bad' commit

In this run the script ran 8 times. Three runs hit the broken cli.py and exited 125, so Git skipped those commits without any input from you.

For a short check, pass a shell one-liner instead of a script file:

git bisect run sh -c "python3 -m compileall -f -q . >/dev/null || exit 125; python3 -m unittest -q test_pricing"

Clean up when the culprit is found#

Git 2.56.0 added --reset-when-found. With this option, Git reports the culprit and then resets the session for you. Put the option directly after run:

git bisect reset
git bisect start HEAD v1.0
git bisect run --reset-when-found=found ../bisect-test.sh

found leaves the culprit checked out. original, the default when you omit the value, returns you to where you started. git bisect start accepts the option too, but not together with --no-checkout. Copy the hash from the output, because the bisect refs and the log are gone after the reset.

5. Handle flaky tests#

A single wrong mark from a flaky run gives you the wrong culprit. First remove the randomness you control, such as random seeds and network calls. Then pick a rule based on how the test flakes.

If the bug itself shows up only some of the time, run the test several times and treat any failure as bad:

1#!/bin/sh
2python3 -m compileall -f -q . >/dev/null 2>&1 || exit 125
3
4# One failure in five runs marks the commit bad.
5for attempt in 1 2 3 4 5; do
6  python3 -m unittest -q test_pricing >/dev/null 2>&1 || exit 1
7done
8exit 0

If the test sometimes fails for reasons unrelated to the bug, retry and treat any pass as good:

1#!/bin/sh
2python3 -m compileall -f -q . >/dev/null 2>&1 || exit 125
3
4# One pass in three runs marks the commit good.
5for attempt in 1 2 3; do
6  python3 -m unittest -q test_pricing >/dev/null 2>&1 && exit 0
7done
8exit 1

Both versions find the same culprit in the demo. Each step now costs several test runs, so keep the range between the good and bad commits small.

6. Use --first-parent on merge-heavy histories#

When every change reaches the main branch through a merge, feature branches often contain WIP commits that do not build. A default bisect tests those commits too. You get more skips, and bisect can blame a commit that broke something inside a branch but was fixed before the merge.

With --first-parent, bisect follows only the first parent of each merge. It tests only your main line and blames the merge that brought in the bug:

git bisect reset
git bisect start --first-parent HEAD v1.0
git bisect run ../bisect-test.sh

On a separate test history with six merged feature branches, this ran 3 tests and reported Merge feature-4 as the first bad commit. It never checked out the WIP commits.

If the result is a merge, bisect between its parents to find the commit inside that branch. $merge^2 is the branch tip (bad) and $merge^1 is the main line before the merge (good):

merge=$(git rev-parse refs/bisect/bad)
git bisect reset
git bisect start "$merge^2" "$merge^1"
git bisect run ../bisect-test.sh
git bisect reset

Verify it works#

In bisect-demo, run a bisect that leaves the session open. Then check that the culprit fails and its parent passes:

1git bisect reset
2git switch main
3git bisect start HEAD v1.0
4git bisect run ../bisect-test.sh >/dev/null 2>&1
5git bisect reset bisect/bad
6culprit=$(git rev-parse HEAD)
7../bisect-test.sh >/dev/null 2>&1; echo "culprit exit: $?"
8git checkout -q "$culprit~1"
9../bisect-test.sh >/dev/null 2>&1; echo "parent exit: $?"

You should see culprit exit: 1 and parent exit: 0. Now revert the culprit on main and test again:

git switch main
git revert --no-edit "$culprit"
../bisect-test.sh

The test prints OK.

Troubleshooting#

The run stops with "exit code ... is < 0 or >= 128"#

The test crashed or a signal killed it, so the script returned 128 or higher. Add || exit 1 after the test command so a crash counts as bad. The session stays open, so run git bisect run again after the fix.

The run stops with "bogus exit code 126 for 'good' revision"#

The script is not executable (126) or does not exist at the path you gave (127). When the first test returns either code, Git runs the script on the good commit and stops if it gets the same code again. Run chmod +x on the script and check its path. Git versions before 2.36 lack this check and mark every commit bad instead.

A commit that cannot build is marked good#

Build output left over from another commit can make the build check pass. In the demo, compileall without -f trusts cached bytecode with a matching timestamp, so a broken cli.py can pass the check. Force a full rebuild in the script, for example with compileall -f, or run make clean first.

The culprit makes no sense#

Either a mark was wrong, or old commits need a different setup. Check git bisect log and fix the wrong mark with replay, as in step 3. If an old commit cannot build, make the script exit 125 on it.

Next steps#

• To look for a change that is not a bug, use custom terms, such as git bisect start --term-old fast --term-new slow. To find the commit that fixed something, mark commits old and new instead of good and bad.
• Narrow the search with a pathspec: git bisect start HEAD v1.0 -- pricing.py.
• Read the git-bisect documentation (opens in a new tab) and the longer Fighting regressions with git bisect (opens in a new tab) paper.