mirror of https://github.com/leanprover/lean4.git synced 2026-03-17 18:34:06 +00:00

Go to file

Kim Morrison b87d2c0fb9 feat: add lean-bisect script for bisecting toolchain regressions (#11727 )

This PR adds a Python script that helps find which commit introduced a
behavior change in Lean. It supports multiple bisection modes and
automatically downloads CI artifacts when available.

- [x] depends on: #11735

## Usage

```
usage: lean-bisect [-h] [--timeout SEC] [--ignore-messages] [--verbose]
                   [--selftest] [--clear-cache] [--nightly-only]
                   [file] [RANGE]

Bisect Lean toolchain versions to find where behavior changes.

positional arguments:
  file               Lean file to test (must only import Lean.* or Std.*)
  RANGE              Range to bisect: FROM..TO, FROM, or ..TO

options:
  -h, --help         show this help message and exit
  --timeout SEC      Timeout in seconds for each test run
  --ignore-messages  Compare only exit codes, ignore stdout/stderr differences
  --verbose, -v      Show stdout/stderr from each test
  --selftest         Run built-in selftest to verify lean-bisect works
  --clear-cache      Clear CI artifact cache (~600MB per commit) and exit
  --nightly-only     Stop after finding nightly range (don't bisect individual
                     commits)

Range Syntax:

  FROM..TO                Bisect between FROM and TO
  FROM                    Start from FROM, bisect to latest nightly
  ..TO                    Bisect to TO, search backwards for regression start

  If no range given, searches backwards from latest nightly to find regression.

Identifier Formats:

  nightly-YYYY-MM-DD    Nightly build date (e.g., nightly-2024-06-15)
                        Uses pre-built toolchains from leanprover/lean4-nightly.
                        Fast: downloads via elan (~30s each).

  v4.X.Y or v4.X.Y-rcN  Version tag (e.g., v4.8.0, v4.9.0-rc1)
                        Converts to equivalent nightly range.

  Commit SHA            Git commit hash (short or full, e.g., abc123def)
                        Bisects individual commits between two points.
                        Tries CI artifacts first (~30s), falls back to building (~2-5min).
                        Commits with failed CI builds are automatically skipped.
                        Artifacts cached in ~/.cache/lean-bisect/artifacts/

Bisection Modes:

  Nightly mode:   Both endpoints are nightly dates.
                  Binary search through nightlies to find the day behavior changed.
                  Then automatically continues to bisect individual commits.
                  Use --nightly-only to stop after finding the nightly range.

  Version mode:   Either endpoint is a version tag.
                  Converts to equivalent nightly range and bisects.

  Commit mode:    Both endpoints are commit SHAs.
                  Binary search through individual commits on master.
                  Output: "Behavior change introduced in commit abc123"

Examples:

  # Simplest: just provide the file, finds the regression automatically
  lean-bisect test.lean

  # Specify an endpoint if you know roughly when it broke
  lean-bisect test.lean ..nightly-2024-06-01

  # Full manual control over the range
  lean-bisect test.lean nightly-2024-01-01..nightly-2024-06-01

  # Only find the nightly range, don't continue to commit bisection
  lean-bisect test.lean nightly-2024-01-01..nightly-2024-06-01 --nightly-only

  # Add a timeout (kills slow/hanging tests)
  lean-bisect test.lean --timeout 30

  # Bisect commits directly (if you already know the commit range)
  lean-bisect test.lean abc1234..def5678

  # Only compare exit codes, ignore output differences
  lean-bisect test.lean --ignore-messages

  # Clear downloaded CI artifacts to free disk space
  lean-bisect --clear-cache
```

🤖 Prepared with Claude Code

---------

Co-authored-by: Claude <noreply@anthropic.com>

2025-12-21 20:04:47 +00:00