Every puzzle hands you real code from a real open-source project, frozen at the commit right before somebody fixed it. It compiles. It has tests. It passed review. It shipped.
It is also wrong.
You do not have to explain what is wrong. You have to prove it. Find the input that breaks it, and write the test that shows the break.
The grader runs your test twice:
| against | your test must |
|---|---|
| the code as it shipped | fail |
| the maintainer's real fix | pass |
Red, then green. Nothing else counts. If your test passes on both, you proved nothing. There is no partial credit and no score for good reasoning.
You read code you did not write. Every day, more of it. You approve it or you don't, and most of the time you are guessing. Nobody measures the guessing.
A failing test is not a guess. It is an artifact. It exists or it doesn't, and nobody has to agree with you for it to be true. Your judgment is invisible until you can hand somebody a repro.
That is the whole exercise: turn a suspicion into an artifact.
- Real code, at a real commit, with a real bug a maintainer took seriously.
- A binary answer in seconds, on your own machine.
- Full attribution after you solve it: repository, commit, license, link.
- No account, no server, no streak, no leaderboard, nothing to sign up for.
- Not a course. There is nothing to watch.
- Not a spot-the-bug quiz. Pointing at the line is not an answer here.
- Not about fixing. The fix already exists. The proof is the work.
- Not a benchmark for models. It is a place to find out where you stand.
1. Get your own copy.
Click Use this template, then Create a new repository. Open it in GitHub Codespaces and everything is installed for you. Locally you need Python 3.11 or newer:
pip install -r requirements.txt
2. Pick a puzzle and read the code.
Each puzzle lives in puzzles/NN-slug/. Start with BRIEF.md: it tells you
what the code is supposed to do. It does not tell you what is wrong. The code
as it shipped is in before/. Write your test in your_test.py.
3. Ask the grader.
python -m grader puzzles/NN-slug
You get one of three answers:
| verdict | meaning |
|---|---|
| PROVEN | failed on the shipped code, passed on the fix. Solved. |
| NOT A PROOF | it does not separate the two. Details in the output. |
| BROKEN TEST | the test could not run, or could not run the same way twice. |
There is no partial credit. Attribution unlocks on PROVEN. Add -v to see
the raw pytest output.
Puzzle 01, start to finish. First, the repo as it arrives — the stub proves nothing, and the grader says so:
$ python -m grader puzzles/01-nothing-to-interleave
01-nothing-to-interleave — Nothing to interleave
------------------------------------------------
against the shipped code : your test fail
against the fix : your test fail
[NOT A PROOF] Your test fails on the fix too, so it is not testing this bug.
You read the code and try something. This one is wrong in an interesting way — it is about empty iterables, and the bug is about no iterables:
def test_empty_iterables_interleave_to_nothing():
assert list(interleave_evenly([[], []])) == [] against the shipped code : your test pass
against the fix : your test pass
[NOT A PROOF] Your test passes on both. It does not notice the bug.
Then you get it:
def test_no_iterables_at_all():
assert list(interleave_evenly([])) == [] against the shipped code : your test fail
against the fix : your test pass
[PROVEN] Failed on the shipped code, passed on the fix. That is a repro.
## Where this came from
- **Project:** `more-itertools/more-itertools`
- **Commit with the fix:** `f51a53bfd2` — "fix: handle empty interleave_evenly input"
- **License:** MIT
- **Function:** `interleave_evenly` in `more_itertools/more.py`
...
The rest of that reveal explains what broke and why, and links the commit.
7 puzzles, Python, easiest first.
| # | puzzle | difficulty |
|---|---|---|
| 01 | Nothing to interleave | warm-up |
| 02 | Already singular | warm-up |
| 03 | A factor of one | warm-up |
| 04 | One unit short | core |
| 05 | A ratio that vanishes | core |
| 06 | Restored to a constant | core |
| 07 | A tie between equals | hard |
The three warm-ups narrow down where to look. From 04 on, they do not.
Nothing here is a trick. Every bug was found, reported and fixed by real people in the last few months, and every fix shipped with a test. No two puzzles break the same way.
The attribution is hidden until you solve the puzzle. Not because it is a secret — it is all public, and you get repository, commit, license and link the moment your test goes red then green. It is hidden because knowing the project and the commit turns the puzzle into a search query.
You can look it up. You can ask a model. There is no way to stop you and this repo does not try. There is also no score to inflate and nobody to fool. If you want the exercise, work the code. If you want to test a model instead, that is a fair use of this repo — just say which model when you report it.
Do not post solutions in issues or pull requests. The whole thing runs on one puzzle being new to one person at a time.
Every puzzle credits its source. Repository, commit SHA, SPDX license
identifier and a link, shipped with the puzzle and revealed on solve. Only
permissively licensed sources are used. The full list is in NOTICES.md.
The grader is the whole product, so it is built to say no. It runs your test against one side at a time, with the other side absent from disk, more than once, and against a compiled copy with the source file removed.
That rules out, in order: reaching for the fix on disk; a test that passes on both; a test that pins the bug instead of the fix; a test that fails on the fix too; a file with no test in it; a test that answers differently on identical runs; a test that never finishes; and a test that greps the source for the patch instead of running the code.
Each of those is demonstrated firing:
python tests/run_guards.py
If you find a way to get PROVEN without a real repro, that is a bug in the grader and worth an issue.
The grader and everything written for this repo are MIT — see LICENSE.
Puzzle code belongs to its original project and stays under that project's
license. Each puzzle ships its own attribution and license notice, revealed
when you solve it; NOTICES.md lists every project used.