Demo: Testing pybind11-stubgen + mypy

[timohl]

Description

This PR shows how pybind11-stubgen and mypy could be used in pytest to check for correct type hints.
It stems from the discussion at #5663.

This is just a quick demo with one positive and one negative test.

Sources for calling pybind11-stubgen and mypy from Python:

• Integrating mypy into another Python application
• Calling pybind11-stubgen from Python:
  Add args to main sizmailov/pybind11-stubgen#252
  (thx @gentlegiantJGC for this 👍)

TODOs:

• Integrate test cases from Experimenting: Annotated[Any, pybind11.CppType("cpp_namespace::UserType")] #4888
• Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)
• Pin versions of mypy and pybind11-stubgen (or even use a version matrix in CI).
• Consider other type checkers (most notably Pyright/Pylance since this is the default in vscode).

---

📚 Documentation preview 📚: https://pybind11--5678.org.readthedocs.build/

[timohl]

Some errors I see so far:

• Python 3.14 seems like causing some mypy internal errors
• Some checks do not properly install pybind11-stubgen, although I added it to requirements.txt and pyproject.toml
• mypy report on https://github.com/pybind/pybind11/actions/runs/15149485803/job/42592785722?pr=5678 seems scrambled (maybe some formatting or windows specific stuff)

[rwgk]

The new test dependencies seem "small/lightweight" and "super lightweight":

$ time pip install mypy
Collecting mypy
  Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl.metadata (2.1 kB)
Requirement already satisfied: typing_extensions>=4.6.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (4.13.2)
Requirement already satisfied: mypy_extensions>=1.0.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (1.1.0)
Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (12.4 MB)
Installing collected packages: mypy
Successfully installed mypy-1.15.0

real    0m1.098s
user    0m0.936s
sys     0m0.150s

$ time pip install pybind11-stubgen
Collecting pybind11-stubgen
  Using cached pybind11_stubgen-2.5.4-py3-none-any.whl.metadata (1.9 kB)
Using cached pybind11_stubgen-2.5.4-py3-none-any.whl (30 kB)
Installing collected packages: pybind11-stubgen
Successfully installed pybind11-stubgen-2.5.4

real    0m0.418s
user    0m0.242s
sys     0m0.048s

I think you're going in a great direction.

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

If you do that, I'd say forget the tests under #4888 (those came mostly from mypy). I believe they'll be 100% redundant.

Pyright/Pylance since this is the default in vscode

I'd look at that is the top priority. — I don't know a lot about typing, but last time I quizzed ChatGPT about it (a few months ago), it told me mypy is a good start, but — liberally reinterpreted by me — what really counts is LSP.

[henryiii]

If you haven't started using uv, you should try it, even if it's just uv venv and uv pip:

$ uv venv
Using CPython 3.13.3 interpreter at: /usr/local/opt/[email protected]/bin/python3.13
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate.fish

________________________________________________________
Executed in   28.56 millis    fish           external
   usr time   12.44 millis    0.25 millis   12.19 millis
   sys time   13.04 millis    1.91 millis   11.13 millis

$ time uv pip install mypy
Resolved 3 packages in 3ms
Installed 3 packages in 50ms
 + mypy==1.15.0
 + mypy-extensions==1.1.0
 + typing-extensions==4.13.2

________________________________________________________
Executed in   81.80 millis    fish           external
   usr time   18.27 millis    0.29 millis   17.98 millis
   sys time   54.27 millis    3.55 millis   50.73 millis

I think ty and pyrefly, the two new Python type checkers in Rust, are likely to have a big impact in the future, but for now, most projects work on passing mypy, and the existing type checkers like pylance know that and try to be compatible. I don't think I use anything other than mypy in CI on all my projects, though I use pyright locally in VSCode and via LSP in Vim.

The bug in Python 3.14.0b1 that we've reported also affects mypy, and it's been marked as a release blocker. So mypy will likely work better in beta 2.