Tokenizers fails to build in Python3.13t (NoGIL build)

[vinayakdsci]

When installing tokenizers from main with the following command invoked:

python3.13t -m pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python

there is a build failure originating from the numpy crate:

        Compiling numpy v0.23.0
      error[E0432]: unresolved import `pyo3::sync::GILProtected`
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.23.0/src/datetime.rs:63:12
         |
      63 | use pyo3::{sync::GILProtected, Bound, Py, Python};
         |            ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
         |
      note: found an item that was configured out
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.23.5/src/sync.rs:47:12
         |
      47 | pub struct GILProtected<T> {
         |            ^^^^^^^^^^^^

      error[E0432]: unresolved import `pyo3::sync::GILProtected`
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.23.0/src/strings.rs:15:5
         |
      15 |     sync::GILProtected,
         |     ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
         |
         = note: unresolved item `crate::datetime::units::GILProtected` exists but is inaccessible
      note: found an item that was configured out
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.23.5/src/sync.rs:47:12
         |
      47 | pub struct GILProtected<T> {
         |            ^^^^^^^^^^^^

      For more information about this error, try `rustc --explain E0432`.
      error: could not compile `numpy` (lib) due to 2 previous errors
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/home/vinayak/.venv/python3.13_ft/bin/python3.13t" PYTHON_SYS_EXECUTABLE="/home/vinayak/.venv/python3.13_ft/bin/python3.13t" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-req-build-7xcc2xt3/bindings/python/Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/vinayak/.venv/python3.13_ft/bin/python3.13t', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

tokenizers appears to be using v0.23.0 of the crate, which has since released a v0.24.0.

Here is more info about my Python version:

$ python3.13t -VV

Python 3.13.2+ experimental free-threading build (heads/3.13:4d7d249, Mar 16 2025, 23:30:31) [GCC 11.4.0]

[vinayakdsci]

cc @Narsil