Build with numpy 2

[jdblischak]

Follow-up to #734

What is preventing us from building TileDB-VCF with numpy 2? Help wanted

[jdblischak]

[sc-50106]

[jdblischak]

It appears to fail whenever tiledbvcf-py is imported on Linux or macOS. The import passed on Windows. The passing Linux and macOS builds don't run import tiledbvcf, and thus avoid the error.

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/tiledbvcf/__init__.py", line 2, in <module>
    import pyarrow
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/tiledbvcf/__init__.py", line 2, in <module>
    import pyarrow
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
  File "pyarrow/lib.pyx", line 37, in init pyarrow.lib
ImportError: numpy.core.multiarray failed to import

[gspowley]

It looks like this is related to the pyarrow==11 pin. I reproduced the error locally (on Linux) by pinning pyarrow==11 and avoided the error by removing the pin.

[gspowley]

An overly aggressive unpinning experiment in #741 failed on windows and macos. I believe the macos failure is known and why we're pinning pyarrow in the first place.

For reference, the build and pytests succeeded on a local macos 14.5 arm64 with these versions:

numpy              2.0.0
pyarrow            16.1.0

[jdblischak]

@gspowley thanks for investigating! So at least for VCF, it looks like the blocker for numpy 2 is this issue with pyarrow

[jdblischak]

Cross-referencing numpy<2 PRs:

• SOMA has pinned numpy<2 [python] Pin to numpy < 2.0 for 1.12 release single-cell-data/TileDB-SOMA#2745 ([python] Tracking for NumPy 2.0 single-cell-data/TileDB-SOMA#2536)
• Vector Search has pinned numpy<2 Pin numpy to fix Python CI failures TileDB-Vector-Search#419

[ihnorton]

I believe the macos failure is known and why we're pinning pyarrow in the first place.

@gspowley is there a story for this? The pin was introduced in #719 after a lot of debugging, but the mac error in #741 doesn't look familiar and I can't find a tracking story.

[gspowley]

The pyarrow pin for the macos nightly CI was introduced in #579. The debugging in #719 included attempts to remove the pyarrow pins, which failed for macos, so the pins were maintained.

@jdblischak and @awenocur may have more context on the original macos failures.

[jdblischak]

xref: #762, TileDB-Inc/tiledb-vcf-feedstock#138

[gspowley]

Addressed in #762.