[Python] Support for SubInterpreters and InterpreterPoolExecutors

[paultiq]

Describe the enhancement requested

InterpreterPoolExecutor's are to be introduced in 3.14, and backported to 3.13 python/cpython#124548, backport:

At present, pyarrow fails with: "ImportError: module pyarrow.lib does not support loading in subinterpreters"

See following example: TPE and PPE work, IPE does not.

from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor
from interpreters_backport.concurrent.futures.interpreter import InterpreterPoolExecutor

def try_tpe():
    with ThreadPoolExecutor() as executor:
        f = executor.submit(exec, "import pyarrow as pa")
        f.result()

def try_ppe():
    with ProcessPoolExecutor() as executor:
        f = executor.submit(exec, "import pyarrow as pa")
        f.result()

def try_ipe():
    with InterpreterPoolExecutor() as executor:
        f = executor.submit(exec, "import pyarrow as pa")
        f.result()

if __name__ == "__main__":
    try_tpe()
    try_ppe()
    try_ipe()

Result:

Traceback (most recent call last):
  File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 181, in run
    self._exec(script)
    ~~~~~~~~~~^^^^^^^^
  File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 127, in _exec
    raise ExecutionFailed(excinfo)
interpreters_backport.concurrent.futures.interpreter.ExecutionFailed: ImportError: module pyarrow.lib does not support loading in subinterpreters

Uncaught in the interpreter:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 111, in _call_pickled
    cls._call(fn, args, kwargs, resultsid)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 100, in _call
    res = func(*args or (), **kwargs or {})
  File "<string>", line 1, in <module>
  File "c:\gitother\iscratch\.venv\Lib\site-packages\pyarrow\__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: module pyarrow.lib does not support loading in subinterpreters

---

Tested on pyarrow 18.0.0rc0 on 3.13 amd64 using the https://pypi.org/project/interpreters-pep-734/ backport which backports the InterpreterPoolExecutor to 3.13.

Component(s)

Python

[paultiq]

This may be a dupe of #42151, so just linking it for clarity.

I guess the question is just whether there's any hope to get this on the roadmap or whether it's in a long hold for Cython.

I opened cython/cython#6445 on the Cython side asking more or less the same Q

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.