Torch-XLA not compatible with static python

[drewjenks01]

❓ Questions and Help

I am trying to use Torch-XLA v2.3.0 but it fails with:

line 7, in <module>
    import _XLAC
ImportError: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

I noticed this message here:

Bazel brings in [pybind11](https://github.com/pybind/pybind11) embeded
python and links against it to provide `libpython` to the plugin using
this mechanism. Python headers are also sourced from there instead of
depending on the system version. These are satisfied from the
`"@pybind11//:pybind11_embed"`, which sets up compiler options for
linking with `libpython` transitively.

which suggests XLA is pulling in a two year old version of pybind11_bazel, which gets its python binary/library/headers/paths by inspecting the copy of the interpreter installed on the operating system. During this probing pybind11_bazel explicitly asks the python interpreter to give it the linker flags it would need to embed the interpreter in its code, leading to that dependency. This renders it unusable with static python.

Is there a way to make this work/could you provide a different build of Torch-XLA which is compatible with static python?

[ysiraichi]

Thank you for the question. I believe the excerpt just means that, for build purposes, it will link against the libpython provided by pybind11.

• What exactly do you mean by "static python"?
• Could you try setting the LD_LIBRARY_PATH variable to the path to your libpython library?

[gholms]

Thank you for the question. I believe the excerpt just means that, for build purposes, it will link against the libpython provided by pybind11.

* What exactly do you mean by "static python"?

* Could you try setting the `LD_LIBRARY_PATH` variable to the path to your `libpython` library?

Hi, I work with @drewjenks01. Our python interpreter is a cc_binary which builds from source during the build. It's statically linked, and there is no libpython shared library that we can point extensions at.

Python 3.8 added support for statically-linked interpreters by changing the standard for how native extensions on unix are expected to depend on libpython. In short, they should not declare dynamic library (DT_NEEDED) dependencies on it, and instead, rely on the interpreter to provide those symbols itself. This standard persists today.

https://docs.python.org/3.8/whatsnew/3.8.html#changes-in-the-c-api
python/cpython#65735

cc_binary is linking _XLAC.so in the way it typically would when given a deps dependency in precompiled shared library form because it doesn't know any better. It just happens that this breaks the standard for native python extensions, and in a way which easily goes unnoticed because it works on dynamically-linked interpreters.

[drewjenks01]

@ysiraichi Just following up to see if you had any thoughts on this / plans to address it? This is currently a blocking issue for us sadly

[ysiraichi]

Sorry for the late response. I don't think there are any plans to address it.
Maybe @miladm @tengyifei can comment more on this.