Add support for building C extensions

[blais]

🚀 feature request

• Add support for getting Python headers (Python.h) -- use @rules_python//python/cc:current_py_cc_headers
• Provide shared libraries (libpython.so et al) -- use @rules_python//python/cc:current_py_cc_libs
• Provide static libraries (libpython.a)
• Add a rule/rules to build C extensions
• Add PyInfo.cc_info field to allow C/C++ info (CcInfo) to propagate (directly
  propagating CcInfo it isn't done to prevent a Python target from
  being accidentally accepted by a plain CC target)

Describe alternatives you've considered

I've cobbled support for these things by hand in the past, was hoping building extension modules would be officially supported at some point.

Some projects, such as tensorflow and pybind11_bazel, have a python_configure.bzl file and/or py_extension rule that attempts to implement this.

(Note: this issue was originally about updating the readme to address the lack of this feature; it's been edited to be a feature request for this feature directly - rickeylev)

[rickeylev]

Can you clarify in the README how this module relates to building extension modules?

Which readme are you referring to, this repo's readme? This repo, pybind11, and tensorflow have different owners. There's some owner-adjacency with this repo and pybind11 (Me and Ralf are on the same team within Google, but don't overlap in work much really), but they're different projects. I think grpc also has a custom solution that mostly-works-except-when-it-doesn't.

Pybind11 is a higher-level way to write C extensions.

Tesorflow's purpose is an ML library -- if it provides a py_extension, then it's almost certainly just incidental. I've talked to various TF folk and they, too, want an official py_extension rule they can use.

I'm pretty certain all of these solutions are incomplete and/or unsound in some way -- I don't think the toolchain provides enough information to implement them entirely correctly.

hoping building extension modules would be officially supported at some point.

Yes, that's the plan. There aren't really any details beyond that, though. While there is a py_extension rule within Google, it's not compatible with Bazel as-is and has some historical baggage that make it unsavory to release as-is.

I'm happy to discuss design and implementation ideas about it, though.

[blais]

Can you clarify in the README how this module relates to building extension modules?

Which readme are you referring to, this repo's readme? This repo, pybind11, and tensorflow have different owners. There's some owner-adjacency with this repo and pybind11 (Me and Ralf are on the same team within Google, but don't overlap in work much really), but they're different projects. [...]

Pybind11 is a higher-level way to write C extensions.

Tesorflow's purpose is an ML library -- if it provides a py_extension, then it's almost certainly just incidental. I've talked to various TF folk and they, too, want an official py_extension rule they can use.

Hi Richard,
I mean this particular repo. From an external user's perspective, the support for building Python extension module isn't clear. This is the repo for Python rules for Bazel. Isn't this the most appropriate location to tell Bazel users about and to locate support for building extension modules for Python? I read rules_python ~= "Bazel + Python".

Tensorflow is only involved because they had to build those rules for themselves before others. As such, their files are used as examples people copy rules from. Pybind11 is indeed another project but closely related ("Bazel + Python + C++"). It probably deserves to be coordinated with this one? I spoke with Ralf a while ago (years ago) and my understanding is that the priority was elsewhere.

I think grpc also has a custom solution that mostly-works-except-when-it-doesn't.

Exactly. (Do you believe this is a great state of affairs?)
It seems to me this is something to gather in one place and fix once and for all for all these core projects.
In an ideal world, there would be a single copy of both these files, at least in the Google-owned projects.

Right now I don't even know which one to model mine from.

I'm pretty certain all of these solutions are incomplete and/or unsound in some way -- I don't think the toolchain provides enough information to implement them entirely correctly.

Here's what's happening now: AFAICT looking at OSS projects, people - including Googlers for Google open source projects - are copying two types of artifacts:

• configuration to build against a local runtime, usually called python_configure.bzl
• rules for building extension modules, usually called py_extension.bzl

Mutating and evolving copies of these two files are proliferating. At the minimum, it would be wise to place a generic version of each in here that serves as either a directly usable version or as an authoritative source for these definitions - one that perhaps, if it is not adequately parameterizable, still requires to be copied and modified by some projects. At least having a single, latest, best, central version of these two files would allow one to track them and merge their changes into their local copies. Not ideal but that would already be an improvement over the current situation.

hoping building extension modules would be officially supported at some point.

Yes, that's the plan. There aren't really any details beyond that, though. While there is a py_extension rule within Google, it's not compatible with Bazel as-is and has some historical baggage that make it unsavory to release as-is.

I'm happy to discuss design and implementation ideas about it, though.

I'm outside now, but I don't remember what the particular baggage involved was. It would be useful to know what is needed to make these rules usable outside of google3, and Bazel folks are thinking and what the current status for supporting those is. For example, is the reason they're not supported out of the box due to a particular technical issue? If so, can it be listed / documented somewhere? Or is it that their definition always involve a particular custom specification?

(This problem is emblematic of the issue with Google open source. Why isn't the company making an effort to make most of its open source projects easily importable together using Bazel? IMHO a worthy goal would be for the core OSS projects to be made usable with a one-liner in a WORKSPACE file, without having to clone files and spend hours figuring out incompatibilities and patches. That's an decent project for promo IMO, requiring coordination across teams all over the org and a deep understanding of how Bazel works and some understanding of all the core projects. Right now it's a bit of a mess, even Google's own OSS projects duplicate these files and it makes it difficult to cobble a set of them together easily.)

[rickeylev]

To build a DSO extension, we basically need 2 + 1 pieces.

First, we need a toolchain to give us the Python headers (really, the CcInfo for them). We have that in rules_python, sort of. The hermetic toolchain has a cc_library with the headers, but I don't think you can actually get to it. I don't think a system toolchain has anything here. A catch here is I think this has to be a separate toolchain (or maybe just toolchain type) for bootstrapping reasons. (see below).

Second, we need APIs to invoke the cc tools. We have that, sort of. Well, maybe fully now. The key piece is being able to tell the cc apis that the output library name needs to be e.g. "foo.so" instead of "libfoo.so"[1]. I think this can be worked around by creating a symlink with the desired name (there's a java-only cc api for this; not sure if its available to starlark yet). But, also, there's the issue of what goes into the DSO. The full transitive dependency? Just the srcs? Some subset of the transitive closure? This is where cc_shared_library comes in; it basically lets you control all that. I'm not sure if it's considered generally available yet; it was only mentioned to me a couple months ago internally (I think they've been working closely with Tensorflow).

The +1 part is py_binary needs some way to consume the DSO. Just sticking it data of a py_library to get it into the runfiles mostly works; this is basically what people do today and works OK. The short coming I see is that downstream targets, like binaries, don't know the DSO is there. It's just another data file in the runfiles. As a concrete example of what this metadata is good for, internally, we use that metadata to help inform how to turn the py_binary into a C executable that embeds Python.

The bootstrapping issue I'm not 100% sure about. The particular example I have in mind is coverage support. This was recently added to the Python toolchain; so to use it your Python toolchain has to depend on it. Coverage has its own C extension (ctrace, iirc). If you want to build coverage from source, then how do you get the Python headers? You have to ask the toolchain. But you're in the middle of constructing that toolchain, so how can you ask that toolchain for information? (Again, not 100% sure here, but this is how I understand things work). I realize most of you are fans of e.g. pre-built wheels (they def have their advantages), but supporting building from source is also major win.

Internally, the baggage our py_extension rule carries is it's very tightly coupled with cc_binary; as in, at the Java level, it's a subclass of it and calls various internal APIs. It has some other quirks, but that's the main thing. This is a case where starting fresh with the modern CC Starlark APIs is probably going to work out better. A lot more thought has gone into the CC rules and how to do this since when our rule came about. I think this might be entirely doable in Starlark today.

Hope this answers part of your question!

re: DSO infix, pybind11, andwheel building from #1113: If the extension-building rule also carried information about its ABI and python-tag needs, then this can help inform what extension a DSO can or should use.

[1] It should have the CPython infixes, too, but you get the gist.

[blais]

Thanks Richard, this is a great insight into the depth of the issues involved in making this work.
I need to chase down definitions for some of the bits you allude to, probably will come back with questions.
Thank you,

[rickeylev]

@rwgk @jblespiau @michaelreneer @laramiel
(various people at Google I know who are interested in this problem as well)

[laramiel]

We also have rules similar to the above in tensorstore:

https://github.com/google/tensorstore/blob/master/third_party/python/python_configure.bzl

[rickeylev]

Oh, another part I should call out is static vs dynamic linking. My depth of understanding C++ linking stuff is fairly shallow, but in any case: IMHO, this detail should be transparent to users. e.g., as Python C extension library author, I should be able to change from one to the other (or provide both and let consumers choose) without anyone's Python code caring. This sort of functionality requires metadata propagation so that the py_binary level can put things together appropriately.

[ethanluoyc]

@rickeylev I recently experimented with using bazel for compiling c++ extensions. I thought I could share some confusion I had when attempting to use bazel from a library development perspective.

Bazel has been used for building C extensions for quite a number of ML-related projects at Google (also DeepMind) see envlogger, reverb, launchpad, and also tensorstore from above @laramiel for examples.

I think at least for me personally, besides the lack of support for bundling extensions in py_wheel, it wasn't clear to me what would be the workflow looks like for creating python distributions with extensions using bazel: should we expect end users to (1) use bazel build and run the py_wheel rules or (2) maybe adopting bazel as a build backend as per PEP517.

(1) would be closest to the existing implementation of py_wheel, but it would probably not be very convenient for end users as they have to now learn a different workflow for installing dependencies built with bazel from source. I personally think 2) would ideally be the way to go but then it probably would take more effort to get there.

Thinking about 2), I realize there are at least a few things that weren't clear to me at the moment.

(2) suggests that a build frontend will create an isolated environment with the necessary build-time dependencies. I suppose in that case bazel should respect the installed interpreter dependencies and use those for building? Using installed dependencies is also convenient, for example, for libraries that depends on TensorFlow's libtensorflowframework.so so that you can avoid building TF from source.
Also, how should this interact with hermetic Python in theory? For library development, it's also quite likely you would want to build a separate version of the distribution for each ABI. In that case, it seems that we need to capability to configure different toolchains based on run-time configuration if we want to support PEP517, is this something that is possible to do now in Bazel? In addition, I wasn't sure how this should interact with the pip_parse rules.

Right now, the packages I mentioned above seems to adopt the python_configure approach. In that case, most of the libraries would require the users to pre-install the dependencies. The entire process is quite brittle overall with all sorts of quirks and hacks.

Personally, I think I would like a hermetic environment for development but still find an easy way to test the built distribution on a range of python versions.
Therefore, switching between a local_config_python and a hermetic python should be easy. From the documentation and examples, I couldn't find an example of how to do this, but maybe it is just because of my lack of understanding of how to configure it to work in this case. I saw there is a transition.bzl sitting around https://github.com/bazelbuild/rules_python/blob/main/python/config_settings/transition.bzl, is this something similar to what I was thinking about above?

[rickeylev]

(2) is basically a superset of (1). In order for something like pep517 to use bazel as a backend, it has to configure the Python toolchains to use whatever is appropriate. In the end, it has to do e.g. bazel build :foo, where :foo is a py_wheel target (or some equivalent).

Note that:

• Using bazel to build doesn't require that dependencies are built from source.
• Using rules_python doesn't require using hermetic Python.

Also, how should this interact with hermetic Python in theory?

It interacts through toolchains. Toolchains are the abstraction rules use that allow someone else to configure what some set of inputs are to the rule. e.g. such as the location of Python header files.

For library development, it's also quite likely you would want to build a separate version of the distribution for each ABI. In that case, it seems that we need to capability to configure different toolchains based on run-time configuration if we want to support PEP517, is this something that is possible to do now in Bazel? In addition, I wasn't sure how this should interact with the pip_parse rules.

Essentially, yes. This is handled by transitions and constraints. As an example, you can define a rule that performs a split transition, which allows building the same target for multiple different configurations (e.g. different abis). Using constraints, targets and toolchains can control what their inputs are depending on mostly arbitrary criteria; this is how the multi-version python rules work today (the transition.bzl file you mentioned).

Right now, the packages I mentioned above seems to adopt the python_configure approach. In that case, most of the libraries would require the users to pre-install the dependencies. The entire process is quite brittle overall with all sorts of quirks and hacks.

Yes, correct on all counts. The main missing piece here is a toolchain to carry the C dependency information (e.g. headers, the :python_headers target in the toolchain repos generated by the hermetic rules).

Note that all this toolchain stuff is not specific to the rules_python hermetic python stuff. You can, for example, implement a repository that tries to locate Python on the local system and define a toolchain using that information. That's basically what those local_config_python things try to do. Downstream rules are largely unaware if a local or hermetic toolchain is used.

is [transition.bzl] something similar to what I was thinking about above?

Its in an adjacent area, yes. That code is an example of how to make multiple Python versions co-exist in one build. You could, for example, define a test_all_the_pythons() macro, which, under the hood, was basically a loop doing py_test(name = "foo_" + V, python_version=V), and then a single bazel test ... invocation would run all N tests concurrently under the different python versions.

[ethanluoyc]

@rickeylev Thanks for the detailed reply!

Yes, correct on all counts. The main missing piece here is a toolchain to carry the C dependency information (e.g. headers, the :python_headers target in the toolchain repos generated by the hermetic rules).

One thing I wasn't sure how to do is to configure a pybind_extension from https://github.com/pybind/pybind11_bazel/tree/master/py. to simultaneously work with multiple toolchains. For the local_config_python approach the workflow mostly asks the user to call a local_config_python in WORKSPACE and a specific interpreter is bound there. Initially, I thought that it would be possible to do some sort of dynamic configuration using a toolchain that was registered but it does not seem as straightforward as I would have hoped (initially I thought if you have registered multiple toolchains there maybe an alias in the toolchain repo that can look up the correct python header and pass to the cc_binary but I wasn't able to find anything like this.) From what you described, ideally the correct thing to do here is to first allow the python toolchain to carry the header information and rewrite the pybind11_extension as a rule?

Another thing which I wasn't sure about is what's the right thing to handle requirements when bazel is configured to produce wheels. In the case of making bazel a PEP517 backend, then I would imagine the dependencies to be installed into build environment via the frontend, but the pip_parse rules will resolve the dependencies using the lock file (which may be different from the version retrieved during installation)

On an unrelated note, I encountered all of this as I was working on a PR here which uses bazel to compile native extensions. I managed to get things to work, but the entire process right now is pretty brittle and many things feel very unnatural.

[rickeylev]

Well, I think I got the toolchain half working. A cc_library can just depend on @rules_python//python/cc:current_py_cc_headers to get the correct headers (see question at the end though). I need to add a couple more tests, and will be unavailable for a bit, but this is otherwise looking very promising!

Fiddling a bit more, I was able to use cc_shared_library to build an importable Python C extension. I didn't vet it thoroughly, but a cursory examination of the .so output looks plausibly correct (eg. libc is marked needed; though libstdc++ is also marked needed, which I thought py extensions were plain C and wouldn't need that? There's probably a compiler flag missing).

There's a fair amount of boiler plate to create the DSO, so we'll definitely want a rule to hide some of that. Also note that cc_shared_library is experimental still. I'm not sure how (or if) a regular cc_library can create an equivalent.

But, a few questions others better versed in C might know:

• There's a :libpython target with DSOs. Should these also be made available? I'm assuming so?
• There's a static library that currently isn't exposed. Should this also be made available?
• The above two mean there are possibly 3 different "things" that need to be exposed -- should they be made available as separate things? e.g., if you want the headers, you depend on :current_py_cc_headers; if you want the DSOs, you depend on :current_py_cc_dynamic_libs, if you want the static libs, you depend on :current_py_cc_static_libs. (These all come through the same toolchain, it's just feeding that information into e.g. a cc_library requires an intermedate target to do the lookup and forward along the information).
• There's also an "interface" library (a cc_import) that is windows only. I have no idea what this is; it looks internal, but the entire toolchain BUILD file is marked public, so I don't really know for sure.

[junyer]

(eg. libc is marked needed; though libstdc++ is also marked needed, which I thought py extensions were plain C and wouldn't need that? There's probably a compiler flag missing).

Anything using pybind11 will be written in C++. ;)

• There's also an "interface" library (a cc_import) that is windows only. I have no idea what this is; it looks internal, but the entire toolchain BUILD file is marked public, so I don't really know for sure.

FWIW, this bit of Windows documentation says:

An import library (.lib) file contains information the linker needs to resolve external references to exported DLL functions, so the system can locate the specified DLL and exported DLL functions at run time. You can create an import library for your DLL when you build your DLL.

I don't know whether it's possible (and reasonable) to "hide" that detail at the Bazel level such that users depend on a single target regardless of the platform.

[rickeylev]

Anything using pybind11 will be written in C++. ;)

Yeah, but this was a hand-rolled Python C API extension, though. Basically a no-op extension. See foo.cc and its build file for how I was building it.

windows docs and hiding interface lib detail

Thanks for the doc link; explains a little bit, I think. The interface lib is already hidden, I think. It's a dependency of the target that holds the DSO files.