How can we make Python's C-API more friendly for C++?

[encukou]

Hello,
I'm figuring out a strategy for the future of Python's C-API, and as one of the goals I'd like to make it friendly for non-C languages. But since I mostly deal with C & Python, I'd like to ask you for any suggestions, pain points or comments on how do better to that.
Would you like to help?

I started a Google doc to collect notes, please feel free to comment there, or here, or suggest a better place.
Would it be better to ask one of nanobind's predecessors?
Thank you for your time!

[wjakob]

Thanks for reaching out! I personally find some of the ongoing efforts to simplify the CPython API a little misguided (especially when it comes to functions always returning strong references). I have added some feedback to the form to clarify what I mean by this.

[encukou]

Apologies for the delay -- right after I asked, “life happened” (literally 👶). I'm back now :)

Thanks for kicking off the Discourse thread in the mean time!

As for borrowed references: yes, that change doesn't help nanobind -- it's there for other reasons.
References need to be borrowed from something, and very few APIs document what the something is, and what the implications of borrowing are. For example, if you use PyList_GetItem, you are borrowing from the list -- and you must ensure the item is not removed from the list until you're done with it. That's not particularly obvious, and can lead to subtle bugs.
(Borrowed references also reportedly prevent some possible optimizations that CPython or other implementations might want to make.)

That said, I'm not opposed to having alternate functions that return borrowed references, for use in inner loops. But they shouldn't be the default/only API, and they might tie the caller to a specific version/implementation of the interpreter.

[wjakob]

Hi @encukou,

having had some more time to think about API/ABI forward-compatibility and the various stakeholders, I've put together a concrete set of 6 requests for Python 3.12 :-)

https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993

Cheers,
Wenzel

[wjakob]

Also, congratulations on your baby! :-)

[encukou]

For anyone following along, there is now a wider effort for mapping out C API issues at: https://github.com/capi-workgroup/problems/issues