C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions

[vstinner]

Feature or enhancement

The private _PyLong_AsByteArray() and _PyLong_FromByteArray() functions were removed in Python 3.13: see PR #108429.

@scoder asked what is the intended replacement for _PyLong_FromByteArray().

The replacement for _PyLong_FromByteArray() is PyObject_CallMethod((PyObject*)&PyList_Type, "from_bytes", "s#s", str, len, "big") but I'm not sure what is the easy way to set the signed parameter to True (default: signed=False).

The replacement for _PyLong_AsByteArray() is PyObject_CallMethod(my_int, "to_bytes", "ns", length, "big"). Same, I'm not sure how to easy set the signed parameter to True (default: signed=False).

I propose to add public PyLong_AsByteArray() and PyLong_FromByteArray() functions to the C API.

Python 3.12 modified PyLongObject: it's no longer a simple array of digits, but it's now a more less straightforward _PyLongValue structure which requires using unstable functions to access small "compact" values:

• PyUnstable_Long_IsCompact()
• PyUnstable_Long_CompactValue()

So having a reliable and simple way to import/export a Python int object as bytes became even more important.

---

A code search for _PyLong_AsByteArray in PyPI top 5,000 projects found 12 projects using it:

• Cython (0.29.36)
• blspy (2.0.2)
• catboost (1.2)
• fastobo (0.12.2)
• gevent (22.10.2)
• guppy3 (3.1.3)
• line_profiler (4.0.3)
• msgspec (0.16.0)
• orjson (3.9.1)
• pickle5 (0.0.12)
• pyodbc (4.0.39)
• rlp (3.0.0)

Linked PRs

• gh-111140: Adds PyLong_AsNativeBytes and PyLong_FromNative[Unsigned]Bytes functions #114886
• gh-111140: Minor doc fixes for PyLong_AsNativeBytes #115375
• gh-111140: Improve PyLong_AsNativeBytes API doc example & improve the test #115380
• gh-111140: Fix edge case in PyLong_AsNativeBytes where large negative longs may require an extra byte #116053
• gh-111140: minor docs typos cleanup in the C example API calls. #118612

[scoder]

Thanks for creating the issue. I agree that the functions should be added. The current replacements seem awful for this kind of basic functionality. Going through an expensive Python call like this for converting between PyLong and large C integer types (int128_t) seems excessive.

Note that at least a couple of projects that you list use Cython implemented parts and thus probably just mention the function in there. I'm sure something like line_profiler would never end up calling it.

[serhiy-storchaka]

It was already discussed several times. This API lacks some features which would be needed for general use. You need to know the size of the resulting bytes array. Calculating it is not trivial, especially for negative integers. Also, it would be core convenient to support "native" ending, not just "big"/"littel".

I have been thinking about implementing a similar API for mpz_import/mpz_export in GMP (https://gmplib.org/manual/Integer-Import-and-Export) or mp_unpack/mp_unpack in libtommath (https://github.com/libtom/libtommath). It is powerful and a kind of standard. It is general enough to support internal PyLong represenation and the marshal format used for long integers (15 bits packed in 16-bit words). But it is only for unsigned integers, you are supposed to store the sign separately. It should be extended to support negative integers in several formats, for convenience and for performance.

[vstinner]

I'm not sure that passing the endian as a string is efficient if this function is part of hot code.

[serhiy-storchaka]

Not as a string. Just 3-variant value native/little/big instead of boolean little/big.

[vstinner]

Not as a string. Just 3-variant value native/little/big instead of boolean little/big.

Sorry, I was confused between C API (int for the endian) and the Python API (string for the endian).

[scoder]

It was already discussed several times. This API lacks some features which would be needed for general use.

Ok, then please put the existing function back in until there is a public replacement.

[scoder]

I created PR #111162

[vstinner]

This API lacks some features which would be needed for general use.

Which features are missing?

Do you have links to previous discussions if it was discussed previously?

[vstinner]

You need to know the size of the resulting bytes array.

wcstombs() can be called with NULL buffer to compute the buffer size. It avoids to have to provide a second API "calculate the buffet size".

I suppose that a common use case is also to convert a Python int object to C int type for which there is no C API. Like int128_t

[scoder]

I suppose that a common use case is also to convert a Python int object to C int type for which there is no C API. Like int128_t

Or something like 256 bytes for a crypto key, hash value or similar data type. Probably a known size, so that users can live with an exception if it doesn't fit (because it's an error or can't-handle-that situation). That said, a function to ask for the bit length of the integer value could be helpful in order to find a suitable integer/storage size. And also more generally to estimate the size of an integer value. Both together would probably cover a lot of use cases.