add PyLong* API (3.14+)

[chirizxc]

/close #6015

(Edit by @davidhewitt) - See codspeed comment for benchmark improvements.

[ngoldbaum]

Did a quick pass and spotted a couple issues.

[ngoldbaum]

I looked over the FFI changes.

While we're touching longobject.rs, I'd appreciate it if you could make sure that your updates are ordered following CPython's Include/cpython/longobject.h as much as possible. For example, there are comments about skipping PyUnstable_Long_IsCompact and PyUnstable_Long_CompactValue, but those comments should follow the definition for PyLong_FromUnicodeObject because that's how the corresponding CPython header is organized in 3.14. Our bindings are also missing Py_ASNATIVEBYTES_ALLOW_INDEX, PyLong_IsPositive, PyLong_IsNegative, and PyLong_IsZero, which we should add.

It also looks like the PyLongLayout and PyLongExport structs are in cpython/longintrepr.h, so the definitions you're adding here should go into a new cpython/longintrepr.rs rust module, along with the C API definitions that are in the corresponding header.

The principle you should keep in mind when touching PyO3's FFI bindings is that it should be organized exactly like the upstream headers, as of the newest Python we support (3.14 right now, but it'll be 3.15 in the next few weeks).

[chirizxc]

I looked over the FFI changes.

While we're touching longobject.rs, I'd appreciate it if you could make sure that your updates are ordered following CPython's Include/cpython/longobject.h as much as possible. For example, there are comments about skipping PyUnstable_Long_IsCompact and PyUnstable_Long_CompactValue, but those comments should follow the definition for PyLong_FromUnicodeObject because that's how the corresponding CPython header is organized in 3.14. Our bindings are also missing Py_ASNATIVEBYTES_ALLOW_INDEX, PyLong_IsPositive, PyLong_IsNegative, and PyLong_IsZero, which we should add.

It also looks like the PyLongLayout and PyLongExport structs are in cpython/longintrepr.h, so the definitions you're adding here should go into a new cpython/longintrepr.rs rust module, along with the C API definitions that are in the corresponding header.

The principle you should keep in mind when touching PyO3's FFI bindings is that it should be organized exactly like the upstream headers, as of the newest Python we support (3.14 right now, but it'll be 3.15 in the next few weeks).

In branch 3.14, PyLong_FromNativeBytes, Py_ASNATIVEBYTES_BIG_ENDIAN, etc. , were moved from cpython/Include/longobject.h to cpython/longobject.h (see: 3.13* | 3.14*). Should I move them as well? I mean, do I need to fully synchronize the current files with branch 3.14?

[ngoldbaum]

Should I move them as well? I mean, do I need to fully synchronize the current files with branch 3.14?

Yes, that's generally what we do. It's hard to keep the FFI bindings perfectly up-to-date, so it's a chore for whoever needs to touch them.

[chirizxc]

I split up some of the extern_libpython! blocks so that the order remains the same as in the *.h files

[ngoldbaum]

I spotted some issues, see below.

[chirizxc]

@davidhewitt can we simply add a new PyLong* API to FFI via a separate PR until a definitive decision is made regarding the error type?

[Tpt]

@davidhewitt can we simply add a new PyLong* API to FFI via a separate PR until a definitive decision is made regarding the error type?

Not @davidhewitt but it sounds like a great idea to me

[davidhewitt]

Sorry for the delay. I've pushed a commit which just keeps the existing inconsistency for now where 3.13+ uses ValueError, and opened #6116 to fix this in the future.

I think let's not delay this further and proceed to ship it as part of 0.29.

[ngoldbaum]

There seem to be test failures on a debug Python build:

conversions::std::num::test_128bit_integers::test_u128_negative stdout ----
  
  thread 'conversions::std::num::test_128bit_integers::test_u128_negative' (6298) panicked at src/conversions/std/num.rs:937:13:
  assertion failed: err.is_instance_of::<exceptions::PyValueError>(py)

[davidhewitt]

I think it might just be an abi3 issue related to #6116, skipped the test on it for now and will re-attempt merge.

[chirizxc]

thanks