Ambiguous return values

[iritkatriel]

Some C API functions can return a value which may or may not indicate an error, and require disambiguation with PyErr_Occurred(). They should be replaced by new versions with unambiguous return values.

Ambiguous return values

PyCapsule_GetDestructor
PyCapsule_GetContext
PyCapsule_GetName

PyDict_GetItemWithError

PyFloat_AsDouble
PyFloat_Unpack2
PyFloat_Unpack4
PyFloat_Unpack8

PyIter_Next

PyLong_AsLong
PyLong_AsLongAndOverflow
PyLong_AsLongLong
PyLong_AsLongLongAndOverflow
PyLong_AsSsize_t
PyLong_AsUnsignedLong
PyLong_AsSize_t
PyLong_AsUnsignedLongLong
PyLong_AsUnsignedLongMask
PyLong_AsDouble
PyLong_AsVoidPtr

PyMarshal_ReadLongFromFile
PyMarshal_ReadShortFromFile

PyNumber_AsSsize_t

PyUnicode_Compare

no return value, but can set an exception that can only be checked with PyErr_Occurred:

PyMarshal_WriteObjectToFile

[iritkatriel]

PyUnicode_Compare is an interesting one. It has only 3 possible return values: -1, 0 and 1. And -1 is ambiguous.

[iritkatriel]

We will track work on this here: python/cpython#105201

[encukou]

Maybe ambiguous return values are OK themselves, but it's the way we disambiguate that needs improving?

Maybe we should look focus on PyErr_Occurred itself. Why is it bad? Is it slow? Does it not work well with alternative implementations? Does it prevent optimizations?

Why is an int *err argument better than a variable in the interpreter state?

[erlend-aasland]

Maybe we should look focus on PyErr_Occurred itself.

IMO that's a related, but slightly different issue. I'd consider creating a separate issue for those problems.

[iritkatriel]

It' not a different issue, it's this issue. And I'm sure we've already discussed it elsewhere.

Ambiguous return values force us to access globals state. This has two problems:

1. It forces all implementations of the API to maintain this global state, whether it makes sense for them or not.

2. The usual problem with global state: you can never be sure where the value came from. You think PyIter_Next set the error indicator, but it may have been something else in the last iteration of the loop.

[encukou]

It forces all implementations of the API to maintain this global state, whether it makes sense for them or not.

Trouble is, they need to do that anyway if PyErr_Occurred stays in the API. Which it needs to, unless we overhaul all of error handling. Ambiguous return values themselves aren't the real problem.

you can never be sure where the value came from

Yup. Users ideally need assert !PyErr_Occurred() (or save/restore it). But that's a relatively minor papercut, so, we run into a general problem with incremental improvement: if the next small step is too small relative to the disruption (deprecation cycles or proliferation of similar API), it doesn't make sense to do it (yet). Perfect is the enemy of good :(

[iritkatriel]

we run into a general problem with incremental improvement

We should have an issue for that, and think how we can avoid being stuck with every mistake forever in the new api.

[iritkatriel]

It forces all implementations of the API to maintain this global state, whether it makes sense for them or not.

Trouble is, they need to do that anyway if PyErr_Occurred stays in the API. Which it needs to, unless we overhaul all of error handling.

I don't see why. If all functions are explicit in their output on whether there was an error or not, then why should users need PyErr_Occurred() to be part of the API? Internally the interpreter can continue to have the error in globals state, but it doesn't have to expose that through the API.

[iritkatriel]

We could change the title of this issue to "PyErr_Occurred() should not be part of the API".