gh-112075: Use atomic exchange in Py_SETREF so updating dict pointers doesn't race

[DinoV]

Updating the dict for an object uses Py_SETREF. This can race if multiple threads are attempting to do this and dec ref the original value multiple times. This uses _Py_atomic_exchange_ptr to update the existing value and get the previous value for a dec ref.

• Issue: Make dict objects thread-safe in --disable-gil builds #112075

[colesbury]

This changes every Py_SETREF to use an atomic exchange, not just for updating dict pointers. That seems too expensive

[DinoV]

This changes every Py_SETREF to use an atomic exchange, not just for updating dict pointers. That seems too expensive

It seems like having this be thread-safe by default and providing a way to opt out of it would be best? There's something like 280 references to this and it seems like around 100 of those are storing into memory which is likely to be shared - and that's just in CPython.

[colesbury]

Making Py_SETREF atomic is not generally sufficient because you still have a race between other loads and the Py_DECREF inside Py_SETREF. For updating dict pointers, we may delay deallocation with QSBR, but that's not a strategy we expect extensions to use. More commonly, we want updates and reads to be protected by a critical section.

I randomly sampled 10 out of the 262 usages of Py_SETREF in C files (excluding Docs/). Two were unsafe (in _warnings.c) and those need additional synchronization. Six were Py_SETREF on local variables and two more were safe for more subtle reasons (updating thread-local state and modification in tp_clear).

Sampled call sites

mathmodule.c `Py_SETREF(total, new_total);`
64: local

stringio.c `Py_SETREF(decoded, translated);`
104: local

bytesobject.c `Py_SETREF(bytes, bytes_subtype_new(type, bytes));`
133: local

floatobject.c `Py_SETREF(result, PyObject_CallOneArg((PyObject *)type, result));`
147: local

funcobject.c: `Py_SETREF(op->func_qualname, &_Py_STR(empty));`
153: shared, but safe (in tp_clear)

longobject.c `Py_SETREF(a, n)`
164: local

_warnings.c: `Py_SETREF(st->once_registry, registry)`
222: shared (unsafe)

_warnings.c: `Py_SETREF(st->filters, warnings_filters)`
224: shared (unsafe)

context.c `Py_SETREF(ctx->ctx_vars, new_vars);`
241: thread-local

import.c: `Py_SETREF(package, substr)`
255: local