-Werror doesn't error out on DeprecationWarning for os.fork()

[fperrin]

Bug report

Bug description:

Hi,

With the recent change that Python now throws a warning for programs that fork() when there are more than one threads. I was hoping to use -Werror to find and pinpoint all such occurrences in my software.

Using python3.13 as installed by uv (cpython-3.13.4-linux-x86_64-gnu). With the following test code:

import os, sys, threading, time

t = threading.Thread(target=time.sleep, args=(1,), daemon=True)
t.start()

assert threading.active_count() == 2

pid = os.fork()
if pid < 0:
    print(f"fork failed: {pid=}")
elif pid == 0:
    print(f"after fork, I am {os.getpid()=}; {threading.active_count()=}")
    sys.exit(0)
print(f"successful fork; I am {os.getpid()=}, child is {pid=}; {threading.active_count()=}")

pid, _ = os.forkpty()
if pid < 0:
    print(f"forkpty failed: {pid=}")
elif pid == 0:
    print(f"after forkpty, I am {os.getpid()=}; {threading.active_count()=}")
    sys.exit(0)
print(f"successful forkpty; I am {os.getpid()=}, child is {pid=}; {threading.active_count()=}")

Running with -Walways gives me the expected warnings:

$ python3 -Walways ex.py 
after fork, I am os.getpid()=275006; threading.active_count()=1
/home/fred/demo/ex.py:8: DeprecationWarning: This process (pid=275004) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
successful fork; I am os.getpid()=275004, child is pid=275006; threading.active_count()=2
/home/fred/demo/ex.py:16: DeprecationWarning: This process (pid=275004) is multi-threaded, use of forkpty() may lead to deadlocks in the child.
  pid, _ = os.forkpty()
successful forkpty; I am os.getpid()=275004, child is pid=275007; threading.active_count()=2

However, running with -Werror, the code silently run (no exception raised, no warning printed):

$ python3 --version
Python 3.13.4
$ python3 -Werror ex.py 
successful fork; I am os.getpid()=275060, child is pid=275062; threading.active_count()=2
after fork, I am os.getpid()=275062; threading.active_count()=1
successful forkpty; I am os.getpid()=275060, child is pid=275063; threading.active_count()=2

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796
• [3.14] gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror (GH-136796) #140191

[ZeroIntensity]

Well, -Werror does error out on DeprecationWarning:

$ python3 -Werror -c "import warnings; warnings.warn('test', DeprecationWarning)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import warnings; warnings.warn('test', DeprecationWarning)
                     ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DeprecationWarning: test

But the problem here is that os.fork specifically isn't raising the exception, because the function that warns is explicitly clearing exceptions. I can't figure out if this is intentional; the relevant PR is #100229. At the very least I would expect that to be an unraisable exception rather than letting it pass silently.

cc @gpshead

[fperrin]

Yes should be a bit more accurate in the title.

[gpshead]

because the function that warns is explicitly clearing exceptions. I can't figure out if this is intentional; the relevant PR is #100229. At the very least I would expect that to be an unraisable exception rather than letting it pass silently.

It is intentional. I put that PyErr_Clear() in while developing the feature here - 1d3ec86

If you don't PyErr_Clear in this code path, the children inherit the state and you wind up with a warning being emitted from the child process as well as the parent which would be likely to cause a deadlock in a many children. The constraint on the function, even before that PR branch commit, was from the start of my design in the first PR commit always "Never leaves an error set."

This isn't a rather unique warning scenario. More of a using the best tool we had available at the time to get a message across without spamming it all the time or reinventing our own wheel on how to emit a message roughly once per incoming code path rather than every time (verbose) or once per process (doesn't highlight call paths) - something the warnings framework provides for. ex: this review comment #100229 (comment).

[gpshead]

The constraints on warn_about_fork_with_threads were later understandably changed slightly in some free-threading work. e728303 . It looks like we were already only calling it after fork so state shouldn't leak into the child. So the original reasoning on not leaving any state set related to that warning is maybe a little less strict now, so the new proposal to make that PyErr_Clear conditional to allow the warning state to be captured rather than only printed might be more feasible. https://github.com/python/cpython/pull/136796/files

I do wonder if allowing that could means the warning will wind up caught and swallowed by some frameworks rather than exposed. But that is basically on the framework authors / users if so.

[ZeroIntensity]

If you don't PyErr_Clear in this code path, the children inherit the state and you wind up with a warning being emitted from the child process as well as the parent which would be likely to cause a deadlock in a many children. The constraint on the function, even before that PR branch commit, was from the start of my design in the first PR commit always "Never leaves an error set."

Yeah, that makes sense, but I don't see why it has to fail silently. Wouldn't an unraisable exception be fine too? That clears the error, but also lets the user know about it.

[gpshead]

a exception wouldn't be squelched by the warnings stacklevel mechanism, it'd spam the output every time one happens instead of just highlighting each caller once upon first use from that caller.

[encukou]

The constraint on the function, even before that PR branch commit, was from the start of my design in the first PR commit always "Never leaves an error set."

I wonder if, after the free-threading change where the warning is emitted after fork, it would make sense to relax this: the code to get the number of threads should never leave an error set, but with -Werror, the DeprecationWarning itself gets raised normally.

After all, -Werror means the user is explicitly asking to turn nitpicks into failures.