Crash during long running proccess in asyncio.create_subprocess_shell on Windows

[esmorun]

Hi, I'm executing ffmpeg in an asyncio subprocess to convert a video. Example:

command = "ffmpeg  -i \"video.mp4\" -c:v hevc_nvenc -rc constqp -qp 27 -c:a libopus -b:a 96k \"video.mkv\""
proc = await asyncio.create_subprocess_shell(command, stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)

Long running processes will eventually crash the entire PyQt GUI. Sometimes it just crashes to desktop without any output in the Python shell, and sometimes this error message appears:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\qasync\_windows.py", line 163, in run
    events = self.__proactor.select(0.01)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\qasync\_windows.py", line 69, in select
    self._poll(timeout)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\qasync\_windows.py", line 99, in _poll
    with QtCore.QMutexLocker(self._lock):
RuntimeError: wrapped C/C++ object of type QMutex has been delet

The ffmpeg process continues as an orphaned process and finishes the job. Can anybody shed some light on what might be happening? I have tested the same code on Ubuntu and MacOS and have not encountered any issues.

I'm on PyQt 6.2.2 and Python 3.9.9.

Edit: I now understand that the error is happening because the QMutex object that is created on line 64 in windows.py is deleted before the loop at line 100 in the _poll function finishes. Just for fun I tried to remove the QMutexLocker by removing line 99:
with QtCore.QMutexLocker(self._lock):

...and this actually prevents my app from crashing, but I assume this is not a solution and the lock was there for a reason.

[awoimbee]

I have the same issue with a long running Future. I solved it by using threading.Lock() instead of QMutex. Not sure if this could have a negative impact.

Edit: I had a freeze using threading.Lock(), i switched to multiprocessing.Lock(), it works well.

[hosaka]

It is a common problem with Qt bindings for python, where a c++ object underneath gets deleted, and python then tries to use a reference to it. This needs a bit more digging and ideally a reproducible example. Perhaps a long sleep call, rather than an ffmpeg encode.

[dnadlinger]

I'm also seeing this using (Py)Qt6 and Python 3.11 in https://github.com/dnadlinger/artiq/tree/qt6. No leads as to a reproducible example, unfortunately. The same application works just fine when running against (Py)Qt5. Backtrace is the same as above, though I've also seen nonsensical overload resolution errors in the QMutexLocker constructor (with the same stack trace, presumably due to memory corruption).

I am not very familiar with the internals of Python's GC and how PyQt integrates with it – any ideas what kind of information might be helpful in tracking this down further?

[dnadlinger]

@esmorun / @awoimbee: You might want to check whether #87 fixes the issue.

[hosaka]

I will address PR later this week, thanks for looking into this.

…

On Mon, 2 Oct 2023, at 23:05, David Nadlinger wrote: @esmorun <https://github.com/esmorun> / @awoimbee <https://github.com/awoimbee>: You might want to check whether #87 <#87> fixes the issue. — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABJ6ZGZXWPOYYDFZVGGZPDX5LCZNAVCNFSM5JNAE222U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZUGMYDQMBZGE2A>. You are receiving this because you commented.Message ID: ***@***.***>