Address 'RuntimeError: lock is bound to a different event loop' in modal

[grzegorz-roboflow]

Description

Observing below stack trace appearing on modal in some rare condition when container is reused (possibly tied to gpu snapshot)

Traceback (most recent call last):
  File "/pkg/modal/_runtime/container_io_manager.py", line 948, in handle_input_exception
    yield
  File "/pkg/modal/_container_entrypoint.py", line 254, in run_input_sync
    values = io_context.call_function_sync()
  File "/pkg/modal/_runtime/container_io_manager.py", line 225, in call_function_sync
    expected_value_or_values = self.finalized_function.callable(*args, **kwargs)
  File "/app/inference/core/interfaces/webrtc_worker/modal.py", line 253, in rtc_peer_connection_modal
    asyncio.run(
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/app/inference/core/interfaces/webrtc_worker/webrtc.py", line 917, in init_rtc_peer_connection_with_loop
    await peer_connection.setRemoteDescription(
  File "/usr/local/lib/python3.10/dist-packages/aiortc/rtcpeerconnection.py", line 1035, in setRemoteDescription
    await asyncio.gather(*coros)
  File "/usr/local/lib/python3.10/dist-packages/aiortc/rtcpeerconnection.py", line 177, in add_remote_candidates
    await asyncio.gather(*coros)
  File "/usr/local/lib/python3.10/dist-packages/aiortc/rtcicetransport.py", line 305, in addRemoteCandidate
    await self._connection.add_remote_candidate(
  File "/usr/local/lib/python3.10/dist-packages/aioice/ice.py", line 496, in add_remote_candidate
    mdns_protocol = await get_or_create_mdns_protocol(self)
  File "/usr/local/lib/python3.10/dist-packages/aioice/ice.py", line 54, in get_or_create_mdns_protocol
    async with _mdns.lock:
  File "/usr/lib/python3.10/asyncio/locks.py", line 14, in __aenter__
    await self.acquire()
  File "/usr/lib/python3.10/asyncio/locks.py", line 106, in acquire
    fut = self._get_loop().create_future()
  File "/usr/lib/python3.10/asyncio/mixins.py", line 30, in _get_loop
    raise RuntimeError(f'{self!r} is bound to a different event loop')
RuntimeError: <asyncio.locks.Lock object at 0x2aa01b9ac6d0 [locked]> is bound to a different event loop

Type of change

• Bug fix (non-breaking change which fixes an issue)

How has this change been tested, please provide a testcase or example of how you tested the change?

CI passing
Tested manually

Any specific deployment considerations

N/A

Docs

N/A