Many logging errors

[mesmith75]

I am suffering from many of these:

  File "/cvmfs/ganga.cern.ch/Ganga/install/dirac_test/lib/python3.11/site-packages/ganga/GangaCore/Utility/logging/__init__.py", line 152, in handleError
    private_logger.error('Error writing to the log file.')
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1518, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1644, in handle
    self.callHandlers(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
    hdlr.handle(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 978, in handle
    self.emit(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/handlers.py", line 77, in emit
    self.handleError(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/dirac_test/lib/python3.11/site-packages/ganga/GangaCore/Utility/logging/__init__.py", line 152, in handleError
    private_logger.error('Error writing to the log file.')
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1518, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1644, in handle
    self.callHandlers(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
    hdlr.handle(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 978, in handle
    self.emit(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/handlers.py", line 77, in emit
    self.handleError(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/dirac_test/lib/python3.11/site-packages/ganga/GangaCore/Utility/logging/__init__.py", line 152, in handleError
    private_logger.error('Error writing to the log file.')
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1518, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1644, in handle
    self.callHandlers(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
    hdlr.handle(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 978, in handle
    self.emit(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/handlers.py", line 77, in emit
    self.handleError(record)
  File "/cvmfs/ganga.cern.ch/Ganga/install/dirac_test/lib/python3.11/site-packages/ganga/GangaCore/Utility/logging/__init__.py", line 152, in handleError
    private_logger.error('Error writing to the log file.')
  File "/cvmfs/ganga.cern.ch/Ganga/install/micromamba_el9/envs/diracenv/lib/python3.11/logging/__init__.py", line 1517, in error
    if self.isEnabledFor(ERROR):
       ^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

[Rakesh-46-VR]

Can you explain the problem in detail ?

[egede]

@mesmith75 Do you have any way of reproducing this problem?

[mesmith75]

Not reliably. It occurs when I am monitoring a large number of jobs on the mass storage file system. I think it gets triggered by a stale file handle initially and then the logging ends up in a recursive loop.

My guess is it is a bit a problem of the file system being slow, the asyncio trying to do too much at one time and a genuine bug in how errors are raised.