Client gets stuck if GPU runs out of memory

[alahiff]

Message from Vignesh:

I think I have chanced upon a very interesting bug. So I am doing some ML training on CSD3 and due to the size of these models and the datasets I end up not having sufficient memory on the GPU and the run crashes, however the run instance is not stopping and it is being shown as running on simvue and will keep maintaining the run on the gpu within the cluster

I think this is a Python process using GPUs (i.e. the client isn't monitoring an external application).

[alahiff]

Actually, just raising an exception triggers the same issue. With this example, the run is marked as failed but the client continues running:

import os
import random
import time
from simvue import Run

if __name__ == "__main__":
    with Run() as run:
        # Initialisation
        run.init(name='crash-test-%d' % time.time(),
                 tags=['tests'])

        time.sleep(10)
        raise Exception("I have crashed")

Control-c results in:

Exception ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1567, in _shutdown
    lock.acquire()
KeyboardInterrupt:

[alahiff]

Also found that running pytest tests/refactor/test_run_class.py hangs once the tests finish. Not sure if this is related.

[kzscisoft]

Closed by #518

[alahiff]

Unfortunately the updates to the Simvue Python API don't fix the problem Vignesh is seeing, so reopening.

[alahiff]

Actually, Vignesh has found that standard RuntimeError exceptions cause the client to hang on CSD3, e.g.:

Traceback (most recent call last):
  File "/home/ir-gopa2/rds/rds-ukaea-ap001/ir-gopa2/Code/Neural_PDE/Expts/Advection_DeepOnet.py", line 264, in <module>
    train_loss, test_loss = train_one_epoch_don(model, train_loader, test_loader, loss_func, optimizer)
  File "/home/ir-gopa2/rds/rds-ukaea-ap001/ir-gopa2/Code/Neural_PDE/Expts/Advection_DeepOnet.py", line 222, in train_one_epoch_don
    im = model(br, tr)
  File "/home/ir-gopa2/miniconda3/envs/torch_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ir-gopa2/miniconda3/envs/torch_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/rds/project/rds-aSo1XX0UOlw/ir-gopa2/Code/Neural_PDE/Models/DeepOnet.py", line 104, in forward
    x = torch.sum(branch_output * trunk_output, dim=-1)
RuntimeError: The size of tensor a (200) must match the size of tensor b (10000) at non-singleton dimension 1

[alahiff]

I can't reproduce this by explicitly raising a RuntimeError.

[kzscisoft]

@alahiff I am going to need a minimal example in order to investigate this

[kzscisoft]

This issue is not related to the Run itself but rather how the user uses it, if not using the context manager then Run cannot handle exception raises so its threads continue running causing it to hang.

There is no solution to this from the API side, we should strongly encourage users to use the context manager (unless they have a Jupyter notebook, in the case they have to handle the exception themselves with:

run = Run()

try:
     # Do things which could raise exceptions
except Exception as e:
     run.close()
     raise e