RuntimeError: CUDA out of memory

[naittaleb]

• Machine learning pipe cannot run on middle GPU machines
• Check if we can reduce graphical memory consumption

[naittaleb]

@Alienor134??

[Alienor134]

Piste:
model.eval() in romiseg.utils.train_from_dataset.evaluation
or model.train()
test if gradients are computed (shouldn't be)

[jlegrand62]

ça ne change rien dans mon cas...

Voici l'erreur sans les modifs suggérées (j'ai le même résultat avec les modifs, les numeros de ligne changent):

DEBUG    [romiscan] cuda:0 used for images segmentation from fileConfig
DEBUG    [romiscan] Model name:Resnetdataset_gl_png_896_896_epoch50 from fileConfig
[...]
DEBUG    [romiscan] Image segmentation by the CNN from fileConfig
  0%|                                                                                                                                   | 0/72 [00:00<?, ?it/s]tensor(2.6226)
  1%|█▋                                                                                                                         | 1/72 [00:00<00:51,  1.37it/s]tensor(2.6226)
  3%|███▍                                                                                                                       | 2/72 [00:01<00:51,  1.37it/s]tensor(2.6226)
  4%|█████▏                                                                                                                     | 3/72 [00:01<00:37,  1.83it/s]tensor(2.6226)
  6%|██████▊                                                                                                                    | 4/72 [00:02<00:40,  1.69it/s]tensor(2.6400)
  7%|████████▌                                                                                                                  | 5/72 [00:03<00:43,  1.55it/s]tensor(2.6226)
  8%|██████████▎                                                                                                                | 6/72 [00:03<00:43,  1.53it/s]tensor(2.6400)
 10%|███████████▉                                                                                                               | 7/72 [00:04<00:43,  1.49it/s]tensor(2.6400)
 11%|█████████████▋                                                                                                             | 8/72 [00:05<00:43,  1.48it/s]tensor(2.6400)
 12%|███████████████▍                                                                                                           | 9/72 [00:05<00:42,  1.48it/s]tensor(2.6400)
 14%|████████████████▉                                                                                                         | 10/72 [00:06<00:42,  1.47it/s]tensor(2.6400)
 15%|██████████████████▋                                                                                                       | 11/72 [00:07<00:43,  1.41it/s]tensor(2.6400)
 17%|████████████████████▎                                                                                                     | 12/72 [00:08<00:43,  1.39it/s]tensor(2.6400)
 18%|██████████████████████                                                                                                    | 13/72 [00:08<00:42,  1.38it/s]tensor(2.6400)
 19%|███████████████████████▋                                                                                                  | 14/72 [00:09<00:42,  1.35it/s]tensor(2.6400)
ERROR    [luigi-interface] [pid 20100] Worker Worker(salt=888313286, workers=1, host=RDP-M7520-JL, username=jonathan, pid=20100) failed    Segmentation2D(output_file_id=out, scan_id=, dilation=1, binarize=True, threshold=0.01, upstream_task=ImagesFilesetExists, model_fileset=ModelFileset, model_id=Resnetdataset_gl_png_896_896_epoch50, query={"channel": "rgb"}, Sx=896, Sy=896, labels=[], inverted_labels=["background"], resize=True) from fileConfig
Traceback (most recent call last):
  File "/home/jonathan/miniconda3/envs/ugly_scan_0.6/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/jonathan/miniconda3/envs/ugly_scan_0.6/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/jonathan/Projects/ROMI/Scan3D/romiscan/tasks/proc2d.py", line 184, in run
    model_file, self.resize)
  File "/home/jonathan/Projects/ROMI/Segmentation/romiseg/Segmentation2D.py", line 101, in segmentation
    outputs = evaluate(inputs, model_segmentation)  #output image
  File "/home/jonathan/Projects/ROMI/Segmentation/romiseg/utils/train_from_dataset.py", line 546, in evaluate
    pred = model(inputs)
  File "/home/jonathan/miniconda3/envs/ugly_scan_0.6/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jonathan/Projects/ROMI/Segmentation/romiseg/utils/segmentation_model.py", line 114, in forward
    x = self.upsample(x)
  File "/home/jonathan/miniconda3/envs/ugly_scan_0.6/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jonathan/miniconda3/envs/ugly_scan_0.6/lib/python3.7/site-packages/torch/nn/modules/upsampling.py", line 131, in forward
    return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners)
  File "/home/jonathan/miniconda3/envs/ugly_scan_0.6/lib/python3.7/site-packages/torch/nn/functional.py", line 3013, in interpolate
    scale_factor_list[0], scale_factor_list[1])
RuntimeError: CUDA out of memory. Tried to allocate 392.00 MiB (GPU 0; 3.95 GiB total capacity; 1.12 GiB already allocated; 51.56 MiB free; 2.17 GiB reserved in total by PyTorch)

Paquets conda:

torch                     1.5.0                    pypi_0    pypi
torchfile                 0.1.0                    pypi_0    pypi
torchnet                  0.0.4                    pypi_0    pypi
torchvision               0.6.0                    pypi_0    pypi

[fabfabBesnard]

Reproducible example:
data available on AWS server: ~/database_fabfab/test_Sangoku/Sangoku_mlpipe_1
72 images taken with RX-0 camera on the dev robot plant imager in Lyon of the 3D printed plant model generated from a Lpy virtual plant.

Run: docker container using the following image of plant-3d-vision (image built from branch Fix/VisualizationTaskForVirtualData):

REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
plant3dvision            Visu_VP             11b1d7e662b8        10 days ago         7.77GB

command line: romi_run_task AnglesAndInternodes --config database/ml_pipe_real_default.toml database/Sangoku_mlpipe_1

Result (run & error log):

INFO     [luigi-interface] [pid 1837] Worker Worker(salt=351061715, workers=1, host=9d60680
9db6a, username=scanner, pid=1837) running   Segmentation2D(scan_id=, upstream_task=Undisto
rted, model_fileset=ModelFileset, model_id=Resnet_896_896_epoch50, query={"channel": "rgb"}
, Sx=896, Sy=896, labels=[], inverted_labels=["background"], binarize=True, threshold=0.01,
 dilation=1) from fileConfig
^M  0%|          | 0/72 [00:00<?, ?it/s]^M  1%|▏         | 1/72 [00:00<00:57,  1.23it/s]^M  3%|▎         | 2/72 [00:01<01:01,  1.14it/s]^M  4%|▍         | 3/72 [00:02<00:59,  1.15it/s]^M  6%|▌         | 4/72 [00:03<00:58,  1.17it/s]^M  7%|▋         | 5/72 [00:04<00:56,  1.19it/s]^M  8%|▊         | 6/72 [00:05<00:54,  1.21it/s]^M 10%|▉         | 7/72 [00:05<00:53,  1.22it/s]^M 11%|█         | 8/72 [00:06<00:52,  1.23it/s]^M 12%|█▎        | 9/72 [00:07<00:51,  1.24it/s]^M 14%|█▍        | 10/72 [00:08<00:50,  1.24it/s]^M 15%|█▌        | 11/72 [00:09<00:49,  1.24it/s]^M 17%|█▋        | 12/72 [00:09<00:48,  1.24it/s]^M 18%|█▊        | 13/72 [00:10<00:47,  1.24it/s]^M 19%|█▉        | 14/72 [00:11<00:46,  1.24it/s]^M 21%|██        | 15/72 [00:12<00:45,  1.24it/s]^M 22%|██▏       | 16/72 [00:13<00:45,  1.24it/s]^M 24%|██▎       | 17/72 [00:13<00:44,  1.24it/s]^M 25%|██▌       | 18/72 [00:14<00:43,  1.24it/s]^M 26%|██▋       | 19/72 [00:15<00:42,  1.24it/s]^M 28%|██▊       | 20/72 [00:16<00:41,  1.24it/s]^M 29%|██▉       | 21/72 [00:17<00:40,  1.24it/s]^M 31%|███       | 22/72 [00:17<00:40,  1.24it/s]^M 32%|███▏      | 23/72 [00:18<00:39,  1.24it/s]^M 33%|███▎      | 24/72 [00:19<00:38,  1.24it/s]^M 35%|███▍      | 25/72 [00:20<00:37,  1.24it/s]^M 36%|███▌      | 26/72 [00:21<00:37,  1.24it/s]^M 38%|███▊      | 27/72 [00:21<00:36,  1.24it/s]^M 39%|███▉      | 28/72 [00:22<00:35,  1.25it/s]^M 40%|████      | 29/72 [00:23<00:34,  1.25it/s]^M 42%|████▏     | 30/72 [00:25<00:42,  1.01s/it]^M 43%|████▎     | 31/72 [00:25<00:36,  1.11it/s]^M 43%|████▎     | 31/72 [00:27<00:35,  1.14it/s]
ERROR    [luigi-interface] [pid 1837] Worker Worker(salt=351061715, workers=1, host=9d606809db6a, username=scanner, pid=1837) failed    Segmentation2D(scan_id=, upstream_task=Undistorted, model_fileset=ModelFileset, model_id=Resnet_896_896_epoch50, query={"channel": "rgb"}, Sx=896, Sy=896, labels=[], inverted_labels=["background"], binarize=True, threshold=0.01, dilation=1) from fileConfig
Traceback (most recent call last):
  File "/home/scanner/.local/lib/python3.7/site-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "/home/scanner/.local/lib/python3.7/site-packages/luigi/worker.py", line 133, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/scanner/.local/lib/python3.7/site-packages/plant3dvision/tasks/proc2d.py", line 201, in run
    images_segmented, id_im = segmentation(self.Sx, self.Sy, images_fileset, model_file)
  File "/home/scanner/plant-3d-vision/romiseg/romiseg/Segmentation2D.py", line 101, in segmentation
    outputs = evaluate(inputs, model_segmentation)  #output image
File "/home/scanner/plant-3d-vision/romiseg/romiseg/utils/train_from_dataset.py", line 546, in evaluate
    pred = model(inputs)
  File "/home/scanner/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/scanner/plant-3d-vision/romiseg/romiseg/utils/segmentation_model.py", line 116, in forward
    x = torch.cat([x, x_original], dim=1)
RuntimeError: CUDA out of memory. Tried to allocate 588.00 MiB (GPU 0; 11.17 GiB total capacity; 8.04 GiB already allocated; 517.31 MiB free; 10.21 GiB reserved in total by PyTorch)
INFO     [luigi-interface] Informed scheduler that task   Segmentation2D_896_896_True_666a04139b   has status   FAILED from fileConfig
INFO     [luigi-interface] Worker Worker(salt=351061715, workers=1, host=9d606809db6a, username=scanner, pid=1837) was stopped. Shutting down Keep-Alive thread from fileConfig
INFO     [luigi-interface] 
===== Luigi Execution Summary =====
Scheduled 10 tasks of which:
* 2 complete ones were encountered:
    - 1 ImagesFilesetExists(scan_id=, fileset_id=images)
    - 1 ModelFileset(scan_id=models)
* 2 ran successfully:
    - 1 Colmap(...)
    - 1 Undistorted(scan_id=, query={}, upstream_task=ImagesFilesetExists)
* 1 failed:
    - 1 Segmentation2D(...)
* 5 were left pending, among these:
    * 5 had failed dependencies:
        - 1 AnglesAndInternodes(...)
        - 1 OrganSegmentation(scan_id=, upstream_task=SegmentedPointCloud, eps=2.0, min_points=5)
        - 1 PointCloud(...)
        - 1 SegmentedPointCloud(scan_id=, upstream_task=PointCloud, upstream_segmentation=Segmentation2D, use_colmap_poses=True)
        - 1 Voxels(...)
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
 from fileConfig
cuda:0
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
tensor(2.6400)
Traceback (most recent call last):
 File "/home/scanner/.local/bin/romi_run_task", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/scanner/plant-3d-vision/romitask/bin/romi_run_task", line 212, in <module>
    subprocess.run(cmd, env={**os.environ, **env}, check=True)
  File "/usr/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['luigi', '--logging-conf-file', '/tmp/tmpxn9taa2b/logging_config.toml', '--module', 'plant3dvision.tasks.arabidopsis', 'AnglesAndInternodes', '--DatabaseConfig-scan', 'database/Sangoku_mlpipe_1', '--local-scheduler']' returned non-zero exit status 30.
(END)