CUDA Out-of-Memory on single L20 — How to train with 2 GPUs?

[pzwstudy]

Hi, thanks for your great work!
I’m trying to finetune the model using LoRA, but I keep running into CUDA OOM even on a single L20 (48GB) GPU.

❗ Error message
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 130.00 MiB.
GPU 0 has a total capacity of 44.53 GiB of which 85.94 MiB is free.
Including non-PyTorch memory, this process has 44.44 GiB memory in use.
Of the allocated memory 44.05 GiB is allocated by PyTorch, ...
If reserved but unallocated memory is large try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

💻 Hardware

2 × NVIDIA L20 (48GB)

CUDA 12.x

PyTorch version: (same as your install guide)

accelerate version: latest

📦 Command I used
accelerate launch train.py --config ./train_configs/train_lora.yaml

❓ My question

Since one L20 is not enough, I would like to run training across two GPUs.

Could you please provide guidance on:

How to configure accelerate to enable 2-GPU training for this project?

Are there additional configs (e.g., device_map, distributed_type, gradient checkpointing, batch size limits) that I should modify for multi-GPU LoRA finetuning?

Any known memory optimizations for Qwen2.5-Image LoRA training?

🙏 Thanks!

Any help would be greatly appreciated!

[ValentinKovalev]

Hello!

Thanks for raising this and for the detailed information.

We have not yet tested the LoRA trainer specifically on NVIDIA L20 GPUs, so we cannot speak to model-specific behavior on that hardware. In terms of VRAM capacity, a single L20 (48 GB) should generally be sufficient for our default LoRA training configurations, but depending on the base model and batch settings you may still hit memory limits.

The trainer does support multi-GPU training via accelerate. With the current configuration you can launch distributed training across multiple GPUs simply by listing them in your accelerate config or using the standard distributed launch flags. The underlying implementation is built on Hugging Face accelerate and should behave consistently across supported devices.

In practice, we have successfully run training on:

• consumer GPUs such as RTX 4090 (from to 4 GPUS)
• data-center GPUs including H100 in multi-GPU setups

These runs confirm that the multi-GPU path and memory management are functional with our configs.

For L20 specifically:

• you may try enabling gradient checkpointing and reducing batch size to lower peak memory usage
• ensure the latest CUDA drivers and PyTorch build are installed
• check that accelerate config correctly reflects multiple GPUs (device_map / distributed_type)