Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError: Triton Error [CUDA]: device kernel image is invalid #112

Open
junsukha opened this issue Dec 12, 2024 · 1 comment
Open

RuntimeError: Triton Error [CUDA]: device kernel image is invalid #112

junsukha opened this issue Dec 12, 2024 · 1 comment
Labels
bug Something isn't working documentation Improvements or additions to documentation

Comments

@junsukha
Copy link

junsukha commented Dec 12, 2024 •
edited
Loading

I'm fine-tuning LoRA.

Running bash ./demos/fine_tuner/run.bash -c ./demos/fine_tuner/configs/lora.yaml -n 1 gives me the error:

torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Triton Error [CUDA]: device kernel image is invalid

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

This seems like Triton version error. Which version should I use?
The current version installed is 3.0.0.

I have torch==2.4.1+cu118 installed.

EDIT

Have done some research and added
export TRITON_PTXAS_PATH=/mnt/singularity_home/jsha/repos/mochi/.venv/lib/python3.10/site-packages/torch/bin/ptxas
in run.bash at the beginning.

Now, the code outputs

bash ./demos/fine_tuner/run.bash -c ./demos/fine_tuner/configs/lora.yaml -n 1
Starting training with 1 GPU(s), mode: single_gpu
Using config: ./demos/fine_tuner/configs/lora.yaml
model=weights/dit.safetensors, optimizer=, start_step_num=0
Found 209 training videos in videos_original_prepared
Loaded 209/209 valid file pairs.
Loading model
Training type: LoRA
Attention mode: sdpa
Loading eval pipeline ...
Timing load_text_encoder
Timing load_vae
Stage                   Time(s)    Percent
load_text_encoder          1.31     29.04%
load_vae                   3.19     70.96%
  0%|                                                                                                                                                     | 0/10000 [00:00<?, ?it/s]W1211 23:44:10.275000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/47_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:24.530000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/46_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:26.699000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/45_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:30.572000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/44_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:32.834000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/43_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:35.093000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/42_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:39.248000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/41_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:41.469000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/40_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:43.660000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/39_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:45.884000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/38_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:48.077000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/37_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:50.305000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/36_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:54.549000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/35_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:56.746000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/34_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:44:58.941000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/33_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:01.177000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/32_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:03.396000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/31_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:08.956000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/30_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:11.190000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/29_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:13.913000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/28_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:16.177000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/27_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:18.405000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/26_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:20.615000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/25_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:23.656000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/24_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:25.893000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/23_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:28.079000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/22_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:30.291000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/21_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:32.484000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/20_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:35.927000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/19_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:38.144000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/18_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:40.310000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/17_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:42.467000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/16_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:45.280000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/15_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:49.483000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/14_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:51.684000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/13_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:53.888000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/12_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:56.112000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/11_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:45:59.584000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/10_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:01.824000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/9_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:04.015000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/8_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:06.221000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/7_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:09.727000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/6_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:11.888000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/5_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:14.056000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/4_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:16.245000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/3_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:18.413000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/2_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:21.911000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/1_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:25.795000 139624957605632 torch/fx/experimental/symbolic_shapes.py:4449] [2/0_1] rindex is not in var_ranges, defaulting to unknown range.
  0%|                                                                                          | 1/10000 [09:37<1603:16:30, 577.24s/it, train/epoch=1, train/loss=0.659, train/lr=0]W1211 23:46:32.766000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/48_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:34.635000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/48_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:46.415000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/49_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:48.637000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/49_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:55.514000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/50_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:46:57.163000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/50_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:03.530000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/51_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:05.151000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/51_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:10.161000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/52_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:11.788000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/52_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:17.140000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/53_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:18.776000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/53_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:24.169000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/54_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:25.787000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/54_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:33.593000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/55_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:35.223000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/55_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:42.318000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/56_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:43.949000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/56_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:49.060000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/57_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:50.691000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/57_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:57.097000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/58_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:47:58.724000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/58_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:03.880000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/59_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:05.504000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/59_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:12.502000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/60_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:14.119000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/60_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:19.372000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/61_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:20.986000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/61_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:30.347000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/62_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:33.338000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/62_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:38.398000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/63_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:40.031000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/63_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:46.404000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/64_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:48.035000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/64_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:53.782000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/65_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:48:55.412000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/65_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:00.442000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/66_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:02.087000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/66_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:08.445000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/67_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:10.055000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/67_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:16.195000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/68_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:17.793000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/68_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:22.781000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/69_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:27.591000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/69_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:32.604000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/70_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:34.793000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/70_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:39.900000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/71_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:41.543000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/71_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:48.148000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/72_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:49.783000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/72_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:55.223000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/73_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:49:56.849000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/73_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:01.941000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/74_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:04.415000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/74_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:09.464000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/75_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:11.076000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/75_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:16.115000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/76_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:17.746000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/76_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:27.005000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/77_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:28.627000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/77_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:35.040000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/78_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:36.665000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/78_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:43.488000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/79_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:45.160000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/79_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:51.455000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/80_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:53.109000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/80_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:50:58.738000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/81_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:00.369000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/81_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:07.567000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/82_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:09.231000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/82_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:15.446000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/83_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:17.102000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/83_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:25.072000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/84_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:27.565000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/84_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:38.907000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/85_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:40.688000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/85_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:52.984000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/86_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:54.612000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/86_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:51:59.941000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/87_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:01.603000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/87_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:07.989000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/88_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:09.665000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/88_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:16.249000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/89_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:17.889000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/89_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:23.702000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/90_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:25.337000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/90_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:30.836000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/91_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:33.807000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/91_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:39.261000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/92_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:40.892000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/92_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:47.387000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/93_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:49.007000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/93_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:54.426000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/94_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:52:56.048000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/94_1] rindex is not in var_ranges, defaulting to unknown range.
W1211 23:53:05.399000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/95_1] xindex is not in var_ranges, defaulting to unknown range.
W1211 23:53:07.022000 139633101632128 torch/fx/experimental/symbolic_shapes.py:4449] [2/95_1] rindex is not in var_ranges, defaulting to unknown range.
  3%|██▌                                                                                    | 300/10000 [45:39<12:45:53,  4.74s/it, train/epoch=2, train/loss=0.59, train/lr=0.0002]

It does generate samples while training. I set sample interval to 300. So looks the training is happening. Not sure if it's really learning something though yet.

Should I be concerned about those warnings?
@vedantroy
Copy link
Contributor

Those warnings are fine to ignore.

@ajayjain ajayjain added bug Something isn't working documentation Improvements or additions to documentation labels Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ajayjain @vedantroy @junsukha