performance gap with pytorch 0.4.1: mIoU 79.99

[Wangzhuoying0716]

Sorry to bother you! I tried to implement the training process using the seg_hrnet_w48_train_512x1024_sgd_lr1e-2_wd5e-4_bs_12_epoch484.yaml. But it came out that the validation mIoU for single-scale and no-flip is 79.99, almost 1% lower than your result shown(80.9). I wonder if this is normal variation or I did something wrong ?
Here is the training log:
https://pan.baidu.com/s/1utKUVuBEjDBtfgOk7A-5sQ 【passward: jckv】
Thank you!

[Hussainflr]

Hi, @Wangzhuoying0716 would you mind sharing your training settings,
I have been trying to train it, but it always gets stuck at this point

I don't know where I'm doing something wrong.

my training settings:
Pytorch 0.4.1
python 3.7
OpenCV 3.4.2.17
rest is according to requirements.txt

[Wangzhuoying0716]

@Hussainflr If there is no error message but just gets stuck, I guess it's because the memory is too full to keep on? You can check the memory usage when it is stuck to see whether it's almost full.
As for me, I just used 4xTitan Xp GPU cards and the setting in seg_hrnet_w48_train_512x1024_sgd_lr1e-2_wd5e-4_bs_12_epoch484.yaml.

[Wangzhuoying0716]

@Hussainflr I also used
Pytorch 0.4.1 and python 3.7

[sunke123]

@Wangzhuoying0716
Sorry for late reply.
I checked your log. When training on cityscapes, we use the "class balance" in the loss function.
You can change the config and set the class balance as True.

[whiteinblue]

Hi @sunke123, LOSS.CLASS_BALANCE defined in lib/config/default.py, but is there any place to use it?
and i found, the loss: criterion = CrossEntropy(ignore_label=config.TRAIN.IGNORE_LABEL,
weight=train_dataset.class_weights), use class weighs, which doesn't matter with CLASS_BALANCE param

[sunke123]

@whiteinblue
Class weights are used to train the model on Cityscapes. If you don't want to use it, you can change it in the yaml.

[whiteinblue]

@sunke123 thank you for your reply， and i confuse, it's where that the param LOSS.CLASS_BALANCE used ?

[XuShoweR]

Hi, @Wangzhuoying0716 would you mind sharing your training settings,
I have been trying to train it, but it always gets stuck at this point

I don't know where I'm doing something wrong.

my training settings:
Pytorch 0.4.1
python 3.7
OpenCV 3.4.2.17
rest is according to requirements.txt

Hello, i met same problem, have you solved it?

[sunke123]

@whiteinblue Class_balance is used in the loss function.

[purse1996]

So where that the param LOSS.CLASS_BALANCE used? I read the code, it seems that no matter what is CLASS_BALANCE param, the balance loss is used in the cityscape dataset.