Transformer training loss with validation loss

[wjc2830]

Hey there! Thanks a lot for your contribution to this awesome open-source project. I ran into a bit of a snag while training the transformer model and thought I'd reach out for some advice.

Specifically, I noticed that while the training loss is decreasing as expected, the validation loss starts to increase pretty early on, around epoch 2 or 3. Is this something normal? I also saw that in the checkpoint saving callback, the criterion is set to train/loss.

Just wanted to check if this is expected behavior or if I should be looking into it further. Thanks!

[RobertLuo1]

Hi, Thanks for your attention to our work. Indeed, we also notice this phenomenon. We suspect that the class-conditional dropout mechanism causes the inconsistency between training and validation loss. The metrics such as FID are more correlated to the training loss.

[wjc2830]

Got it, thx for reply!