Tends to produce garbled characters during inference

[chaixinning]

The model I reproduced tends to produce garbled characters during inference, like this, and then fails to generate the correct answer. However, your pre-trained model doesn't exhibit this problem. I also trained it for 2000 steps using only VQA stage, but the issue hasn't significantly improved. Have you encountered similar problems during training? How should you solve or improve them?

[DDDaar]

The model I reproduced tends to produce garbled characters during inference, like this, and then fails to generate the correct answer. However, your pre-trained model doesn't exhibit this problem. I also trained it for 2000 steps using only VQA stage, but the issue hasn't significantly improved. Have you encountered similar problems during training? How should you solve or improve them?

Hello! Could you share the configuration you used for each stage in your reproduction? I tried both the settings from the repository (6000, 3000, 3000, 4000) and the paper (4000, 3000, 3000, 5000), but in both cases, the model fails to output any visual tokens after training. Parameters are merged after 1 and 234 stage.

[Wakals]

May I ask the GPU you use? This is kind of weird, I am using (6000, 3000, 3000, 4000) and it works.
Best,

[Wakals]

Because I have heard the same situatioun on some GPUs different with the NVIDIA GPUs, so maybe this is a problem.