Train speech task with IBQ, but the codebook may collapse to a few values(quant_loss -> 0)

[yufan-aslp]

We introduced IBQ into speech-related tasks to model the intermediate hidden layer features of the speech encoder. However, during training, the quantization loss suddenly drops to zero at a certain stage, resulting in codebook collapse. Have you encountered a similar issue on your side?

[ShiFengyuan1999]

Hi @yufan-aslp,
We did not observe this phenomenon during training on images. The sudden drop in quantization loss to zero may indicate that the model is overfitting to a very small subset of codes. To investigate this, you can print the selected indices during training and monitor the codebook usage. If this is indeed the case, we recommend modifying training hyperparameters such as the learning rate and loss weights.