Applying IBQ to Multi-Mcale VQVAE in VAR

[RohollahHS]

Hi, Great work.

Can IBQ be applied to the Multi-Scale VQVAE in VAR? and did you try it?

Thanks

[ShiFengyuan1999]

Hi @RohollahHS, thanks for your interest in our work.

IBQ aims to render all codes differentiable and update the entire codebook in each backward process to ensure consistency with the distribution of the visual encoder by index backpropagation. It is compatible with other widely used quantization tricks like multi-scale and residual quantization. We did not try these for simplicity. It is easy to combine IBQ with these methods, by replacing the indices achieved by these methods with

SEED-Voken/src/IBQ/modules/vqvae/quantize.py

Line 448 in 3c58097

one_hot = hard_one_hot - soft_one_hot.detach() + soft_one_hot

in IBQ.

[RohollahHS]

Thanks for the information and your reply.