the CoVT code implemented on top of LLaVA

[FENGye-sjtu]

Hello, and thank you very much for your work!

I wanted to ask whether you have tried adding three tokens in LLaVA. If so, do you have any evaluation results on this setting?

Also, would it be possible for you to release the CoVT code implemented on top of LLaVA? I would greatly appreciate it.

[Wakals]

We have not tried aligning LLaVA with these three visual tokens at the same time, because we think aligning three types of tokens simultaneously is a challenge to the intrinsic performance of model. As we observed when utilizing Qwen as the base model, a sligt decline in performance occurred when attempting to align four types of tokens concurrently comparing to the three. So it is likely that LLaVA may face degration when it comes to the integration of SAM, Depth, and DINO.

Also, it is a good point to release the LLaVA training code, but I may need some time to clean our code before doing it.