SigLIP2 in UniPic-1

[wavinflaghxm]

Similar to #9, I'm also confused why SigLIP2 isn't used in the code.

The paper says, 'Image understanding is performed using a SigLIP2 encoder to extract rich visual features, which are subsequently passed to an LLM for autoregressive text generation.'

However, the code in image2text.py uses VAE+MAR.
https://github.com/SkyworkAI/UniPic/blob/main/UniPic-1/scripts/image2text.py#L64

Also, the loss calculation does not involve the SigLIP2.
https://github.com/SkyworkAI/UniPic/blob/main/UniPic-1/src/models/skywork_unipic_dev.py#L334

[Orlando-CS]

The released HF weights already include SigLIP2, so you don’t need to load it separately. The code looks VAE+MAR-only, but SigLIP2 is fused in the hf checkpoint. We’ve been busy with Unipic3, so haven’t cleaned that part up yet.

[wavinflaghxm]

Thank you for the reply.
So, can I replace the vae+mar part in image2text.py with siglip2?

By the way, I'm still very confused. Why (according to the paper) siglip2 is used as the visual encoder during training, but using vae+mar in image2text.py still works?