issue about the latent token

[threegold116]

How did you incorporate the generated visual tokens into the input? Just use the id and the embedding layer?

[Wakals]

Yes, it's the way we do

[threegold116]

Yes, it's the way we do

If this is the case, the latent visual token at the input side is the same for different questions. So how does the model use this latent token to adapt and influence subsequent answer generation?

[Wakals]

These tokens are technically conditioned on the previous tokens, so in the subsequent process, the latents of these tokens are influenced by the question tokens. Therefore, the last layer's latents are different.

[threegold116]

But why not directly append the last hidden state corresponding to the latent token to the input layer, like Mirage or Coconut?

[Wakals]

Sure, we wanted to keep the consistency for the four stages, so we did what we did. But it seems to be a great point, I'll try this out, and give the difference! Thank you!

[windygoo]

These tokens are technically conditioned on the previous tokens, so in the subsequent process, the latents of these tokens are influenced by the question tokens. Therefore, the last layer's latents are different.

the last layer's latents may be homogeneous because they are fed into the lm_head to predict the same token index?

[wnn2000]

These tokens are technically conditioned on the previous tokens, so in the subsequent process, the latents of these tokens are influenced by the question tokens. Therefore, the last layer's latents are different.

the last layer's latents may be homogeneous because they are fed into the lm_head to predict the same token index?

Specifically, I cannot quite understand how multiple hidden states (e.g., 8 tokens) can all be predicted as the same token (e.g., <|sam_pad|>), while still encoding different information that is later used to recover different segmentation masks.

Another question is about the inference process. Since the authors do not seem to enforce the model to output a fixed number of these tokens, what happens if the model does not generate all of them? For example, if the model only outputs 4 <|sam_pad|> tokens instead of 8 and then proceeds to generate other tokens, how is this situation handled?

I hope the author can help us. @Wakals

[Wakals]

The 8 <|sam_pad|> are the same before training. However, along with the Hungarian matching loss, and there is a projection layer for these tokens, they will converge into different continuous tokens, and for different masks. The reason we use the same <|sam_pad|> is because it will be easier if we want to modify the number of the segmentation token on the top of the current ebedding architecture and the current checkpoint. Additionally, the reason we want to apply the lm_head after these visual tokens is because we want the model can automatically predict these tokens and use them. And we want to keep a small amount of discrete feature within these tokens because these tokens are like the bridging tokens between image tokens (fully continuous) and text tokens (fully discrete). (They are like 1.5-dimentional tokens, capturing image features and dilever to the text space.)

For the another question, I would like to say we don't really care about the number of the output tokens, because we don't really need to decode them. Therefore, if the model only output 4 sam tokens, it means this is a better reasoning path for the model on the current question.

Best,

[wnn2000]

Hi, @Wakals , thanks for your reply.

Can I understand it as follows: at the positions corresponding to the latent CoT, an additional regularization constraint on the latent states is imposed on top of the token-generation (here you want to generate special token like <|sam_pad|>) cross-entropy loss? Since at these positions two losses are applied simultaneously, I want to confirm whether my understanding is correct.

Thank you very much!