Question about BEV loss

[LaBaZh]

Thanks for the great work!

I read the training code and found that the BEV loss is calculated at each step once enabled, unlike the multi-view reconstruction loss that is applied in an interleaved manner. Are there any experimental results to try the interleaved way for BEV loss as well? Or I misunderstood the code :)

Best,
Li

[Haochen-Wang409]

Could you be more specific? I think the loss computation here means they are the same, as ross_enable=True and ross_multi_task=True.

[LaBaZh]

When computing compute_vm_loss, once the mask is all zero, no vm_loss will be computed, which aligns with the interleaved reconstruction task mentioned in the paper.
When computing compute_vm_loss_bev, it seems that whether the mask is on or not, the bev loss will always be computed, which makes the bev loss involved in each step.

[Haochen-Wang409]

Yes, this is correct. Using all visible frames to reconstruct BEV images is also a meaningful pre-text task.

[LaBaZh]

So vm_loss_bev is used for training through the whole process, unlike the vm_loss, is that correct? Would it be beneficial to interleave the BEV task? Since the reconstruction task might lead to unstable in training as mentioned in the paper.