Reproducing Nr3D results in Table 6.

[haomengz]

Hi,

I am trying to train from scratch to reproduce the 49.4 overall accuracy on Nr3D, as is reported in Table 6. However, I could only get around 46.5 under the settings of provided config (changing data to nr3d and also set the use_gt_proposal flag). Could you provide more details on the training settings to reproduce your result on Nr3D? Thanks!

[haomengz]

When testing your provided checkpoint for Nr3D, I could only get around 36 overall accuracy. I am running test.py with use_gt_proposal=True and then running the evaluate.py. Am I missing something?

[JiayuXu829]

I have the same problem, have you resolve it?

[JiayuXu829]

(m3drefclip) lm3@admin123-ESC8000A-E11:~/projects/M3DRef-CLIP$ python evaluate.py data=nr3d pred_path=/home/lm3/projects/M3DRef-CLIP/predictions/Nr3d_gt_bbox/val data.evaluation.split=val
Initializing ground truths: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7485/7485 [03:33<00:00, 35.05it/s]
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7485/7485 [00:00<00:00, 10857.99it/s]

easy hard view-dep view-indep overall

42.5 30.8 32.4 38.6 36.6

This is my results.

[eamonn-zh]

Hi @haomengz and @JiayuXu829 , thanks for pointing this out and sorry for the late reply. Looks like the provided checkpoint for Nr3D dataset has issues. I attached the wandb log for Nr3D training we did before. We are currently investigating it and will reply to you later. Sorry for the confusion.