About the performance

[fjzpcmj]

Hello，thanks for your great work. I repeat your experiments report in readme.md. Trains the fbocc-flow.py and voxelsplat-r50.py with 8 GPUs. However, I cannot get the performance reported in the readme, The performance of my repetition is as follows::

fbocc-flow: 'RayIoU': 0.3683420040726819, 'mAVE': 0.4596367791978842, 'final_Occ_Score': 0.3855441257456253
voxelsplat-r50: 'RayIoU': 0.3693224973566454, 'mAVE': 0.5159745717857621, 'final_Occ_Score': 0.38079279044240466

Is There Any Problem With My Reproduction?

[ZZY816]

Hello, thank you for your detailed feedback and for reproducing our experiments!
Could you please share how many iterations you trained for, and if possible, the training loss curves?
Sometimes the convergence may be a bit unstable, so I’d suggest testing several checkpoints to see if the performance varies.
In my runs on 8×3090 GPUs, the mAVE is always below 0.32 after convergence.

[fjzpcmj]

Thanks for your reply. I train the models with 84350 iterations. Here is my training logs. I repeat the experiments several times, but The performance remains stable across multiple experiments.
fbocc.log
voxelsplat.log

[ZZY816]

Thank you for the follow-up.
I will provide our training checkpoints and logs soon to help verify whether the discrepancy may come from the evaluation process.
In the meantime, please try testing several checkpoints other than latest.pth to see if the performance fluctuation might be caused by unstable convergence.

[fjzpcmj]

Thanks very much for providing training checkpoints and logs.
Here is different checkpoints results of my another repetition（I save checkpoint every 12 epoch）:

fbocc-flow:
iter_84384: 'RayIoU': 0.3617900100687457, 'mAVE': 0.5555162320961031, 'final_Occ_Score': 0.3700593858522609
iter_63288: 'RayIoU': 0.35400335911886405, 'mAVE': 0.0019347289417202711, 'final_Occ_Score': 0.41840955031280563
iter_42192: 'RayIoU': 0.33198306553719337, 'mAVE': 0.002854098142922746, 'final_Occ_Score': 0.3984993491691818

voxelsplat-r50:
iter_84384: 'RayIoU': 0.37060707616880634, 'mAVE': 0.528432099509049, 'final_Occ_Score': 0.38070315860102083
iter_63288: 'RayIoU': 0.35956061010751467, 'mAVE': 0.5171833294002774, 'final_Occ_Score': 0.3718862161567355
iter_42192: 'RayIoU': 0.35864110562416956, 'mAVE': 0.5358279848625069, 'final_Occ_Score': 0.3691941965755019

[ZZY816]

Thanks a lot for sharing the checkpoint results.

I think I’ve figured out where the issue might be. An mAVE (mean average velocity error) as low as 0.0019 is essentially impossible in our setup—it would mean the predicted flow is almost perfectly identical to the GT flow. This strongly suggests that, during validation, the GT flow is nearly all zeros due to some error.

Could you please:

1. Inspect the downloaded occ_flow annotations for the val split to see if the flow fields are zero (or near-zero) everywhere.

2. Double-check the validation pipeline to ensure the gt_flow isn’t being zeroed out by mistake.

[fjzpcmj]

Thanks for your kindly reply. Could you share how to generate data/nuscenes/my_gts.gz? I can't find how to generate my_gts.gz in ReadMe. I use ray_iou/ray.py to generate my_gts.gz with following setting. Maybe something wrong with my generated my_gts.gz?

parser.add_argument("--data-info", default='/home/work_dir/occupancy/VoxelSplat-main/data/nuscenes/bevdetv2-nuscenes_infos_val.pkl')
parser.add_argument("--pred-root", default='/home/work_dir/occupancy/VoxelSplat-main/data/nuscenes/gts')
parser.add_argument("--output-dir", default='/home/work_dir/occupancy/VoxelSplat-main/data/nuscenes')
parser.add_argument("--output-name", default='my_gts.gz')
parser.add_argument("--out-type", default='labels.npz')
args = parser.parse_args()

torch.random.manual_seed(0)
np.random.seed(0)

main(args)

[fjzpcmj]

I have found why the mAVE is close to zero. There is a bug in ray_iou/ray.py to generate my_gts.gz. In L336 of ray.py, the velocity gt is always zero.

VoxelSplat/ray_iou/ray.py

Line 336 in 279de66

flow_pred = flow_map

.

I evaluate the predictions with the corrected gts. Here is the results:

fbocc-flow:
iter_84384: 'RayIoU': 0.3617900100687457, 'mAVE': 0.5296566001556084, 'final_Occ_Score': 0.37264534904631036
iter_63288: 'RayIoU': 0.35400335911886405, 'mAVE': 0.5029214675374732, 'final_Occ_Score': 0.36831087645323035

voxelsplat-r50:
iter_84384: 'RayIoU': 0.37060707616880634, 'mAVE': 0.30693271308025405, 'final_Occ_Score': 0.4028530972439003
iter_63288: 'RayIoU': 0.35956061010751467, 'mAVE': 0.3105972687431061, 'final_Occ_Score': 0.3925448222224526

I can reproduce the performance of voxelsplat. But it seems that the RayIoU of fbocc is closed to that of voxelsplat. In different repeated the experiments, the RayIoU varies between 36 and 37.