Stuck at processing scene 0% and rendering 0%

[ykn96]

I am running this command

python batch_demo.py --model_path C:\Users\Administrator\Desktop\User\3D_reconstruction\lingbot-map-main\lingbot-map.pt --output_folder C:\Users\Administrator\Desktop\User\3D_reconstruction\lingbot-map-main\results_output --video_path C:\Users\Administrator\Desktop\3D_reconstruction_drone_videos\Chinatown1_1920_1080_25fps.mp4 --use_sdpa --config config/outdoor_drive.yaml --keyframe_interval 10 --overlap_keyframes 8 --camera_vis default --keyframes_only_points  --frame_tag --frame_tag_position top_right --save_predictions --image_stride 1

and oddly i keep getting stuck at the rendering stage
I have a NVIDIA 4090, 64 GB RAM and i9 processor, running on windows.

Moving results to CPU...%|███████████████████████████████████████████████████████████| 375/375 [06:00<00:00,  6.24s/it]
Inference done in 363.1s (1.0 FPS)
Predictions saved to C:\Users\Administrator\Desktop\user\3D_reconstruction\lingbot-map-main\results_output\Chinatown1_1920_1080_25fps/ (375 frames, 6 keys/frame)
Rendering video to C:\Users\Administrator\Desktop\user\3D_reconstruction\lingbot-map-main\results_output\Chinatown1_1920_1080_25fps_pointcloud.mp4...
[render] [depth] range = [0.477, 3.022] m (p1.0-p99.0)
[render] [scene] scale = 0.120 m, 375 frames
[render] [sky] Running sky segmentation (model=skyseg_batch.onnx)...
Processing scenes:   0%|                                                                         | 0/1 [06:07<?, ?it/s]Warning: Sky segmentation model not found at skyseg_batch.onnx
[render] [sky] Sky segmentation skipped (no masks generated)
[render] [octree] Unprojecting + building level-16 octree...
[render] [gpu] build batch_size=4
[render] [warn] --keyframes_only_points requested but NPZ has no is_keyframe / frame_type mask; falling back to all frames
Unprojecting: 100%|██████████████████████████████████████████████████████████████| 375/375 [00:00<00:00, 815.88frame/s]
[render] [octree] 10,872,448 raw points → building level-16 octree
[render] [octree] SPC(L15) — L0:1, L2:24, L4:319, L6:5890, L8:171733, L10:3654844, L12:10411142, L14:10864796
[render] [octree] centroids: L0:1, L2:24, L4:319, L6:5890, L8:171733, L10:3654844, L12:10411142, L14:10864796, L16:10872300
[render] [octree] finest level: 10,872,300 cells
[render] [octree] 10,872,300 cells, built in 1.1s
[render] 10,872,300 voxels, 375 frames
[render] 375 camera frames
[render] [render] Output: C:\Users\Administrator\Desktop\user\3D_reconstruction\lingbot-map-main\results_output\Chinatown1_1920_1080_25fps_pointcloud_render_frames, workers: 4
[render] [parallel] D-lite: 4 workers, per-frame SharedMemory
Processing scenes:   0%|                                                                         | 0/1 [06:08<?, ?it/s]
Rendering:   0%|               

[LinZhuoChen]

Thanks for the detailed log. This looks like two issues interacting rather than insufficient hardware:

1. batch_demo.py defaults to --mode streaming, but that path currently does not save is_keyframe / frame_type. Therefore, --keyframes_only_points falls back to all 375 frames, producing roughly 10.9 million voxels.
2. Open3D’s multi-process offscreen renderer can stall during worker initialization on Windows. The producer then blocks while waiting for the four workers to consume rendering jobs, leaving progress at 0%.

You can try:

• Add --mode windowed so keyframe metadata is generated.
• In config/outdoor_drive.yaml, set:

pipeline:
  num_workers: 1

For those constrained by limited VRAM or GPU usage, you may also refer to the implementation at: ureeey@eeee84a

You can also try to share your case with me and I will try to test this case.

[ykn96]

@LinZhuoChen hey thank you. I tried the new setting, and now i get some onnx sky masking error

[render] [depth] range = [0.088, 2.863] m (p1.0-p99.0)
[render] [scene] scale = 5.223 m, 286 frames
[render] [sky] Running sky segmentation (model=C:\Users\Administrator\Desktop\QQ\3D_reconstruction\lingbot-map-main\skyseg.onnx)...
Processing scenes:   0%|                                                                         | 0/1 [33:01<?, ?it/s]Generating sky masks from image array (batch_size=64)...
Sky segmentation:   0%|                                                                       | 0/5 [00:00<?, ?batch/s]
Error processing courthouse: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: input.1 for the following indices
 index: 0 Got: 64 Expected: 1
 Please fix either the inputs/outputs or the model.
Processing scenes: 100%|███████████████████████████████████████████████████████████████| 1/1 [33:03<00:00, 1983.48s/it]

i think i should have enough VRAM, the GPU usage does not max out. So that should be ok.

[LinZhuoChen]

we have uploaded this model in https://huggingface.co/robbyant/lingbot-map/tree/main named skyseg_batch.onnx

you can update this sky mask model and have a try.
We will update the readme later.

[LinZhuoChen]

We have updated the code and Readme, you can have a try @ykn96

[ykn96]

@LinZhuoChen thank you so much. There is some progress, with a different error

Open3D Error] (__cdecl open3d::visualization::rendering::EngineInstance::EngineInstance(void)) D:\a\Open3D\Open3D\cpp\open3d\visualization\rendering\filament\FilamentEngine.cpp:104: EGL Headless is not supported on this platform. Error processing Chinatown1_1920_1080_25fps: [Open3D Error] (__cdecl 

open3d::visualization::rendering::EngineInstance::EngineInstance(void)) D:\a\Open3D\Open3D\cpp\open3d\visualization\rendering\filament\FilamentEngine.cpp:104: EGL Headless is not supported on this platform.

I am trying the newest codes with one of the given courthouse examples.
I think this is because i am using windows, not linux?

[LinZhuoChen]

Yes—this error is caused by the Windows rendering path, not by the courthouse/Chinatown data.

The offline pipeline constructs Open3D’s OffscreenRenderer. In Open3D 0.19, that renderer always enables EGL headless mode, while its implementation supports EGL headless only on Linux. The latest Windows
fallback switches multiprocessing to serial rendering, but serial rendering still uses the same OffscreenRenderer, so --num_workers 1 will not fix this particular error.

1. Use the interactive browser viewer on Windows:

python demo.py --model_path /path/to/lingbot-map.pt
--image_folder example/courthouse --mask_sky
```
Then open http://localhost:8080. This uses Viser rather than Open3D’s headless renderer.

[ykn96]

@LinZhuoChen hey i found out that i could run it on WSL and it managed to complete running. Key problem if using WSL is to install WSL-Ubuntu CUDA drivers instead of the usual ubuntu cuda drivers.
Thanks for your help.

Can the output files be combine into a .ply file?