Questions Regarding Multi-View Usage and Dataset Compatibility in SAM3D

[Hank5120]

Hello @devinli123,
Thank you for extending SAM3D to support multi-view images — I’m very interested in your work and truly appreciate the code you’ve released.

I have successfully run the code and reproduced the results shown in your examples.
I’m now preparing to apply SAM3D to my own dataset, and I would like to ask a few questions:

Does the pipeline strictly require RGBA images?

The sample masks in your demos are extremely clean — how were they generated? If the mask quality is not ideal, can the system still perform multi-view fusion reliably?

What is the maximum number of views you have tested for multi-view reconstruction so far?

Is it possible to export the camera poses and intrinsic parameters?

Thank you again for your excellent work!

[devinli123]

Thanks for your questions and for trying MV-SAM3D!

1. Images: RGBA is not required – the images in images/ can be standard RGB files (e.g. .jpg).

2. Masks: In our examples, the clean masks are generated with a SAM‑style model using text prompts (e.g., “stuffed toy”), but in principle any segmentation model / prompting strategy works (SAM2 with point prompts, box prompts, etc.).
   We have not yet done a systematic study on mask quality, but intuitively: slightly incomplete or noisy masks usually don’t break the reconstruction, because SAM3D has a fairly strong prior and handles occlusions reasonably well; however, if a mask in some views segments the wrong object, that can indeed hurt multi‑view fusion quite a bit. We’d be very interested if you try this on your own data and are happy to look at examples you’re willing to share.

3. Number of views: So far we have tested up to around 20 views. In principle there is no hard limit; the main trade-off is increased runtime and memory with more views.

4. Camera poses: When you run scripts/run_da3.py, DA3 saves both camera extrinsics and intrinsics into da3_output.npz (keys extrinsics and intrinsics), which you can load directly with NumPy.

Thanks again for your interest in MV‑SAM3D — we really appreciate it.

[stihuangyuan]

Thanks for your questions and for trying MV-SAM3D!

1. Images: RGBA is not required – the images in images/ can be standard RGB files (e.g. .jpg).
2. Masks: In our examples, the clean masks are generated with a SAM‑style model using text prompts (e.g., “stuffed toy”), but in principle any segmentation model / prompting strategy works (SAM2 with point prompts, box prompts, etc.).
   We have not yet done a systematic study on mask quality, but intuitively: slightly incomplete or noisy masks usually don’t break the reconstruction, because SAM3D has a fairly strong prior and handles occlusions reasonably well; however, if a mask in some views segments the wrong object, that can indeed hurt multi‑view fusion quite a bit. We’d be very interested if you try this on your own data and are happy to look at examples you’re willing to share.
3. Number of views: So far we have tested up to around 20 views. In principle there is no hard limit; the main trade-off is increased runtime and memory with more views.
4. Camera poses: When you run scripts/run_da3.py, DA3 saves both camera extrinsics and intrinsics into da3_output.npz (keys extrinsics and intrinsics), which you can load directly with NumPy.

Thanks again for your interest in MV‑SAM3D — we really appreciate it.

sorry, i have to two questions:

1. Can the extrinsic and intrinsic parameters of the depth anything3' output be replaced with other inputs? Is a depth map input must be need?
2. The intrinsic parameters don't seem to work during multi-view inference. Can the generated point cloud or mesh be projected onto the image and match the image region?