Camera controlled rendering

[wuzirui]

Hi! Thanks again for the great work. Will you open-source the code for camera-controlled rendering (as specified in Appendix B) and the weight of the pose mapper?

[daeyeonkr]

I also face the same issue regarding camera-controlled rendering. I think training pose mapper is challenging due to scale ambiguity. It would be very helpful if you could release the code and the pose mapper weights mentioned in Appendix B.

[ou524u]

Hi, thanks a lot for your interest and for raising this point!

Here are some of our observations that might be helpful:

1. On the pose mapper: We observed indications (via t-SNE visualizations shown in the paper) that even a simple linear mapping could work to some extent. However, while linear mapping can produce meaningful videos, it often brings noticeable high-frequency artifacts (e.g., subtle jitter in camera motion) when rendering long sequences. Replacing the linear mapping with small MLPs seems to improve this situation in practice.

2. On pose ambiguity: The challenge of scale and pose ambiguity is indeed real. That said, on datasets like RE10K (where the camera distribution is relatively simple), the mapping still yielded reasonable results in our experiments. We also refer to Table 6 in the RayZer paper, where the authors explored the inverse direction (predicting actual cameras from latent poses). Taken together, these complementary results suggest that—even though ambiguity is there—pose mapping can sometimes be made to work in practice.

Hope this provides some useful insights.