Currently when doing inference and novel playthroughs, the button inputs work as expected. However, there is a bug where any novel mouse inputs seem to completely destroy the coherence of the scene. This is most likely due to the mouse signals during training being extremely specific values unique to the training dataset. To resolve this, we should decompose mouse into angle + direction. Angle is simple and will generalize from train->test
Magnitude is a bit trickier. Adding a very small amount of noise is probably a good idea, and doing some sort of scaling where the average mouse movement has magnitude 1 is a good idea. Best to visualize on a compass plot to figure out something reasonable. We want an easy way to account for how end-users might have different sensitivity scalings.
Currently when doing inference and novel playthroughs, the button inputs work as expected. However, there is a bug where any novel mouse inputs seem to completely destroy the coherence of the scene. This is most likely due to the mouse signals during training being extremely specific values unique to the training dataset. To resolve this, we should decompose mouse into angle + direction. Angle is simple and will generalize from train->test
Magnitude is a bit trickier. Adding a very small amount of noise is probably a good idea, and doing some sort of scaling where the average mouse movement has magnitude 1 is a good idea. Best to visualize on a compass plot to figure out something reasonable. We want an easy way to account for how end-users might have different sensitivity scalings.