Match CapturedMouseEvent with video frame

[chrisguttandin]

I think it would be beneficial if it would be possible to match a CapturedMouseEvent with the corresponding video frame.

Let's say there is an application which records a video using getDisplayMedia(). The application wants to record the position of the mouse, too. It does that by storing every CapturedMouseEvent. How would it match the individual mouse positions with the recorded video later on?

Maybe it's worth adding some sort of timestamp to the event which correlates with the timestamp that can be obtained from a video frame when using insertable streams.

[eladalon1983]

I agree that correlating frames and events is desirable.

Hopefully the timestamp can do most of this; Event.timeStamp might help there, if it can be easily and accurately correlated with the timestamp on the frame. If not, we could maybe add a secondary timestamp that's relative to the same start point as the that of the frames, and uses the same source clock.

Of course, when handling a new frame, we'd not want to buffer it until the next mouse-event surfaces, even if that happens almost immediately. IMHO, the right solution is to expose the mouse-coordinates on the frame itself, not the frame on the CapturedMouseEvent. Wdyt?

[beaufortfrancois]

I wonder if we could add captured display surface coordinates to the existing HTMLVideoElement.requestVideoFrameCallback() callback. That way, video frames and captured display surface coordinates would always be in sync.

partial dictionary VideoFrameCallbackMetadata {
  long capturedDisplaySurfaceX;
  long capturedDisplaySurfaceY;
}

Example:

const video = document.querySelector('video');
const stream = await navigator.mediaDevices.getDisplayMedia();
video.srcObject = stream;

function logCapturedDisplaySurfaceCoordinates(now, metadata) {
  console.log(metadata.capturedDisplaySurfaceX);
  console.log(metadata.capturedDisplaySurfaceY);
  video.requestVideoFrameCallback(logCapturedDisplaySurfaceCoordinates);
}
video.requestVideoFrameCallback(logCapturedDisplaySurfaceCoordinates);

WDYT?

[chrisguttandin]

Maybe it makes sense to keep the CapturedMouseEvent separate from the video frame. I still think it would be beneficial to be abled to place them both on the same timeline. But I also think they can be produced at very different rates.

When capturing a video of a mostly static source the video might be at 1 fps or even lower. But the mouse can still move quite rapidly.

On the other hand the captured video might be running at 30 fps while the mouse is not moving at all.

I think if the video frame and the CapturedMouseEvent would both have a timestamp based on the same clock it would be possible to reproduce the original video with the mouse pointer later on.

[beaufortfrancois]

Those are good arguments to not use HTMLVideoElement.requestVideoFrameCallback(). We may want to add an "Alternative API shapes" section to the README.md and explain why this path was not chosen.

[jozefchutka]

I think if the video frame and the CapturedMouseEvent would both have a timestamp based on the same clock it would be possible to reproduce the original video with the mouse pointer later on.

Such timestamp will be very useful. Esp. when "embedding" a custom cursor into recorded video in postproduction