What is the desired behavior for capturing the cursor?

[eladalon1983]

What is the desired behavior for capturing the cursor?

IMHO, the cursor should:

1. Only be visible in the captured video frames when the cursor over the target-element's bounding box. (Henceforth: TEBB.)
2. When the cursor only partially intersects with the TEBB, the captured portion of the cursor should match the intersection with the TEBB.
3. When the cursor is over the TEBB, the cursor should always be captured, even if there are (from a z-order perspective) intervening elements in between the target-element and the cursor.
4. The cursor should always be drawn at the same offset to the TEBB as the user observes on their screen.

[markafoltz]

This seems reasonable to me.

However, as observed in Chrome, the cursor is painted separately from the content, typically by the underlying OS; and has to be re-rendered on top of the captured frame before delivery to consumers. The process of capturing the cursor position and re-rendering it is async from the rendering and capture of the element, so the captured element + cursor in the media stream will not match exactly what is rendered on-screen.

[eladalon1983]

Under normal conditions[*], I'd expect the discrepancy to be small wrt coordinates delta, short wrt time, privacy-trivial, and not impactful wrt usability. I could add the original proposal to the spec, but use SHOULD rather than MUST, then clarify with a note the possibility of discrepancies due to implementation inaccuracy. Wdyt?

--
[*] I.e. real human being operating a reasonable mouse, as opposed to software intentionally snapping the cursor around at high speed.

[markafoltz]

Yes, the discrepancy should be small, but could be noticeable when e.g. dragging an element around.

[bradisbell]

I've found that my use case for this API is completely hindered by the inability to hide the cursor in the capture.

const stream = await navigator.mediaDevices.getDisplayMedia({
  audio: false,
  video: {
    width: 1280,
    height: 720,
    frameRate: 30,
    cursor: 'never'
  },
  preferCurrentTab: true,
  monitorTypeSurfaces: 'exclude',
  surfaceSwitching: 'exclude',
  systemAudio: 'exclude'
});

This results in a stream that does have a cursor, unfortunately.

@eladalon1983 mentions in a related issue that this is not implemented in Chromium: #60 (comment)

Another previous issue mentions that the CursorCaptureConstraint would resolve this issue (and I agree): #6

Can anything be done?

[eladalon1983]

IIUC, @bradisbell and other commenters are looking to completely and permanently hide the cursor. In that case, CursorCaptureConstraint should be the solution. The topic of completing the Chromium implementation probably belongs in the Chromium tracker rather than here, though.

Btw, while the correct solution would be to implement the missing functionality in the browser, and that's what I'd advocate for... Not everyone is comfortable contributing to these codebases, and business problems still need solving in a timely fashion. Perhaps an app-level hack might fit you better in the short term? Could you use cursor: none to hide the cursor from the local user and the capture, then "manually" draw a cursor-like element to the page as a non-child of the target-element, thereby avoiding its capture?

[bradisbell]

I've opened up a Chromium bug for its implementation: https://issues.chromium.org/u/1/issues/394133543

Thanks for the idea on cursor: none... I experimented with that last night but ran into trouble around edges of the capture where the cursor point is off the capture area but still gets in. I hadn't considered making my own manual cursor overlay though in-page. Might be a little laggy but probably acceptable for now. Thank you, Elad.