Exposing cursor image (SVG/PNG/...) and dimensions (width, height)

[eladalon1983]

Many of the potential use-cases of this API consist of suppressing the "traditional" capture of the cursor in the video frames, so that it could be redrawn later, other faithfully or after some enhancing modification. Often, this would require a way to discover the original look and size of the cursor. It seems that it would be useful to align on a way to expose those.

Examining for the time being exposure of the image, but not the dimensions...

One way would be to expose some image of the cursor on each CapturedMouseEvent. That seems a bit wasteful, though. For the user agent, this might require some avoidable work with each event. For the Web app, checking whether the cursor changed between two subsequent events, might be less efficient than it could be.

Another way might be to expose a cursor-ID, which could simply be a long, and then introduce a separate API surface which would allow an interested app to query the user agent for the actual image of cursor. An example might be:

partial interface CapturedMouseEvent {
  readonly attribute long cursorId;
};

partial interface CaptureController {
  SVGElement cursorAsSvg(long cursorId);
};

(I'm using SVG as an example. It might very well be that we prefer a different format.)

[ldenoue]

Why not follow the current CSS way of doing it with a string?
CSS cursors

E.g. pointer, text.

But that string can also be a URL, e.g. cursor: url("hyper.cur")

[eladalon1983]

Maybe.

1. Would the URL potentially point to some locally-available blob? Would there be a cost associated with exposing it even if no reader examines it? If so, that'd be one reason to prefer CaptureController.cursorAs...(), where there might not have to be any performance penalty to exposing cursors nobody reads.
2. Text-based cursors are only meaningful on the local machine. Is there a convenient mechanism to get an SVG/PNG/etc. representation of them, for remote transmission?

[beaufortfrancois]

An ImageBitmap could also come handy.

partial interface CaptureController {
  ImageBitmap getCursor(long cursorId);
};

[fideltian]

i like the cursor idea. and it is needed for on-line meeting case, which need to transmits to the remote.

[eladalon1983]

In another thread, @fideltian seemed to suggest exposing the cursor shape directly on the events, but only for the first event following a change of cursor.

1. Did I get you right, Fidel?
2. This sounds reasonable to me. Possibly even preferable to my original proposal. I'd love to hear additional opinions, though.

[fideltian]

Hi Elad, I just read the spec(https://screen-share.github.io/mouse-events/) before and i did not notice the #4. i think your original proposal is also great. App just compare the cursorId for the change, correct? if changes, use the query API to get the picture of cursor. If that, i like it.

[eladalon1983]

Thanks, I love your proposal too! :-)

Maybe we can synthesize them to get an API that has the best of all worlds:

partial interface CapturedMouseEvent {
  readonly attribute boolean cursorShapChanged;  // ...since last event.
  ImageBitmap getCursorShape();
};

This way:

1. We don't need to add ImageBitmap directly to the event, which means better efficiency both in the user agent as well as the Web app.
2. It is possible to get the cursor shape even if it's not the first event since the last change of shape.
3. UA-implementers (such as yours truly) need not worry about caching ImageBitmaps for old shapes from 30min ago or earlier, in case CaptureController.cursorAsSvg(veryOldValue) is called - an issue with my original proposal.

[jozefchutka]

I can see there are more proposals already discussed in this thread. I would like to share my idea of having access to currently displayed cursor, which is rather simple: having CapturedMouseEvent with a parameter having one of these values https://developer.mozilla.org/en-US/docs/Web/CSS/cursor i.e default, pointer, text...

My use case / idea would be, to avoid replacing system cursor (on system), and rather "hide" it from capturing .getDisplayMedia({video: {cursor: "never"}}) and generate it on top of video or embed into final video. Either on fly (p2p chat) or in postproduction.

[eladalon1983]

I can see there are more proposals already discussed in this thread. I would like to share my idea of having access to currently displayed cursor, which is rather simple: having CapturedMouseEvent with a parameter having one of these values https://developer.mozilla.org/en-US/docs/Web/CSS/cursor i.e default, pointer, text...

I believe that's what @ldenoue proposed in the second comment on this thread. Afaict, this solution would not scale - it won't work for custom cursors, and even with the standard CSS ones, it won't work correctly between one user on (Chrome, Mac) and another one on (Firefox, Windows). I think a more ideal solution would be to expose a bitmap or vector-graphic.