Consider single frame snapshotting use-case

[Zubnix]

The idea is to support a good solution for the plethora of slow and hacky solutions that currently exist:

• http://cburgmer.github.io/rasterizeHTML.js/
• https://github.com/PixelsCommander/HTML-GL
• https://gist.github.com/PixelsCommander/a0b5882139cbb8a1781c#file-proposal-md
• http://pixelscommander.com/javascript/state-of-html-content-rasterization-draw-html-to-canvas-image/

The current element-capture proposal could provide an excellent solution with no additional changes to its current API proposal, but only on the condition that it provides an option to do raw [passthrough] video encoding (including alpha channel support) and the option to produce only a single frame (perhaps by setting the desired fps to 0?).

[JonnyBurger]

I would want this as well - more popular examples are html2canvas https://html2canvas.hertzen.com/ and Remotion https://remotion.dev (by me).

Capturing images from the video stream is not a good solution, becauseMediaRecorder compresses the picture.

[eladalon1983]

For those interested - @Zubnix and I have also discussed this over WICG/proposals#73.

I think that the Element Capture API produced here would not be the ultimate replacement for a dedicated screenshot API. One reason is that for screenshots, the browser should ideally prompt the user to grant permission for a single frame. An API built on top of video cannot do that.

I think that Element Capture is a great stopgap measure for screenshots, using a combination of (1) preferCurrentTab or getViewportMedia, and (2) Element Capture. But ultimately, I think we'll want to build a separate, dedicated API for screenshots. (Likely this will be easier for browsers that have already put in the work to support Element Capture, as Chrome is already doing.)

For those interested, I have previously proposed a screenshots API version of gVM. We're reviving work on gVM, so it'd be a good time to also revive discussions of screenshots there.

including alpha channel support

Another way in which a non-video API is required, differentiating the screenshots use-case from the video use-case that Element Capture targets.

[markafoltz]

This should work, if applications can live with the restrictions:

1. The need to use the preferCurrentTab picker flow to initiate capture.
2. The lack transparency-capable format. Even though Chrome now supports RGBA tab capture, currently there's no way I'm aware of to force Chrome to produce it from getDisplayMedia.
3. Cooperation from the site from which you're capturing screen shots. It will need to mint tokens, attach them to elements and pass them to your application.

The main advantage of a dedicated screenshot API would be additional guardrails around the capture process. For example we could immediately terminate capture after the screenshot is taken, and allow the user to preview the screenshot before sharing it with the site.

[hrgdavor]

I just caught something that looks perfectly in line with my needs, things I was trying to push for here, as I see this proposal also as something I could possible use to generate thumbnails of certain elements (fancy elements on a graph drawing app where nodes are HTML to use for minimap ..) .. I have ideas in my head...

There is talk with Thomas Steiner about webassembly but he mentions something very interesting (link with timestamp)
https://www.youtube.com/watch?v=lbUhVLi9X7M&t=1404s

Quote:

This is a Wasm podcast, so I won't say too much, but it will let you place DOM elements inside of a Canvas and actually then render those pixels. And our hope is that it'll provide that mix of good performance and cross-platform capabilities but also bring that back to the user agent features.

I am sharing it here, as some of ppl commenting here might be interested in this too.

[markafoltz]

If you are interested in capturing screenshots of content your own app is rendering, then the canvas approach would potentially work (not familiar with the details). If you want to capture screenshots of arbitrary content or allow the user to pick which element to capture, you'll need a more general API.

[StephenHaney]

@eladalon1983 do you think there's any hope on the horizon for a dedicated "screenshot an element" API?

We make a design tool that uses DOM/CSS for rendering and exporting the user's work as an image does not have a good solution. We've tried playwright servers, Chrome extensions using the debugger's screenshot capabilities, screen capture API, and the libs that use SVG/canvas tricks. They all have some drawback that makes either the experience or the resulting image poor for the user.

Strict needs before we can use something:

• User grants permission once, or, if needed every export, it is auto-locked to the current tab so they don't have a confusing tab selection modal
• No confusing warning banner that shifts content (like attaching the debugger for the screenshot API)
• Ability to capture things that are larger than the user's current viewport
• Alpha channel support
• Pixel perfect, no compression or color differences from what the user is seeing in their browser

Needs that could be worked around:

• Automatically remove occluding items (but this could be handled in app code)
• Specify an element by selector and it is captured regardless of whether it's on-screen or off-screen due to transforms (this could be handled in app code by removing transforms temporarily but it causes flashing for the user)

I hope these are useful notes on real-world needs and that this is an appropriate place 🙏

[eladalon1983]

@StephenHaney, the question of screenshots comes up semi-often, but unfortunately, it never has enough momentum behind it for meaningful work to be done.

If you are satisfied with a Chromium-specific solution, as your comment suggests, then you might want to replicate what Google does. Namely, try to file feedback from something like Google Docs, and you will see that it calls navigator.mediaDevices.getDisplayMedia({preferCurrentTab: true}), grabs a single frame, then stops the capture. This triggers a yes/no prompt to the user. An infobar will unfortunately appear and disappear, but if you use Element Capture in conjunction with it, you will be mostly unaffected for capture-purposes (although your users might be slightly disturbed by the visual noise).

This would let you remove occluding and occluded content.

This would not allow you to capture with alpha, or further than the viewport. But you could capture multiple frames and compose the image later.

I am not sure about "pixel perfect" here. I'd guess yes, but I am not sure.

This proposed solution does not address all of your requirements, but then again, you are getting this advice for free. :-)

[StephenHaney]

Thanks @eladalon1983, free advice is always appreciated :) For a design tool, alpha capture is a requirement (imagine making icons, for instance). We will likely use elementCapture for our video export though. For PNG export, I think we'll have to recommend downloading our desktop app. We'll likely use something like dom-to-image-more for runtime screenshots to feed context to LLMs (which can live without perfect accuracy).

If a screenshot API ever comes up again at Google, please keep us in mind!

[hrgdavor]

@eladalon1983 Is there any path to creating a group and start lobbying for this ?
@StephenHaney @Zubnix would likely join and may be able to find more interested folks.

[eladalon1983]

The Screen Capture Community Group and/or the WebRTC Working Group would be possible avenues.

• In the former (SCCG), you could use the Discussions repo.
• In the latter (WebRTC WG), maybe the getDisplayMedia spec's repo could be the most appropriate.