Does an element need to be inside the viewport?

[JonnyBurger]

Can the following element be captured or can it not because it is outside of the croppable area?

<div style="top: -999999px; left: -999999px; height: 100px; width: 100px; background: green; position: absolute;">

It would be useful if it is possible!
For image and video editor, the media is often bigger than the browser viewport (let's say a 5MP image or a Full HD video).

[eladalon1983]

Capturing content of unlimited dimensions is not practical from an implementation standpoint. Allowing each user agent (browser) to implement its own limitations is bad for interoperability. Limiting to the intersection of the element's bounding box with the viewport gives a clear and simple definition that's sufficient for the vast majority of use-cases, and which can be implemented consistently between browsers.

[markafoltz]

The answer to the original post is currently "no" and that is made explicit in the spec; step 3 of Applying the restriction transformation.

I think it would be useful to keep this open to note the use case for capture outside the viewport, however.

[bradisbell]

Being unable to capture stuff outside of the viewport makes this API useless for pretty much any use case I can think of.

I want an off-screen iframe that I can postMessage data to, that is rendered with alpha channel to a MediaStream or similar, so that it can be composited onto other video and output as video. Even better would be to capture audio for that iframe and redirect it to a MediaStream. Essentially, we just need a captureStream for an iframe.

This feedback has been given in many places previously, by myself and several others working in similar spaces. It's very frustrating that these nice specifications keep popping up, and then become wrecked along the way with seemingly arbitrary restrictions.

What can us non-Googler non-W3C-dues-payer folks do? We appreciate the efforts made by several to include us in open discussions and meetings (shoutout to @eladalon1983), but the actual decisions always seem to be made elsewhere at other times. Often by the time anyone has heard of the new feature, it's already too solid to change.

Every artificially-limited API does irreparable harm to the web-as-a-platform. New APIs have significant implementation cost but a much higher modification cost. Let's be real, the likelihood of revisiting decisions such the "restriction transformation" is near zero.

[hrgdavor]

I have a use case in mind that could be related to this, but would love some feedback if it can be done well without.

When creating an editor like figma for example does with huge ammount of elements, it becomes necessary to optimize the rendering that it can support those huge documents. taking a snapshot of an document and using it as thunbmail when zoom out happens sounds like something useful for such case.

There are things done in the backend with headless browsers, for testing, for thumbnail generation it would be sad if this snapshoting of elements would be pushed to backend (ofc google, MS, AWS really like selling cloud compute) when it can happen on the client.

I may not be explaining it well enough, but there are many use-cases for taking element snapshot that would open the gates to many new cool applications, instead only focusing on conferencing apps (zoom, meet, wahtever).

I really find it hard to accept that you really are sincerely covering "wast majority" of use cases. Spec could define size as 1x,2x of screen size to be max and allow much more freedom, it does not have to allow infineite. Even for conferencing apps this is silly, as you would be limited by ui elements from generating shapshot size of whole viewport (I may be wrong, and misunderstanding, as I have not read the spec).

I apologize if I made false claims here(I am very interested in the spec, and in sucha cases I easily get abrasive), since I do not know spec in depth, and hope you can answer some of my concerns even if from your point of view it is obvious in the spec.

One glimmer of hope would be if I could use parent element to transform scale. And when taking snapshot of the element I get full resolution of it, this way I could at least have a chance to snapshot elements, and generate thumbnails. I did this trick with openlayers and it's canvas to generate maps larger (in pixels) than the viewport.

I hope I am making sense here, not sure if I conveyed my concerns well here. Could some the performance issues that concern me be helped maybe with inert attribute ?

[eladalon1983]

Hi @bradisbell. These APIs are debated at such forums as the WebRTC WG and the Screen Capture Community Group. You are very welcome to join the latter (joining link). Note the two upcoming meetings next week, whose details are here.

useless for pretty much any use case I can think of.

I understand and appreciate the general sentiment here, but I think you're not 100% intellectually honest in this message. Surely you can see that these APIs are focusing use cases that you simply don't share? Otherwise, why would we have gone to the trouble of specifying and implementing them? I am sure you can think of use cases beyond those which you happen to have. And if not - here is one concrete example for the current API.

Of course, we aim to build APIs that are optimally useful for as wide an audience as possible. Please work with us to get there, and kindly understand our perspective when convergence is not achieved.

artificially-limited API

I don't believe any of the restrictions are artificial, and I would gladly discuss each and every design decision. Some decisions might be outdated, some might even be misguided; but nobody is imposing limitations for entertainment's sake, I assure you.

[hrgdavor]

Of course, we aim to build APIs that are optimally useful for as wide an audience as possible. Please work with us to get there, and kindly understand our perspective when convergence is not achieved.

The example shared is exactly the first thing that comes to my mind as I see it fit well into Google/MS business with conferencing applications (ofc. all other similar apps will benefit too). Could it be that nobody else needs API to snapshot an element for a thumbnail in a way that is not obtrusive for the user ? Is there really a segment of the API that is not required for that exact use case, and covers one more use-case at least ? I honestly am not seeing a use case beyond conferencing, please don't get angry at me for asking this.

I do not expect to have much influence here, but at least I want to try 😃

I will still be thrilled to see this proposal adopted widely even in this form (I have some ideas for workarounds).

[eladalon1983]

I will still be thrilled to see this proposal adopted widely

Thank you!
To signal-boost this part of your message, let me quote it yet again it and rendered it bold:

"I will still be thrilled to see this proposal adopted widely" – @hrgdavor

(ofc. all other similar apps will benefit too)

The class of "video conferencing and similar apps" is an important segment of the Web. It is valid to design APIs that primarily target that segment and their use cases.

Could it be that nobody else needs API to snapshot an element for a thumbnail in a way that is not obtrusive for the user?

It is very likely! And when someone produces such an API, you will not find me complaining that their API does not work well for video. Sometimes one stone can kill two birds, sometimes not.

I do not expect to have much influence here

I hope you will join the Screen Capture CG, attend our future meetings, and change your mind about the level of influence you are likely to have. The link is here. Formally joining will ensure you receive a notification on future meetings. And you are always welcome to email me (eladalon@google.com or elad.alon@gmail.com) if you want to present at a future meeting.

You are always welcome to contribute to our existing specifications. If you want even more influence, you can always produce new specifications, build consensus around them, and implement them in such open source projects as Chromium and Firefox.

---

I think you (@bradisbell and @hrgdavor) wonder why we built an API shaped to work on video tracks. The answer is twofold:

1. Let me be perfectly frank - the author of this spec, your most humble servant, is primarily interested in video these days. That is legitimate. Open source contributions are made by interested parties. Your own contributions will reflect your own interests, be they financial or personal.
2. Web applications have established methods for initiating a video-capture of the current tab. These APIs (getDisplayMedia, getViewportMedia) have cleared the oh-so-significant hurdles of obtaining the approval of privacy and security experts, and were implemented to the satisfaction of browsers' UX experts. With that groundwork laid, it is now far easier to build upon it, than to start from scratch. I speak from experience.

[markafoltz]

The viewport restriction is not a technical one from Chrome's point of view, although there are complexity implications because Chrome like other browsers tries hard not to render parts of the document that are offscreen and don't have a way to be scrolled-to (i.e. are hidden or detached from DOM). These mechanisms would have to be defeated for offscreen element capture to work. While we could in theory make it work in Chrome, Elad's point above about interoperability is valid and input from other engines would be important before relaxing these restrictions.

And obviously, there are going to be performance implications if browsers are forced to render elements of arbitrarily large size for element capture.

Chrome does implement capture upscaling. If you capture with maximum resolution constraints that are greater than the element's rendered size, Chrome will attempt to upscale the rendering to provide a higher resolution stream (currently up to 2X). This may help your use case in the meantime.

For the headless use case, I suppose your headless browser can create a virtual viewport as large as it likes (if you are using e.g. a memory backed framebuffer) to work around this restriction. If you are trying to use a Chromium-based headless browser and want to contribute some special behavior for headless element capture, we can take the discussion over to the Chromium issue tracker to see if we can get folks to agree on how that would work.

iframe-based audio capture is an interesting use case, I can see that being something worth exploring and is worth discussing in the community group. Audio capture from arbitrary elements does not really make sense to me given there are two audio sources in HTML: media elements and WebAudio (which is not associated with an element at all).

[bradisbell]

although there are complexity implications because Chrome like other browsers tries hard not to render parts of the document that are offscreen

Yes, it's worse than that too when you include the throttling Chrome does when it isn't focused. We have been fighting these problems for many years, hacking around it in any way that we can, resorting to things like Web Audio API ScriptProcessorNode timers. All the optimizations around rendering/not-rendering and script throttling need to be considered.

And obviously, there are going to be performance implications if browsers are forced to render elements of arbitrarily large size for element capture.

If the purpose of the web application requires element capture, performance implications do not matter. The user wants their computer to do what they want it to do, even if it eats all available resources.

Chrome does implement capture upscaling. If you capture with maximum resolution constraints that are greater than the element's rendered size, Chrome will attempt to upscale the rendering to provide a higher resolution stream (currently up to 2X). This may help your use case in the meantime.

That's awesome!!!! Yes, that definitely helps. Is it possible to remove the 2x limit entirely? Possibly with a flag?

[jonrimmer]

A use case I'm interested in is allowing users to capture screenshots of elements to save as local image files. In my experience, this is a commonly requested feature for apps with reporting capabilities that show charts and other visualisations, and one where all existing options have some drawbacks[1]. I'm not sure if this use case is considered in scope for this feature/group, but the viewport restriction does present something of a hurdle to using it reliably for this purpose.

I note that all the major browsers already do provide the ability to screenshot entire nodes through their devtools. So code to do this already exists in some form. Although whether it can be done so performantly enough for video is another question.

If video performance is a dealbreaker, perhaps RestrictionTarget.fromElement() could be extended to take an optional mode argument 'continuous' | 'snapshot'. In snapshot mode, the browser will do its best to capture the entire node, ignoring the viewport, and then emit this single image continuously instead of a live view of the element. The browser may also not need to show the screen capture banner in this case, improving the user experience. There would still need to be a failure mode, throwing an error of some sort, if the target element is too large to snapshot. But hopefully this limit would be sufficiently high, if only a one-off render is required, as to not present significant interoperability concerns?

[1] Canvas.toDataURL() capture cannot be used when data is displayed using HTML/CSS or dynamic SVG. Server-side capture using Puppeteer has cost, complexity and scalability concerns. Client-side capture libraries like snapDOM and html2canvas are inherently somewhat flaky as they need to clone the target element in some fashion instead of capturing it directly.

[itsdouges]

Hi folks — I'm exploring using element capture currently and have run into this issue. I can work with elements having to be visible and inside the viewport but I'm being stung for elements that are partially out of view, take this video export:

Frame@2x.9.mp4

It should be a 3024× 1800 export but because of it being partially outside the viewport it ends up being 3024 × 1386. Is there a hard limit that the elements must remain completely within the viewport? Is there any method of working around this? Thanks to retina devices with devicePixelRatio being set to 2 I'm able to get double the viewport size which is great, but this is the hard limit it seems. Would love any insight 🙏.

For context I'm on the team at https://paper.design working on a video export feature.

Chrome does implement capture upscaling. If you capture with maximum resolution constraints that are greater than the element's rendered size, Chrome will attempt to upscale the rendering to provide a higher resolution stream (currently up to 2X). This may help your use case in the meantime.

So this means say if my element is 100x100 but I pass getDisplayMedia video: { width: 200, height: 200, displaySurface: 'window' }, it would bump up the resolution automatically, up to 2x?

[hrgdavor]

@JonnyBurger @eladalon1983 , from my perspective what my ideas were in this regard, chrome dev team has delivered something even better https://youtu.be/TUtKGTeFWjQ?is=dtQ4TPY7dU6oR5TX

https://developer.chrome.com/blog/html-in-canvas-origin-trial

[JonnyBurger]

@hrgdavor Yes really cool! We also started adopting it already :) https://www.remotion.dev/docs/html-in-canvas