Renderer from Canvas support

[Zageron]

Problem:
I cannot run pixels in the shadow root.
Current create surface ends up running in to this:
https://github.com/gfx-rs/wgpu/blob/master/wgpu-hal/src/gles/web.rs#L117

Solution:
Offer an interface that allows the use of a canvas instead of requiring a canvas to be found on the window.
https://github.com/parasyte/pixels/blob/main/src/builder.rs#L259

Using either or of these interfaces would be very helpful:
https://github.com/gfx-rs/wgpu/blob/master/wgpu-hal/src/gles/web.rs#L32
https://github.com/gfx-rs/wgpu/blob/master/wgpu-hal/src/gles/web.rs#L53

[parasyte]

pixels does not know anything about your environment. There is no assumption that a canvas is found on the window, as far as pixels is aware. It's up to you to provide a surface in the form of a HasRawWindowHandle implementation (not a canvas) that wgpu is capable of rendering to. IMHO, this is a wgpu issue because it is wgpu that expects to find a canvas element on the HasRawWindowHandle implementation.

Because pixels does not know anything about your environment, I'm very hesitant to have it learn what a canvas is.

---

That leads me to a follow up question: what are you passing to pixels::SurfaceTexture as the HRWH implementation in your environment? Presumably it's just a winit::Window, but you mention the shadow root. Is that because you have an HTML template and you're defining it as a custom element?

I guess there is a deeper question regarding how these kinds of cases should be handled. From the perspective of this crate, everything under the graphics umbrella is generic and unknown except for the trait that allows wgpu to convert the surface into something it can render to. I really, really appreciate this design because it means we don't need any conditional compilation within this codebase. Ultimately that means a cleaner and easier to maintain project.

On the other hand, that complexity needs to exist somewhere. So far it has been the responsibility of wgpu and its web backend. Obviously, I would prefer to keep the complexity there, where it can be reused by other wgpu dependents. But I'm not against a solution that would retain the generic know-nothing design of pixels if it is necessary to allow wgpu to convert a HRWH into a renderable surface even if a canvas is not accessible through the HRWH interface.

To that end, maybe it isn't HasRawWindowHandle that we need, but something more like Into<Surface>... which could be a concrete RawWindowHandle enum, or an HtmlCanvasElement? This all implies that it isn't strictly an issue with wgpu but it spreads through the entire community using the raw-window-handle crate.

---

The docs for WebWindowHandle indicates that what wgpu is doing is the supported way to query a canvas through the window handle: https://docs.rs/raw-window-handle/0.5.0/raw_window_handle/struct.WebWindowHandle.html#structfield.id

The disconnect between shadow root and the top-level window is most likely in here.

[parasyte]

@Zageron I'm confident this isn't a pixels issue per se, but it's in our best interest to investigate and come up with an adequate solution. If you can provide some more info about how you've setup your DOM, I think we can either provide a minimal reproduction for the raw-window-handle project developers, or maybe even come up with something better that works for your particular use case right now.

I expect it will be the former. But I have no way at the moment to create a repro that concretely shows why this is a problem.

[Zageron]

Hey, thanks for the extensive response.

My full understanding:

I think the primary "issue" is that Pixels is fully generic, as you stated. WebGPU offers three interfaces:
create_surface_from_canvas
create_surface_from_offscreen_canvas
create_surface

Pixels uses create_surface because it is generic and accepts a raw_window_handle::RawWindowHandle, whereas the former two are specific to WebSys types (canvas types).

The generalized solution in wgpu:

let canvas: web_sys::HtmlCanvasElement = web_sys::window()
                .and_then(|win| win.document())
                .expect("Cannot get document")
                .query_selector(&format!("canvas[data-raw-handle=\"{}\"]", handle.id))
                .expect("Cannot query for canvas")
                .expect("Canvas is not found")
                .dyn_into()
                .expect("Failed to downcast to canvas type");

means that it expects query_selector to be able to find an element in the page with the data-raw-handle property.

My Problem:

• I'm using Shadow Dom, using LIT, like react, to render a component that holds the canvas I am passing in to Winit and Pixels to be rendered. However, Shadow Dom components cannot be accessed by query_selector.

The Conundrum:

• There is no generalized interface in winit to support web's new "offscreen_canvas" functionality. (So there is no incentive to add it to pixels.)
• There is not an explicit way to inform wgpu through Pixels to use an offscreen canvas. (This would add wasm specific functionality to Pixels and remove the fully generic feature.)
• As such there is no API for Canvas either, because it can be generalized with a property and query_selector when not using a Shadow Dom.

Solutions:

• Write my own renderer that accesses wgpu's create_surface_from_canvas similarly to how you access create_surface but create a non-generic interface for myself to render to web and native targets.
• Pixels offers an API for constructing web specific surfaces from either Canvas or Offscreen Canvas.

I will attempt to create an example repository today.

[parasyte]

means that it expects query_selector to be able to find an element in the page with the data-raw-handle property.

Yep, that's a requirement from the raw-window-handle crate.

I think you're missing a third "solution" in your list, and that's allowing HasRawWindowHandle consumers (like wgpu) to gain access to a canvas without the need to query the DOM for it.

My Problem:

• I'm using Shadow Dom, using LIT, like react, to render a component that holds the canvas I am passing in to Winit and Pixels to be rendered. However, Shadow Dom components cannot be accessed by query_selector.

How would an app normally gain access to the HtmlCanvasElement or WebGL2RenderingContext from outside the shadow DOM? The most obvious approach I can think of is that component authors need to provide a method that returns these references, and applications are expected to call them instead of querying the DOM out-of-band.

This would be an architectural change for raw-window-handle and would likely involve a lot of people to make it happen. But it would also fix all apps using raw-window-handle with Shadow DOM components, and not just wgpu or just pixels.

[Zageron]

https://github.com/Zageron/pixels-shadow-dom-example
The important "feature" is .with_canvas() on line 42.
https://github.com/Zageron/pixels-shadow-dom-example/blob/main/src/main.rs#L42

[parasyte]

We have some additional eyes on this issue now. Discussion has been upstreamed to rust-windowing/raw-window-handle#102.

[Zageron]

We have some additional eyes on this issue now. Discussion has been upstreamed to rust-windowing/raw-window-handle#102.

Thanks! I've been following all the discussions the past day, awesome development for Bevy if you didn't see it.
bevyengine/bevy#6147 (comment)

[parasyte]

I saw that PR but not the specific comment, thanks. What I noticed about the PR is that it implements practically exactly what I expect raw-window-handle should be doing; an enum with HtmlCanvasElement and OffscreenCanvas variants. This type would better serve the community in a minimal shared dependency like RWH. :)

We'll see how it goes!

[Zageron]

I hate it, but I figured a shadow dom workaround out.
See new code in index.js.
https://github.com/Zageron/pixels-shadow-dom-example

Note: This doesn't work with libraries that use the Shadow DOM out of the box, but I should be able to explode the Shadow DOM for a moment under a loading screen while we construct the surface.

[markusmoenig]

Is this the same reason why pixels does not work with the latest Tao version ?