Native media Equirect layers creation

[danrossi]

Related issue: #31029

Description

Adding native media layer creation for WebGPU XR. This fixes a context loss issue doing it outside the manager. There is a potential Oculus browser bug complaining makeXRCompatible isn't set when setting up a media layer still.

Media layers are prepended into the background before the other layers.

Example https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	336.29 78.32	336.29 78.32	+0 B +0 B
WebGPU	548.37 152.03	549.48 152.37	+1.11 kB +331 B
WebGPU Nodes	547.72 151.88	548.83 152.21	+1.11 kB +331 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	466.23 112.58	466.23 112.58	+0 B +0 B
WebGPU	623.15 168.73	624.26 169.05	+1.11 kB +323 B
WebGPU Nodes	578.02 158.02	579.13 158.34	+1.11 kB +323 B

[danrossi]

It took a bit to design how to go about it. The only issue is you have to toggle the video mesh layers when going into XR and ending the session. Maybe there is other ways to do it.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L130
https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L151

live example https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

[Mugen87]

I have tried to improve the JSDoc a bit. For a functional review I hope @cabanier can have a look.

BTW: Please create a E2E screenshot for the new example webgpu_xr_media_layer. You can do that with npm run make-screenshot webgpu_xr_media_layer.

[Mugen87]

Please include the new example in files.json. Otherwise it won't appear on the homepage.

[danrossi]

looks ok

[danrossi]

The requirement to toggle off the video texture with the meshes. I'm not sure how to make better. It causes a crash if the texture is also rendered.

[cabanier]

Please match behavior of quad/cylinder layers

[cabanier]

I don't think this code implements emulation for 2D or browser that don't support xr layers. It will also not dynamically add them while in VR.
Please look at the flow of the quad and cylinder layer.

[danrossi]

It does as normal with the meshes added to the scene, just when going to XR the mesh layers need to be disabled, if it doesn't there is a crash in the Oculus browser. I can't get my head around those methods and there is no texture render required.

There is specific logic for the stereo and mono video that shouldn't be in the manager perhaps. There is multiple meshes for the stereo version.

So even if it was created in the manager and added to the scene the meshes need to be disabled.

The native layer only gets setup if layers are supported. Works with the Rift in XR that doesn't support layers it gets rendered in the projection layer.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L129

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

I could automatically disable/enable the mesh layers perhaps. That part I find is little clunky. There is nothing for it to do in the renderLayers method unless it's non layers and not in XR.

If you have another idea it would be good.

[cabanier]

The code that creates the video layer in xr should also the emulation in 2D mode. Right now you create a video texture in your sample code and disable it on session start. That logic needs to go to xrmanager.
This code also doesn't work if you add a media layer while in VR.

[danrossi]

So create the single or multiple mesh in the method. Return and add the geometries before adding to the scene. the issue is while in XR. Is a special render target needed like the others where its then only rendering when not in XR ? so a different render method is needed to the renderLayers method. To prevent that layer disable hack needed. I'm assuming that is the requirement if it actually works like that.

[danrossi]

I've refactored it quite a bit. And followed the structure of modifying the geometry externally to keep that external. I could add that in. It returns a group of meshes which works in 2D, Rift, Quest.

There is still the problem of needing to turn off meshes for native layers. Can be done internally or keep it external. I'm not across everything to figure out a better way to render from 2D to native.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

mono example https://electroteque.org/dev/threejs/examples/webgpu_xr_media_mono_layer.html

[danrossi]

Is this looking good setting the geometry externally.

I have it working implementing and tested. Unfortunately media layer needs to be added in the manager as doing it external caused a context loss fault. It may be a problem adding other layers externally as the media layers list needs to be prepended in the list as backgrounds as I have done.

The only issue still I can't figure out when going from the 2D mesh to native layer in XR. It will be rendered including the video texture in the projection layer as its added to the scene so those layers need to be turned off. Or maybe simply removing those mesh groups from the scene ?. I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer.

[cabanier]

... doing it external caused a context loss fault.

Do you have an example for me to look at? You shouldn't be getting context losses.

I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer.

Yes, can't you do the same as what I did for quad/cylinder layer?

[danrossi]

looks ok

[danrossi]

If the external geometry setter is ok. Or I make them default SphereGeometry requiring to update it's settings. I just set them to PlaneGeometry.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L105

I add the mesh group to the layers params.

https://github.com/danrossi/three.js/blob/equirect-layers/src/renderers/common/XRManager.js#L667

So this requirement to turn off texture rendering in the projection layer can be done internally perhaps and make it transparent.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114

There is an issue I can't figure out if it's the browser or three. Native media layer rendering is blank if it's an immersive-vr session. Only works in immersive-ar session. The controllers don't connect all the time either and can't figure out where to report. There is still the makeXRCompatible error in the console I can't find where to report.

[danrossi]

It might need a blank render target like the other layer examples. When using immersive-vr mode instead of immersive-ar mode like the XRButton uses. The projection layer render target blocks out the background native media layer. So it seems it's a blank render.

https://github.com/mrdoob/three.js/blob/dev/src/renderers/common/XRManager.js#L1480
https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl-fallback/WebGLBackend.js#L354

https://electroteque.org/dev/threejs/examples/webgpu_vr_media_layer.html

Doing something like this in the renderLayers method seems to work but not sure if its worth it and just keep it an immersive-ar session. If it peforms the same. Is just to be able to not have the projection layer render target cover it up. I noticed all the demos show up weird because they are AR sessions though. So stuff around the camera.

for (const layer2 of this._mediaLayers) {
    layer2.renderTarget.isXRRenderTarget = this._session !== null;
    layer2.renderTarget.hasExternalTextures = 
    layer2.renderTarget.isXRRenderTarget;
    layer2.renderTarget.autoAllocateDepthBuffer = ! 
    layer2.renderTarget.isXRRenderTarget;

    if ( layer2.renderTarget.isXRRenderTarget && this._supportsLayers ) {

        const glSubImage = this._glBinding.getSubImage( layer2.xrlayer, this._xrFrame );
	this._renderer.backend.setXRRenderTargetTextures(
		layer2.renderTarget,
		glSubImage.colorTexture,
		glSubImage.depthStencilTexture );

    }

}

[cabanier]

Can this also move to XRManager?

[danrossi]

Do you mean adding the group to the scene or managing the 2D layers internally ? Adding to the scene is external like the other demo. I may need to look at a render option for immersive-vr as it's a black render.

[danrossi]

If I understood correctly place the 2D mesh toggling internally. As the group adding to the scene is external like the other layer example. Example updated.

There is still an issue with immersive-vr mode and may need that render target code in the frame loop.

I added the XR controllers to show the projection layer render however the model is displaying artifacts for some reason. It's ok on the dev branch.

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

[danrossi]

I just worked out this isn't possible with an equirect layer. So in immersive-vr mode the native layer is being hidden by the three render. So not sure what method to use to "punch through". It's ok for immersive-ar mode.

 const glSubImage = this._glBinding.getSubImage( layer2.xrlayer, this._xrFrame );