Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post-Process example #5648

Open
wants to merge 37 commits into
base: master
Choose a base branch
from
Open

Conversation

enzofrancescaHM
Copy link

Description:
Minimal example to show the possibility of implementing Post-Processing in A-Frame.
NOTE: for Post-Processing to work also in VR mode, supermedium/three.js#20 must be implemented in supermedium three.
Changes proposed:
-index.html running a simple scene with simple geometries with and without emission
-bloom.js, a minimal implementation of Bloom effect

@vincentfretin
Copy link
Contributor

Where does all those 1087 lines of code comes from?
You should be able to import the postprocessing library with an importmap, I advice you to base your example on the new importmap example https://github.com/aframevr/aframe/blob/master/examples/boilerplate/importmap/index.html and write a small bloom component of a few lines.
The bloom effect integration in r3f is just this https://github.com/pmndrs/react-postprocessing/blob/master/src/effects/Bloom.tsx

@enzofrancescaHM
Copy link
Author

Where does all those 1087 lines of code comes from? You should be able to import the postprocessing library with an importmap, I advice you to base your example on the new importmap example https://github.com/aframevr/aframe/blob/master/examples/boilerplate/importmap/index.html and write a small bloom component of a few lines. The bloom effect integration in r3f is just this https://github.com/pmndrs/react-postprocessing/blob/master/src/effects/Bloom.tsx

Yes, thanks for pointing that, I completely missed the new feature of A-Frame to use importmap. Sorry for that, I've reduced the bloom.js to the bare minimum to work.

@vincentfretin
Copy link
Contributor

Yes, thanks for pointing that, I completely missed the new feature of A-Frame to use importmap

That new importmap example exist since 2 days ago ;) I'm glad I did it so we can have a simpler example here without copying all the code.

@vincentfretin
Copy link
Contributor

Didn't we agree from supermedium/three.js#20 to use
https://github.com/pmndrs/postprocessing that is more performant than the three/addons effect composer?

@dmarcos
Copy link
Member

dmarcos commented Jan 30, 2025

Didn't we agree from supermedium/three.js#20 to use https://github.com/pmndrs/postprocessing that is more performant than the three/addons effect composer?

Whatever yields 90fps with the simpler code and least amount of dependencies

@enzofrancescaHM
Copy link
Author

Didn't we agree from supermedium/three.js#20 to use https://github.com/pmndrs/postprocessing that is more performant than the three/addons effect composer?

pmndrs does not work in VR at the moment. I have opened tickets there, but at the moment no one is working on it.

@dmarcos
Copy link
Member

dmarcos commented Jan 31, 2025

I merged THREE changes and updated A-Frame so should work on top of master.

Can you put the examples under showcase and rename to post-processing: showcase/post-processing?

Thanks so much for all the effort

this.scene = this.el.object3D;
this.renderer = this.el.renderer;
this.camera = this.el.camera;
this.composer = new EffectComposer(this.renderer);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of recreating EffectComposer, RenderPass and UnrealBloomPass each update, they can be created once in init and only updated here.

},
bind: function () {
const render = this.renderer.render;
const system = this;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: this isn't a system, probably better to name it self.

* Unreal Bloom Effect
* Implementation for A-Frame
* Code modified from Akbartus's UnrealBloomPass.js
* https://github.com/akbartus/A-Frame-Component-Postprocessing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that the source is MIT licensed, the original copyright and license notice should be included.

"imports": {
"aframe": "../../dist/aframe-master.module.min.js",
"three": "https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.js",
"three/addons/": "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be

          "three": "../../../super-three-package/build/three.module.js",
          "three/addons/": "../../../super-three-package/examples/jsm/",

like https://github.com/aframevr/aframe/blob/master/examples/boilerplate/importmap/index.html
so it uses the three version from node_modules.

shadow="type: pcfsoft; autoUpdate: true"
background="color:black;"
renderer="anisotropy:4; stencil:true; alpha:false; colorManagement:true; exposure:1.0;"
bloomm="threshold: 1.0; strength: 0.6; radius: 1; exposure: 1.0">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the example still work? there is two m here.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

adb logcat | grep VrApi

Interesting. I think we are seeing different results for some reason. This is what I see with your command, Quest 3 and https://incluverse.eu/examples/bloomtest5.html Result matches what I see in the OVR Metrics Tool HUD.

I'm in v72 if that makes any difference.

02-04 16:21:59.001 22949 18279 I VrApi   : FPS=85/90,Prd=39ms,Tear=0,Early=0,Stale=8,Stale2/5/10/max=0/1/0/5,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=3/4,1651/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=2092MHz,Free=2510MB,PLS=0,Temp=24.0C/0.0C,TW=0.90ms,App=10.27ms,GD=0.00ms,CPU&GPU=21.66ms,LCnt=2(DR0,LM2),GPU%=0.97,CPU%=0.23(W0.41),DSF=1.00,CFL=12.68/17.31,ICFLp95=15.96,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:00.001 22949 18279 I VrApi   : FPS=87/90,Prd=39ms,Tear=0,Early=0,Stale=7,Stale2/5/10/max=1/0/0/2,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=2092MHz,Free=2510MB,PLS=0,Temp=24.0C/0.0C,TW=0.89ms,App=10.44ms,GD=0.00ms,CPU&GPU=20.89ms,LCnt=2(DR0,LM2),GPU%=0.91,CPU%=0.15(W0.26),DSF=1.00,CFL=12.71/17.10,ICFLp95=15.84,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:01.001 22949 18279 I VrApi   : FPS=85/90,Prd=39ms,Tear=0,Early=0,Stale=11,Stale2/5/10/max=2/0/0/3,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=2092MHz,Free=2510MB,PLS=0,Temp=24.0C/0.0C,TW=0.88ms,App=9.12ms,GD=0.00ms,CPU&GPU=20.88ms,LCnt=2(DR0,LM2),GPU%=0.98,CPU%=0.18(W0.30),DSF=1.00,CFL=12.53/17.05,ICFLp95=15.77,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:02.001 22949 18279 I VrApi   : FPS=88/90,Prd=39ms,Tear=0,Early=0,Stale=1,Stale2/5/10/max=0/0/0/1,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=1708MHz,Free=2510MB,PLS=0,Temp=24.0C/0.0C,TW=0.88ms,App=9.56ms,GD=0.00ms,CPU&GPU=20.86ms,LCnt=2(DR0,LM2),GPU%=0.97,CPU%=0.22(W0.30),DSF=1.00,CFL=12.67/16.92,ICFLp95=15.79,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:03.020 22949 18279 I VrApi   : FPS=90/90,Prd=39ms,Tear=0,Early=0,Stale=0,Stale2/5/10/max=0/0/0/0,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=1708MHz,Free=2508MB,PLS=0,Temp=24.0C/0.0C,TW=0.88ms,App=10.10ms,GD=0.00ms,CPU&GPU=20.52ms,LCnt=2(DR0,LM2),GPU%=0.98,CPU%=0.18(W0.20),DSF=1.00,CFL=12.80/16.72,ICFLp95=15.68,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:04.002 22949 18279 I VrApi   : FPS=90/90,Prd=38ms,Tear=0,Early=0,Stale=0,Stale2/5/10/max=0/0/0/0,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=2092MHz,Free=2508MB,PLS=0,Temp=24.0C/0.0C,TW=0.89ms,App=9.76ms,GD=0.00ms,CPU&GPU=20.57ms,LCnt=2(DR0,LM2),GPU%=0.99,CPU%=0.26(W0.45),DSF=1.00,CFL=12.69/16.66,ICFLp95=15.62,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:05.002 22949 18279 I VrApi   : FPS=91/90,Prd=41ms,Tear=0,Early=0,Stale=2,Stale2/5/10/max=1/0/0/2,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=2092MHz,Free=2508MB,PLS=0,Temp=24.0C/0.0C,TW=0.90ms,App=9.32ms,GD=0.00ms,CPU&GPU=21.79ms,LCnt=2(DR0,LM2),GPU%=0.99,CPU%=0.19(W0.40),DSF=1.00,CFL=12.63/17.11,ICFLp95=15.95,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:06.002 22949 18279 I VrApi   : FPS=88/90,Prd=42ms,Tear=0,Early=0,Stale=2,Stale2/5/10/max=0/0/0/1,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=1708MHz,Free=2508MB,PLS=0,Temp=24.0C/0.0C,TW=0.89ms,App=9.38ms,GD=0.00ms,CPU&GPU=22.11ms,LCnt=2(DR0,LM2),GPU%=0.99,CPU%=0.21(W0.29),DSF=1.00,CFL=12.81/17.02,ICFLp95=15.92,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:07.002 22949 18279 I VrApi   : FPS=89/90,Prd=42ms,Tear=0,Early=0,Stale=1,Stale2/5/10/max=0/0/0/1,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=1708MHz,Free=2508MB,PLS=0,Temp=24.0C/0.0C,TW=0.89ms,App=9.82ms,GD=0.00ms,CPU&GPU=22.06ms,LCnt=2(DR0,LM2),GPU%=0.99,CPU%=0.25(W0.38),DSF=1.00,CFL=12.64/17.14,ICFLp95=15.89,LD=0,SF=1.00,LP=0,DVFS=0
02-04 16:22:08.002 22949 18279 I VrApi   : FPS=90/90,Prd=42ms,Tear=0,Early=0,Stale=0,Stale2/5/10/max=0/0/0/0,VSnc=1,Lat=-3,Fov=0,CPU4/GPU=2/4,1382/545MHz,OC=3F,TA=0/0/0,SP=N/N/N,Mem=2092MHz,Free=2508MB,PLS=0,Temp=24.0C/0.0C,TW=0.46ms,App=10.04ms,GD=0.00ms,CPU&GPU=22.17ms,LCnt=2(DR0,LM2),GPU%=0.96,CPU%=0.32(W0.48),DSF=1.00,CFL=12.35/17.13,ICFLp95=15.79,LD=0,SF=1.00,LP=0,DVFS=0

@CodyJasonBennett
Copy link

CodyJasonBennett commented Feb 5, 2025

Thanks! Felix and I sit next to each other and we already talked this through.
I'm sorry you feel like I was acting in bad faith. That is certainly not my intent!
I'm not against post-processing but I'm wary to add features that are known not to work well. Users are very sensitive to low framerate experiences and we don't want them to get motion sickness or abandon the platform.

So instead you want people to test that all is well on everything, including emulator, but breaks on your devices? This isn't just post-processing, but anything offscreen rendering, and renderer state is permanently broken! Who are you to even decide this for everyone? Should we do feature detection to disable the most requested feature in three.js for iPhone too? I'm sure Apple themselves would be happy to see their own pages no longer work. My entire career in real-time graphics has been on handheld mobile/TBDR, and even without post-processing, I've used these (currently broken) codepaths to simply compromise and perform basic upsampling or blitting, which people should be doing in XR also. I couldn't disagree more with this notion, as you are actively gatekeeping in the process and preventing the rest of us from further improving this area. If you really want to shut down or remove anything, it should be three's (unmaintained) upstream implementation, not the whole ecosystem with it. That is why I find this in bad faith since the strategy here is self-sabotaging in my view, and I'm not willing to believe either of you didn't observe nor foresee this, nor acknowledge that this could be a simple combination of poor parameters.

What other PRs / changes we need on our side to implement pmndrs/postprocessing approach? Sorry to insist. It's for me hard to tell from the conversation.

There are no other PRs on our side; they must be made in pmndrs/postprocessing, and they can't be done without upstream improvements. This is what I mean by the ecosystem being in stasis, and it doesn't stop there but anything that uses offscreen rendering (shadows, impostors, LUTs -- although these should probably be parametric functions aside from color grading). Most of Drei is affected, for instance, and it would take many months to fix all the components.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

and they can't be done without upstream improvements.

I'm happy to take those improvements if there's a reasonable path to maintain them until upstream concerns are resolved

@CodyJasonBennett
Copy link

CodyJasonBennett commented Feb 5, 2025

I don't think you understood what I meant, but those libraries I maintain in Poimandres, and upstream means three.js or mrdoob/three.js#26160, which is among the simplest PRs I've made to three.js and by far the most valuable at $10,000 since I thought that would motivate people to repeat my own work and see past this nonsense. It's not the most complicated to research or repeat, and the changes are still exhaustive today. Again, I'm happy to personally issue devices or maintain this myself, and I've said elsewhere, but I will award proof of existing apps regressing with $5,000 as I mean when I say "strict improvement". I put an additional $500 here since I feel the same responsibility with aframe, and I only see Rik shutting this down no matter what I do myself. I don't believe him for a second. All of my time has been wasted, and I preach to deaf ears. Not letting that continue to happen.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

I don't think you understood what I meant, but those libraries I maintain in Poimandres, and upstream means three.js or mrdoob/three.js#26160, which is among the simplest PRs I've made to three.js and by far the most valuable at $10,000 since I thought that would motivate people to repeat my own work and see past this nonsense. It's not the most complicated to research or repeat, and the changes are still exhaustive today. I put an additional $500 here since I feel the same responsibility with aframe, and I only see Rik shutting this down no matter what I do myself. All of my time has been wasted, and I preach to deaf ears. Not letting that continue to happen.

I think understood and probably talking past each other.

What I meant is that I would be love to recreate whatever optimizations / approach exist or have in mind for Poimandres in A-Frame. Poimandres is great but don't want to take in new dependencies just yet. If there's any more pending three work happy to take those changes in our fork https://github.com/supermedium/three.js We already have mrdoob/three.js#26160 Is that all?

If it is. Does Poimandres have a bloom approach that is better than in this PR?

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

Thanks for upping the bounty. I think will get something out for people to play with in A-Frame 1.7.0

@CodyJasonBennett
Copy link

I give up and retract all bounties and pending work. Meta won.

@enzofrancescaHM
Copy link
Author

Well, this PR is full of life.. good sign. I invite everyone to keep this discussion useful and not toxic, I personally have thousands of other things to do, but I choose to spend some of my time to help the WebXR and XR community to have better tools to work with, but I have limited skills and I cannot contribute as I would, but you all are the very experts in this field and you should cooperate to improve things.
Now, back in topic. Just to summarize, we have worked together to enable threejs post-processing in VR using superthree to host some vital modifications needed and building the simplest possibile example that shows the bloom effect implementing some hacks in user-land to make things work. We have arrived to the point where visually the effects are very good, but we have noticed that performance aren't, in particular we managed to have some scenes that perform 90fps on Quest3, but very poorly on Quest2.

Now, in order to proceed I think we should:

  1. find a standard way to measure performance
  2. understand if it is possible to use superthree to host other modifications useful to improve performance
  3. give pmndrs/postprocessing a try in order to have some better performance and have a robust PP platform to operate in user-land

Thanks

@vincentfretin
Copy link
Contributor

Can someone confirm my results with the OVR Metrics Tool HUD and https://incluverse.eu/examples/bloomtest5.html?

Quest 3: Stable 90fps Quest 2: Stable 30fps

If we can bring those Quest2 FPS up we can merge this. Separate paths for Quest 2 and 3 would be acceptable.

I have the same metrics, 90fps on Quest 3 and 30fps on Quest 2.
Even if we can't bring up the fps on Quest 2, it shouldn't block an example working well on Quest 3 to be merged imo.
We can just add a warning about the fps on this example.

@enzofrancescaHM
Copy link
Author

Another 2 cents:
similar scene I've just built on playcanvas: https://playcanv.as/b/b2b8b290 similar perfs on Quest3 (if not worse)
generic bloom scene in wonderland: https://wonderlandengine.com/showcase/postprocessing/ (VR is broken completely)

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

@vincentfretin Thanks so much for confirming the numbers

@enzofrancescaHM Thanks for the patience and all the hard work. This is pretty close. Can we incorporate your car example to this PR (2-3 lights up to you)?

We can improve after merge. I’m super happy to have something we can iterate over.

@enzofrancescaHM
Copy link
Author

OK, done. Css is local, model is in te model folder of A-Frame Examples

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

OK, done. Css is local, model is in te model folder of A-Frame Examples

Thanks. Can you submit the model into the assets repo? https://github.com/aframevr/assets/tree/master/examples create a directory for the example under ‘examples’

@enzofrancescaHM
Copy link
Author

done.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

The car is available now via CDN:

https://cdn.aframe.io/examples/post-processing/fancy-car.glb

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

For model credit and other instructions we use a "standard" info panel:

https://aframe.io/aframe/examples/showcase/comicbook/

You can just import the info-message component and use it on the scene

@@ -158,6 +158,7 @@ <h2>Examples</h2>
<li><a href="mixed-reality/anchor/">Anchor (Mixed Reality)</a></li>
<li><a href="mixed-reality/real-world-meshing/">Real World Meshing (Mixed Reality)</a></li>
<li><a href="boilerplate/importmap/">Importmap (import teapot geometry from three/addons)</a></li>
<li><a href="showcase/post-processing/">Post-Processing (bloom effect)</a></li>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to add (bloom effect) for simplicity

@mrxz
Copy link
Contributor

mrxz commented Feb 5, 2025

Here's a proof-of-concept of a Bloom effect mostly hitting 90fps on a Quest 2: https://thrilling-alkaline-headline.glitch.me/
Note that the output is not visually identical to the UnrealBloomPass and there are some additional caveats.

This is not meant to further complicate or delay this PR. Just curious if it was doable and with what trade-offs, as that might help determine if we could/should pursue it. Even with the current performance implications this PR can be merged IMHO. In fact, the UnrealBloomEffect of Three.js is already used quite often with 8thwall. Even though the implementation isn't optimal for mobile GPUs it tends to work good enough for small, focussed experiences for handheld AR.

As @dmarcos said, once merged we can iterate over it. If post-processing ever becomes a core feature, then we should be a lot more careful.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

Here's a proof-of-concept of a Bloom effect mostly hitting 90fps on a Quest 2: https://thrilling-alkaline-headline.glitch.me/ Note that the output is not visually identical to the UnrealBloomPass and there are some additional caveats.

This is not meant to further complicate or delay this PR. Just curious if it was doable and with what trade-offs, as that might help determine if we could/should pursue it. Even with the current performance implications this PR can be merged IMHO. In fact, the UnrealBloomEffect of Three.js is already used quite often with 8thwall. Even though the implementation isn't optimal for mobile GPUs it tends to work good enough for small, focussed experiences for handheld AR.

As @dmarcos said, once merged we can iterate over it. If post-processing ever becomes a core feature, then we should be a lot more careful.

Any improvements you can suggest on this PR for Quest2? Two paths for Quest 2 and 3 is acceptable. Thanks!

@enzofrancescaHM
Copy link
Author

Wow, amazing work @mrxz ! Yes, as you were saying the effect seems a little bit cheaper than the one on this PR, but it is super acceptable and nice. Maybe two paths with different levels of quality / performance are the right way. And, I agree that we should merge and than iterate over it.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

Wow, amazing work @mrxz ! Yes, as you were saying the effect seems a little bit cheaper than the one on this PR, but it is super acceptable and nice. Maybe two paths with different levels of quality / performance are the right way. And, I agree that we should merge and than iterate over it.

Haven't tried in VR. Is the "more expensive" one noticeably better visually? If yes two paths is fine. If not maybe we can just have the cheaper one.

@enzofrancescaHM
Copy link
Author

Screenshot 2025-02-05 alle 21 16 36

@mrxz
Copy link
Contributor

mrxz commented Feb 5, 2025

The UnrealBloomPass uses multiple different Gaussian blur radii and combines them. Instead I only approximate one (large) blur and composite it, similar to what PlayCanvas is doing. The larger the intended bloom radius, the more noticeable the difference will be.

But it's actually possible to get closer while still (though barely) maintaining 90fps on Quest 2. See: https://thrilling-alkaline-headline.glitch.me/index2.html

This PR Old New
image image image

But it becomes questionable how useful a post processing effect is that eats up practically all your rendering budget.

Any improvements you can suggest on this PR for Quest2?

The biggest gains are in reducing the render targets used while rendering, but this is mostly caused by the internal implementation of the EffectComposer and UnrealBloomPass. So not much that can be done without modifying them directly or 'forking' them. Not sure if that's the best approach for an example. In that sense it might be worth looking into pmndrs/postprocessing as they do combine passes where possible.

One thing that can be done is optimizing the car asset using gltf-transform, as it does cause more draw calls than needed. But that isn't directly related to the bloom effect, of course.

@enzofrancescaHM
Copy link
Author

I think your new iteration looks very good! Yes, it is very veery similar to the PR's one.

@dmarcos
Copy link
Member

dmarcos commented Feb 5, 2025

How much of the rendering budget is used in Quest3? How much is reasonable? Any target we should aim at?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants