Enhanced Layering facilities

[RadianM]

Combining multiple effects into one pattern is relatively easily done by overlaying certain effects in overlapping Segments but it's a bit hit and miss. Only a handfull of effects actually cater for this. I think the potential for combining effects is underestimated.

Embracing "Photoshop like layering" would literally add many new dimensions to the ability of WLED to allow the creation of interesting composite patterns. For example, I hacked some existing patterns with code such as...

aRGB=effectPixel(i);   //pixel created by effect
if(overlay){
 aRGB.nscale8_video(SEGMENT.intensity);  //"Opacity"
 bRGB=SEGMENT.getPixelColor(i);          //underlying pixel
 bRGB|=aRGB;                             //"or" operator brings each channel up to the higher of the two values
 SEGMENT.setPixelColor(i, bRGB);
}

..to make them mix the RGB from the effect with the underlying RGB from lower numbered segments.

Using code tweaks such as the above I created a "Bonfire night party" with a "smoky sky" background in Segment 0 (Courtesy of Aurora) overlayed with licking bonfire flames (Coutesy of Fire 2012) and the fireworks effect. All pretty simple to arrange - but the results are so much more engaging than the effects in isolation,

Unfortunately the Segment controls don't make life quite so easy (although recent fixes to issues #3403 #3405 have helped). The Segment brightness slider is the main issue: as it's applied to the pixels written by the effect, it also affects the brightness of all layers beneath. For layering applications, as a crude solution, it could be exposed to the effect code which would apply it before merging with underlying pixel data. In this way it could operate more like "Layer Opacity". But ideally, it would be applied in interactions with the bus_manager.

One other unfortunate consequence of the layering as-is is that Mirror effect also mirrors all underlying segments as the mirror is applied to the pixels as they arrive via the setPixelColor (the same problem as with brightness I mentioned above) so it does seem to make it vital for layering to be handled deeper down in the architecture (as opposed to being up to each individual effect)

Then, along with a new drop-down menu for different "Layer mode" options (simulated in image below), it could be used to apply the overlay using some of the popular image blending techniques and make layering a possibility for all effects. How much of this could practically be rolled into the existing code I'm not sure. I'm still finding my way around it.

Proposal for enhanced Segment controls:

[blazoncek]

Easier said than done.
Would require re-writing majority of effects, which is impractical and for some of them impossible.

[RadianM]

Sure, I agree that hacking effects (as I've done) is impractical - and doesn't get around the Segment brightness and mirror issues.

That's why I queried the possibility of putting the layering enhancements "on the other side" of the calls to setPixelColor i.e. the effects just write out pixel data as usual and the layering is handled further down - thus making it available to all effects.

[RadianM]

With this tiny change to FX_fcn.cpp to re-purpose the segment opacity sliders:

////  uint8_t _bri_t = currentBri(on ? opacity : 0);  //Opacity slider mod...
  uint8_t _bri_t = currentBri(on ? 255 : 0);          //...force segment opacity sliders to 255 and use value in effect code instead.

Then in the effect code, at the time of writing out pixel data:

aRGB=effectPixel(i);   //pixel created by effect
if(overlay){
 aRGB.nscale8_video(SEGMENT.opacity);  //scale RGB by segement opacity slider
 bRGB=SEGMENT.getPixelColor(i);          //underlying pixel
 bRGB|=aRGB;                             //"or" operator brings each channel up to the higher of the two values
 SEGMENT.setPixelColor(i, bRGB);
}

This achieves my goal of getting individual layer opacity from each segment. The control is even labelled opacity in the code already.

(The above example for the effect code is exactly the same as I posted earlier except SEGMENT.opacity is used instead of SEGMENT.intensity - which is reserved for its usual role of modulating some other aspect of the effect.)

I even appreciate the transition that kicks-in on the slider change as it draws attention to the layers presence in the stack!

Of course, for unmodified effect code, effects will always be rendered at full brightness. No big deal for me as I'm only interested in a using few as candidates for layering.

[blazoncek]

This achieves my goal of getting individual layer opacity from each segment. The control is even labelled opacity in the code already.

Thanks, but WLED is used by thousands of people. Considering only "my goal" is a wrong approach as "your goal" may not align with everyone else's "goals".

(The above example for the effect code is exactly the same as I posted earlier except SEGMENT.opacity is used instead of SEGMENT.intensity - which is reserved for its usual role of modulating some other aspect of the effect.)

The above approach would require re-writing of every effect and some effects do not like to be rewritten. 😄 If you want to make such fundamental change it has to be on a different layer.

Of course, for unmodified effect code, effects will always be rendered at full brightness. No big deal for me as I'm only interested in a using few as candidates for layering.

Again "not a big deal for me" is a wrong approach. Consider rather thinking "how can I achieve my goal without disturbing everyone else".

I've managed to create effect blending without any change in the looks of effects and without too much code changes. I consider the same for segment "layering" or blending. If it is to be done it must not require any change in effect logic and has to be done in appropriate layer.

Currently I can see the possibility only by adding triple buffering to the code, with the third buffer used in Segment class (it used to have it by calling SEGMENT.setUpLeds() in the past but we removed it in favour of global buffering).

I'll tag @softhack007 and @ewoudwijma to join the discussion as they have plenty of experience by writing effect code and know how buffering and underlying pixel rendering works.

[RadianM]

I appreciate the fact that you've opened this up for discussion. I think the emphasis I placed on this "working for me" could have been explained better. What I intended to convey is that if a full embrace of layering is too big a step for the general release of WLED then I'm content with compiling my own private version given how one small change makes it so much more workable for my own application. I feel additional buffering should be avoided at all costs.

But I'm still blown away by the artistic possibilities pattern layering brings. Behind an acrylic diffuser, 1D and 2D scenes can be constructed in Bob Ross style. It's a massive step up from staring at a single twinkly effect. However I also acknowledge that the latter is far more appropriate for festive light strings.

[blazoncek]

But I'm still blown away by the artistic possibilities pattern layering brings.

As a photographer I know the possibilities of layers. 😄 Please prepare POC with the above considerations I mentioned and we'll be able to talk in more detail.

[softhack007]

Hi, i appreciate the idea, too.

Just a few additional points to consider for a generic implementation:

• optimization - a few extra rounds of getPixelColor/setPixelColor per effect are expensive in terms of CPU time. Its means that FPS will drop (maybe by 50%), especially on larger fixtures.
• I expect it will be too much computation/memory load for 8266.
• Best to make the blending feature an option that users can switch on/off (similar to global buffer option)
• currently there is no explicit concept for Segments that are "above" and "below". The result of blending several layers will depend on order, so its important that users get a possibility to specify which segment is "in from of" another segment.
• Probably @blazoncek is right, we might need some tripple buffering, as many effects render based on previous colors (for example: scrolling left/light/up/down, fading out (->trails), or "growing" a pattern). If the global buffer would return merged (blended) pixels that will be blended again, some effects will look very unexpected.

[blazoncek]

Currently I only see the option to make setPixelColor/getPixelColor paint segment's local buffer and then "blend" segment buffers in show call.

[softhack007]

Currently I only see the option to make setPixelColor/getPixelColor paint segment's local buffer and then "blend" segment buffers in show call.

agreed. and we need an idea how to control the order of painting segments that overlap.

[blazoncek]

The "rendering" order is the same as seen in UI (from top-down). There is currently no way to reorder segments as they are stored in vector. Rearranging vector items should be avoided for performance reasons.

[RadianM]

The "rendering" order is the same as seen in UI (from top-down). There is currently no way to reorder segments as they are stored in vector. Rearranging vector items should be avoided for performance reasons.

Can't just the properties of two segments be swapped with each other? Rendering order can then remain top-down if the UI lets the user 'move' a segment within the list.

[blazoncek]

Please check segment copy/move constructors and tell me if it is easy or possible.

[RadianM]

Please check segment copy/move constructors and tell me if it is easy or possible.

Not sure who you were directing that request to but I still haven't found out how to live preview the web served files and until I do it's too time consuming for me to try changing the UI.

[blazoncek]

Not sure who you were directing that request to but I still haven't found out how to live preview the web served files and until I do it's too time consuming for me to try changing the UI.

It was directed towards you.
Open data folder (within wled00) and there is index.html. Double click it and when it opens enter IP address of your WLED device.

[RadianM]

Many thanks. I must have fumbled it somehow, when trying it the for first time, because just opening from local file still gave me the same results until I cleared the local storage (also described on the changing-web-ui web page). Not sure how that happened as I was just mistakenly trying to use the live Server in Visual Studio.

[blazoncek]

POC coming along.

[blazoncek]

@RadianM care to test-drive the POC? I have something ready.

[RadianM]

Can do, if you point me to it.

[blazoncek]

PM me on Discord.

[blazoncek]

Ok, I've played a bit with my POC yesterday evening and decided it is not worth further investment.

There are several layers to the problem:

1. not all effects run at the same speed (frame rates) producing flashing output when blended
2. memory allocation issues (segments come and go during UI interactions)
3. insufficient CPU resources for blending (FPS drop noticeable on 20x20 matrix)

In itself no. 1 is already a dealbreaker, it would require rewriting of effects that do not return FRAMERATE;
I tried to circumvent no. 2 by not using local segment buffer (producing uglier result) but still got occasional heap corruption though I cannot pinpoint where that might happen.
For 3 the most likely cause is the implementation (done in such way to prevent major code changes) which relies on reading a pixel value before writing to it (for each setPixelColor() call).

Additional problem encountered is UI/UX. There is no easy way (requiring no or very little computation) to determine if two segments overlap. On top of that the order of segments does not guarantee that consecutive segments overlap. I am unable to see an easy solution (requiring low coding effort) for this problem.

[RadianM]

What a pity we can't seem to make this workable for mainstream release. I'm still actively pursuing the idea because it's providing great results with certain combinations of effects - some of which only require minor tweaks.
Is it not the case that memory issues only emerge if we employ an additional buffer? I've not found this to be necessary.
I've not quantified it thoroughly yet, but an ESP32 seems to have enough bandwidth to overlay a handful of 1D effects - e.g. over a 5m WS2812 tape run (@100fps).

[blazoncek]

You need to consider matrices of 32x32 at least for performance testing (using 2 or 4 outputs).
Certain combination of effects already provides nice looking output the problem is the other ones that do not. 😄

I've considered enhancing segment to host several effects (layered) and that could play nicely but would still break when segments overlap (which is possible).

[RadianM]

would still break when segments overlap

I've yet to see any problem with overlapping segments when using the get/modify/set method I outlined earlier and I can't see any reason why this should be the case... can you explain what I would have to do to get it to break when segments overlap?

[blazoncek]

get/modify/set incurs hefty penalty if not done locally.
getting it to play nicely with partially overlapping segments is/may be a challenge.
BTW this is my last post regarding layering (with current segment implementation).

[RadianM]

get/modify/set incurs hefty penalty if not done locally.

I'm sorry if I'm testing your patience here but I have to ask what you mean by 'doing it locally'? Maybe someone else may be able to explain and give Blaž a well deserved break 😄

[blazoncek]

Related to #3633

[blazoncek]

@RadianM almost a year and a half later, I've done it!

It may not work well on ESP8266 due to memory constraints (or S2 for that matter) but it does look good on ESP32 and decently sized matrices.

[blazoncek]

See #4550