Mesh overlays

[OlehKSS]

Add mesh overlays. For the implementation of overlays transparent mesh might be needed, it is possible that there is no such function in ipyvolume project, so we might need to make a pull request in it as well.

Example from GSOC wiki page.

Example with the data usage for showing activity as color-map.

Related ipyvolume issue

[OlehKSS]

@choldgraf, @larsoner from the ipyvolume issue I see that there is no transparency settings available for the meshes in ipyvolume, so what would you suggest me to do? It seems like three.js has this feature, so should I try to make changes on the JavaScript side? After this I plan to display activity patterns as part of the transparent mesh.

[larsoner]

Can you read this comment about ShaderMaterial? In particular:

The multiple overlays should be possible already now, since I'm exposing
the ShaderMaterial, all transparancies/blending rules can be set.

widgetti/ipyvolume#132 (comment)

It sounds like it should be possible already.

[OlehKSS]

Ok, it should be possible according to the code, but for some reason I couldn't find material attribute on a Mesh object. I am trying to install ipyvolume from the source code, but it doesn't work so far.

[larsoner]

In theory doing pip uninstall ipyvolume followed by the recommended pip install -e . while in the ipyvolume directory should do it. This is how I installed it.

[OlehKSS]

I think I have error with pythreejs v.1.0.0 (installed via pip), since my notebook can't find three.js and jupyter-three.js. I tried to run jupyter nbextension enable --py --sys-prefix pythreejs, and it was complaining about three.js The same command with pythreejs v.0.4.1 (installed via conda) works fine.

[OlehKSS]

Ok, it works now and I have material attribute on the mesh as well. I have tried to install/delete pythreejs of different versions and now it has three.js on the server, I have no clue what caused my earlier problems.

[OlehKSS]

ShaderMaterial objects are exposed by ipyvolume, it is working. Concerning overlays, which shape should they be? Should it be another cortex, on top of the given one? I suppose that volume of a transparent cortex should be enlarged so it would fit a non-transparent one.

[larsoner]

I have no clue what caused my earlier problems.

This seems to be a common sentiment after fixing these .js-missing problems :)

Concerning overlays, which shape should they be? Should it be another cortex, on top of the given one?

Yes, use the same rr and tris.

I suppose that volume of a transparent cortex should be enlarged so it would fit a non-transparent one.

No it shouldn't need to be. There are other ways to solve the problem if we need to, but let's see if rendering order (creating the opaque underlying mesh before creating any translucent overlays) already takes care of it.

[OlehKSS]

From what I find on the internet, three.js renders objects based on their distance from the camera, so closer ones will be rendered first, I think I will need to change renderer properties, so it will render the object depending on the time when objects were added to the scene, e. g. first to add - first to render.

[larsoner]

I think I will need to change renderer properties, so it will render the object depending on the time when objects were added to the scene, e. g. first to add - first to render.

Don't think too hard about what might be done, try things first. In other words, see if adding the underlying mesh first and overlay second works. Regardless of outcome, see if adding the overlay first then the underlying mesh second works. Hopefully one does. If it does not, we can think about a solution. But let's make sure there is a problem first.

[OlehKSS]

Yes, I have tried this scenario and it doesn't work. I have added one mesh, than another one transparent mesh (I changed material properties for this). As an outcome I obtained two brain meshes, one on top of another, but no transparency. I have tried to change material attribute material.depthTest = False, which should turn off depth testing for implementing transparency, and it gives me transparent mesh, but I don't see brain mesh inside.

[agramfort]

See how we do it with maplotlib backend by fusing rgb colors.

[OlehKSS]

@agramfort If it is possible, could you share a link? Because I am not sure where to look for it.

[larsoner]

See how we do it with maplotlib backend by fusing rgb colors.

This is possible (we don't even need a proof of concept to know) but I don't think it's worth pursuing unless "the right way" proves too difficult. It should really be doable in WebGL directly. As we want to add labels and so forth the do-color-mixing-ourselves approach will become untenable.

[choldgraf]

I ran into some folks last week who were working on a threejs multi-band 3-D viewer...I wonder if that could inform some of the color blending work here: https://github.com/imaging-tools/ivvv

[OlehKSS]

@larsoner, @choldgraf I have created an example of 3d plot, where I have one simple brain mesh, and I have transparent overlay, where some vertices are red and blue. Is it what you expect to obtain by using mesh overlays?

[agramfort]

can you share screenshots?

[OlehKSS]

Here some screenshots

[larsoner]

That looks right! Next would be to show the activation map (with mne colormap) like in my gist

[choldgraf]

looks beautiful @OlehKSS !