add 2d text (as the texture of a plane)

[shensquared]

Following the suggestions in #41
Here's an initial attempt

Still a lot to clean up but want to first check, does this look like what you had in mind?

[rdeits]

Yeah, awesome! So, one question we should resolve is how we want to represent this in the meshcat interface. For example, one way to do it would be to add a new kind of object (like the way _meshfile works) which includes the specifications for the plane and the text on it. A second option would be to just expose the relevant pieces: for example, we could add a Plane geometry and create a TextTexture, and then drawing text on a plane would consist of sending a Plane with a TextTexture from the Python side.

I think I prefer the second option: it requires a bit more logic on the Python side, but it means that we can write text on anything, rather than just planes. And we can create convenience functions in Python so that simple tasks like drawing text on a flat surface is still easy. What do you think?

[shensquared]

Agreed, option 2 is more versatile and python friendly, which I guess is the point.

I think here’s what’s probably necessary for the job:

• Add Plane(Geometry) and TextTexture(Texture) in geometry.py
• Add set_text() method for the Visualizer() class, which is then handled by a new SetText() class in command.py, and it looks kinda like:

class SetText:
    __slots__ = ["object", "path"]
    def __init__(self, text, geometry_or_object=None, material=None, path=[]):
        if geometry_or_object is None:
            geometry_or_object = Plane(width, height, h_segments, v_segments)
        text_texture = TextTexture(text)    
        material = MeshBasicMaterial(map=text_texture)
        # may also need a separate treatment if the object already has material 
        # and we are replacing the texture instead of initializing it like in here
        self.object = Mesh(geometry_or_object, material)
        self.path = path

    def lower(self):
        return {
            u"type": u"set_object",
            u"object": self.object.lower(),
            u"path": self.path.lower()
        }

• Either add a call of set_object at the end of set_text on python side, or in handle_command on JS side, change this line to include or =="set_text" https://github.com/rdeits/meshcat/blob/c7dcbca3af0e76f3c22573076a8d8e54b52dff67/src/index.js#L626

The only part I don’t immediately know how to do is actually creating TextTexture() in python. I know how to do it in JS now, but the tutorials I found all depend on creating a canvas in HTML first and then add the text via fillText. Not sure how this canvas business can be handled on the python side?

[rdeits]

Yeah, this sounds pretty good. Right, writing text into a canvas is possible in python, but I'd rather not bother with it. Instead, I think we can send a set_object command as you suggested, and we can make the TextTexture.lower() method return something like this:

{
    uuid: "<insert uuid here>",
    type: "_texttexture"
    text: "Hello world",
    font: "sans-serif",
    width: 200,
    height: 100,
    position: [10, 10],
}

Then on the Javascript side, we would need a little bit of logic to look for _texttexture textures and replace them with images by rendering the text into a Canvas using the provided arguments.

In that way, we can still rely on the browser's very convenient text rendering, while still staying very generic and composable.

Does that seem reasonable?

[shensquared]

Yep, sounds very good and doable. Will try it and update here.

[shensquared]

Finished a draft on both the JS and the python ends. Here's what it looks like writing on a cube:

Still need clean up and figuring out details like text placement, optimizing the default font size, canvas size etc...

[rdeits]

Hey Shen! Are you still working on this branch? Anything I can do to help?

[shensquared]

Hey Robin, sorry for leaving it hanging here. I honestly had forgotten about this -- functionally it was working properly and hardcoding a few parameters served me well enough for then, so I lost track of cleaning it up here...

But yeah, I still do want it in. Will revisit it by the weekend and hopefully it will be pretty quick. But will definitely bother you if needed help :)

[rdeits]

Ok, sounds good! Thanks!

[shensquared]

Hey @rdeits, the PR is ready for review now.

[rdeits]

Hey @shensquared I just wanted to let you know I haven't forgotten about this! I'm playing around with some ways to simplify the text loading logic (and also simplify some of the mesh loaders as well). I'll let you know if it works out (or if it doesn't...).

[shensquared]

Ah, sounds good, thanks for letting me know.

[rdeits]

I think shensquared#1 should work pretty well, and it avoids the complexity of needing a separate SetText command type.

[shensquared]

Updated both the javascript and python codes for the new design.

Something weird came up. I noticed ctx.font isn’t getting updated correctly if using ${} syntax. By splitting out a dummy font middle-man, it seems ${} itself isn't to blame, though something is lost down this road.

Using “” + “” syntax works correctly, hence the small change.

[rdeits]

Oh, fascinating (and baffling). Thanks for figuring that out.

This looks great, so I'm going to merge it!

We can still make more changes if necessary, since meshcat-python won't actually start using any of this new code until we merge meshcat-dev/meshcat-python#32