SFML Backend

[jasonbeach]

I'm sure I'm like many who are frustrated with the extremely limited options there are for plotting in C++. I really like this library (see here for a particle filter demo I made), but like others who want real-time performance are left wanting with the gnu-plot backend (it absolutely blows my mind the only API to that library is through pipes). I am not super familiar with OpenGL, but was looking at SFML since they recently released version 3.0 of their library. I had their demo built and running in less that one minute after cloning the repo and had a simple ball bouncing around the screen in 15 minutes without looking at the documentation. It was really pretty intuitive.

I'm curious what interest there would be in pulling in an SFML backend if I started working on one.

[alandefreitas]

Yes. I'm also frustrated by that. :)

Gnuplot was a terrible choice, and we learned a lot from other mistakes in this library. However, I don't have the time to implement a new version. If anyone is willing to take over v2 of this library, I'm happy to help and guide them.

• Gnuplot provides no control over details, and the high-level features break between versions. My intuition is also that the backend in Matplot v2 should be SFML+Cairo. Or SDL+Cairo. The backend interface should be more like Python's matplotlib interface: just a few functions to plot lines, etc. The library would handle all other transformations because they're always the same. Gnuplot makes that impossible. But once we get this right, people could write their own backends more easily and the choice of SFML+Cairo or SDL+Cairo is not even that relevant anymore. A basic backend could just write the plot to the console. What's important is getting rid of Gnuplot. Once we do that and the interface is simplified, people can easily write whatever backend they want (OpenGL, QT, Webassembly, ImGui, the console, or whatever) by defining a few functions.

• Another problem with backends now is that people want different kinds of representations for points and colors. This is not really a concern in MATLAB or Python's matplotlib but it makes a huge difference in C++ applications. This probably implies we would need templated canvases/backends and bring over some concepts from Boost.GIL to matplot. This way, the backend member functions can have any type that satisfies the point/line/color/... concept instead of relying on a specific representation. Of course, these templated backends could be (in later iterations of the library) type erased in the implementation to reduce compile times.

• A third problem is the dual interface. Python's matplotlib provides an interface based on classes and an interface based on free functions. The rationale for the second interface is to make it easier for people to migrate from MATLAB. We inherited this dual interface, but nowadays, that makes no sense at all. Very few people are still migrating from MATLAB, and the free function API is even less suitable for C++ applications. This problem is easy to solve as we have to remove the free functions.

Still, at this point, most member functions in the figure and axes classes also need to be removed. A figure has at least one member function for each plot type. And the functions are duplicated on the axes and figure classes. Calling the function on the axes puts more content on the chart, and calling the function on the figure redirects the call to the default chart. This is repetitive and prone to errors. What's worse, that pattern from Python, when translated to C++ means we have to have all headers for all plot types included in figure header file. There should be no member functions attached to plot types. A figure is simply a sequential container of axes and axes are just sequential containers of plots. This way, users can also get used to the idea of writing their own plot types, by just defining how the inputs should be represented as fundamental data in the axes. If some plot type becomes very popular, we can port it to the main library. A minor point here: these names also came from matplotlib who took them from MATLAB, but the names are really unintuitive. It makes sense to rename them to something like figure -> figure, axes -> subplot, plot -> ?, backend -> canvas.

A fourth point we need to improve is the documentation. The documentation is only composed of lots of examples: code and figures. The code examples are used as tests, but this is not great. We should have the manually written exposition describing the concepts in an intelligible order and the reference for the headers, which can be generated with mrdocs.

If anyone is willing to take over v2 of this library, I'm happy to help. I'd like to see this situation improve, even if it's by some new library or by someone else.

[jasonbeach]

I've been working through the white paper and I must say overall it's very impressive what you've been able to do and the sheer number of different plotting use-cases that are supported.

Unfortunately I don't have time either to take over (as much as part of me would like to say I could) but I think it would be useful to start documenting these ideas (which I think I agree with all of them) in something like a "v2_planning.md". That is something I can help with and hopefully that would provide an idea of the level of effort required for a v2 re-write. My gut feeling (which is probably very naive) is that rewriting the core of a v2 with a couple of basic plot types, while non-trivial wouldn't be that difficult. I assume that 80% of the work to get back to parity with the current version would be in converting the all of the different plot types once the core was re-written.

[jasonbeach]

I'd really like to see this improve as well. I've been poking through the code and for the figure class, I don't see the duplicated functions you mention. I do see them in the axes_type.h header and freestanding/plot.h. It appears these functions form the first abstraction layer that is built on top of the classes that derived from axes_object.

I think a v2 of the library would have sort of the following structure:

The primary class is still a figure similar to how it is now. One of the objects in that class would be a canvas (to borrow from your terminology). The canvas class would be a type-erased container that would hold what is currently called a backend.

A figure is simply a sequential container of axes and axes are just sequential containers of plots.

I like that although I might add one addition level of "plot window" which is roughly equivalent to a subplot. So a figure is a sequential container of 1..n plot windows, plot windows are a sequential container of 1..2 axes and axes are sequential containers of 1..n "plots". A "plot" again is a type-erased object container. So all the specific plot functions from the axes_type.h header would be removed and converted to stand-alone plot object classes that really wouldn't have to know anything about a figure, axes, etc. and a figure and axes objects would only have to know about the type-erased plot container object. So the call to create a plot would be something like:

fig.axes.back().emplace_back(scatter(x,y)); // we can add syntactic sugar on this to make it a little nicer

One thing I'm really curious about and is what I think you hinted at in your second point is moving from a struct-of-arrays concept (i.e. plot(vector<double> x, vector<double> y)) to an array-of-structs concept (i.e. plot(vector<point2d>, Projection proj) where vector could actually be any "range" whose element has something like a point2d. The projection converts an element in that container to something the library understands like a point2d. (I'm unfortunately not super familiar with C++ ranges yet, but I think they could be helpful here.) So something like this:

struct particle{
 double position_x;
 double position_y
 double vel_x;
 double vel_y;
};
using particles = std::vector<particle>
...
particles current_dist;
...
f.gca().emplace_back(scatter(current_dist, [](const auto&p){return point2d{p.position_x, p.position_y};})):

[alandefreitas]

I don't see the duplicated functions you mention

Here they are: https://github.com/alandefreitas/matplotplusplus/tree/master/source/matplot/freestanding

The primary class is still a figure similar to how it is now. One of the objects in that class would be a canvas (to borrow from your terminology). The canvas class would be a type-erased container that would hold what is currently called a backend.

Yes.

One thing I was thinking about is that type-erasing the canvas would be important and kind of complex at the same time because the canvas would support different structures to represent points. That's one of the main limitations stopping other backends from existing now. But the abstract type used for type-erasing could just have variants of the same function to accepts different data structures for points.

A figure is simply a sequential container of axes and axes are just sequential containers of plots.

Exactly.

I like that although I might add one addition level of "plot window" which is roughly equivalent to a subplot.

We usually refer to axes as a subplot. matplotlib inherited this convention from here: https://www.mathworks.com/help/matlab/ref/subplot.html The user refers to it a subplots because that's the public API but that's stored internally as this Axes class.

That's why I wanted to rename axes to subplot. The name is awful (plural for one thing and singular for something completely different) and subplot is the name the user interacts with.

I'm not sure this equivalence of axes and subplot is what you meant by "plot window" but the rest of your description makes sense with this assumption.

fig.axes.back().emplace_back(scatter(x,y));

Yes. That's exactly what I was thinking about. This way, you don't have to have included headers for plot types you don't need.

we can add syntactic sugar on this to make it a little nicer

Yes, we would have a helper function like:

fig.push_back(scatter(x,y));

that would be internally equivalent to:

fig.subplots_[default_subplot_idx_].push_back(scatter(x,y));

Minor detail: notice I used push_back because emplace_back should be reserved for in-place construction:

fig.emplace_back<scatter>(x,y);

So something like this:

struct particle{
 double position_x;
 double position_y
 double vel_x;
 double vel_y;
};

Exactly. To be useful in C++, we need to support other types of points. For instance, OpenGL applications usually require the points to be float. There's not much point in using double for graphical applications. Sometimes, we might even need to attach the color and alpha to the points.

[jasonbeach]

One thing I was thinking about is that type-erasing the canvas would be important and kind of complex at the same time because the canvas would support different structures to represent points. That's one of the main limitations stopping other backends from existing now. But the abstract type used for type-erasing could just have variants of the same function to accepts different data structures for points.

I think that would really be a challenge. On the one hand if you use templates, you would have to template the type-erased container itself which would mean you would have to template the figure object with I don't think is desirable. If you have overloads of the same function, the type-erased container would have to be aware of what the contained backend type is so it would know which overload to call which again I think is not desirable.

I've been thinking about what the main interface between the the library and the backend should be and to that end, I've really been exploring how SFML does things and it really comes down to everything is a vertex array. My best idea is that essentially the backend should output a something like a vertex array in the form of triangles that draws lines, axes, etc. and a vector of strings for labels, legends, annotations, etc. Maybe also need a vector circles. The way SFML formats them is very close to what OpenGL expects. So while a plot class may take a range of an arbitrary type, by the time the backend receives that data it's transformed into a common format that is highest performing for the default backend.

Here's a straw implementation: https://godbolt.org/z/1hcboT4Mn hopefully to illustrate what I mean.

[alandefreitas]

If you have overloads of the same function, the type-erased container would have to be aware of what the contained backend type is

We can avoid that by providing overloads for all combinations of point types that we can type erase. Still a hassle, but it works.

it really comes down to everything is a vertex array

Yes. Something equivalent to a vertex would be the main point type. For instance, an OpenGL application can also use it because it contains all the information contiguously. The plots can do the transformations for other point types to become vertices and extensions to the backend would mostly related to plotting 3d points directly, but we can skip this for now (or potentially forever: anyone who needs it can propose their adaptation to the canvas type).

Here's a straw implementation: https://godbolt.org/z/1hcboT4Mn hopefully to illustrate what I mean.

Yes. That's precisely what we need! That implementation looks great.

[Please-just-dont]

I'm not interested in an SFML backend exactly, but I'm interested in seeing a full backend implementation that isn't the gnuplot one, because I want to write my own backend for my own renderer, and I want to see how it's supposed to be implemented. Currently, that's not possible as I don't know what to do, because it isn't implemented. I think a "standalone" implementation (ie., not using gnuplot) needs to be made at least as a "reference implementation" so people at least know what to do.

[alandefreitas]

needs to be made

That's the problem. We don't have the human resources for that :)

[Please-just-dont]

@alandefreitas Yes. OK. What I meant is that apparently the vision of this library was to work with a backend other than gnuplot. I tried to look how to implement a backend based on the OpenGL one, but there's nothing there to base it on. I don't really know what it involves. I guess the shapes are simple enough, just send the triangles and the colour and that's done. What I don't understand so much is the text part. The library keeps does the laying out of the text? If all of it is handled by gnuplot itself then I guess that could be tricky to implement the layout out of the text. I don't really know how it works.

[alandefreitas]

the vision of this library was to work with a backend other than gnuplot

Exactly. We started with Gnuplot and later on tried to generalise it to other backends. However, this Gnuplot backend was a poor starting point, and we never really developed a good API for backends.

[Please-just-dont]

@alandefreitas OK. Does matplotplusplus actually create the vertices of the charts, and then send them to gnuplot? The shapes and lines of the charts I can see being easy enough to implement, but how does it do text for labels? It doesn't have any dependencies on stuff like Freetype or anything that does laying out of text or anything like that, so how does it know where to place the labels and stuff? Or does gnuplot do everything?

[Please-just-dont]

@alandefreitas So what would have to happen in order to implement another backend? If gnuplot does the laying out... another backend isn't really possible without implementing the laying out of the text? Does this library actually lay out the shapes/bars/lines etc? Or is that done gnuplot as well? In the OpenGL backend the part that receives triangles and draws them is implemented, which means that matplotplusplus does the laying out and creation of the shapes/lines at least, right?

[alandefreitas]

Exactly. This library draws things as much as gnuplot requires the drawing. For instance, sometimes a given plot type simply doesn't exist in gnuplot, so we just emulate it by drawing the lines individually. But when it does exist, we just use the gnuplot commands directly. The API for backends is not great. Even if matplot++ handles it right now, it would probably have to be reconstructed some other way to look like a matplotlib backend where there are higher level functions like draw_line etc. In C++ we also have an extra problem with the type of data because it could just be many ways to represent points, and a reasonble API would have to support all of them, which makes the problem much harder than the problem in, say, matplotlib. Some people/applications have continuous storage, some people have ranges, some people have structs for each point, some structs also include the color or another dimension and so on.

[Please-just-dont]

@alandefreitas Thank you for the help and information. I can see the post you made about your opinion of the library, and that the entire structure is kind of... a little unfortunate in terms of implementing other backends. We'll see what happens.

[Please-just-dont]

@alandefreitas Oh, have you seen this project?

https://github.com/sciplot/sciplot

Also based on gnuplot. It's not related to this? And they're doing exactly the same thing as this? I never really understood why so many projects start anew and doing the same thing.

[alandefreitas]

Also based on gnuplot. It's not related to this?

Only in the sense that they solve the same problem. And also that the authors there also don't seem to have to time to maintain it. This is a recurring problem. A plotting library takes a lot of time. It's not an incremental improvement over something. So many projects start and don't get completed.

And they're doing exactly the same thing as this?

I think the logic is a bit different. They have more of a 1-to-1 relationship with gnuplot if I understand correctly.

I never really understood why so many projects start anew and doing the same thing.

They start the project because other projects are not maintained properly and because they want to experiment with a new strategy. Then they usually get abandoned because someone hires these people for a ton of money to work on something completely unrelated.