Start: create thumbnails using f3d

[chennes]

This is based on, and a replacement for, #17069 -- it extends the work in that PR to shift the processing into a set of workers running from the QThreadPool and returning their data as it arrives.

It also adds Start to the C++ test suite, and adds a couple of tests.

[chennes]

@andesfreedesign my version of f3d, 2.2.1, does not understand the --config=thumbnail or --verbose=quiet arguments. Which version are you using in your tests?

[chennes]

@adrianinsaval could you check this out and make sure it does not freeze up your system when it's generating a lot of previews? I'm just using QThreadPool without doing any extra work, but I can probably do some equivalent of nice to reduce the threads' priority if needed.

[adrianinsaval]

Haven't tested big step files yet but this how the thumbnails show up for me on windows, just a logo

[chennes]

It didn't even occur to me that this might work on Windows! Definitely didn't test that.

[chennes]

@benj5378 what am I supposed to be doing with all these clazy-qstring-allocations warnings?

[adrianinsaval]

Haven't tested big step files yet but this how the thumbnails show up for me on windows, just a logo

nevermind, it was just because f3d wasn't on path so wasn't actually being called, once I added the parameter with the binary Path it works but the filename is overlayed on top of the images

I believe that was not the case in the previous PR, maybe because we're not using --config=thumbnail anymore

[chennes]

Maybe - I had to remove that because on my Ubuntu 22.04 test system the installed version of f3d didn't support the option. Maybe we should be smarter about it, and detect the version?

[benj5378]

@benj5378 what am I supposed to be doing with all these clazy-qstring-allocations warnings?

As far as I can tell from https://forum.freecad.org/viewtopic.php?t=65560 and https://github.com/KDE/clazy/blob/master/docs/checks/README-qstring-allocations.md , then the idea behind using QLatin1String is QString that have comparative overloads that are faster than QStringLiteral.

However, QString and QLatin1Strnig are different classes, so constructing a QString from QLatin1String means allocating.

Clazy docs:

QString s = QStringLiteral("foo"); // No allocation
QString s = QLatin1String("foo"); // Allocates, use QStringLiteral("foo") instead
s == QLatin1String("foo") // No allocation
s == QStringLiteral("foo") // No allocation

Qt docs:

Note: If the function you're calling with a QLatin1String argument isn't actually overloaded to take QLatin1String, the implicit conversion to QString will trigger a memory allocation, which is usually what you want to avoid by using QLatin1String in the first place. In those cases, using QStringLiteral may be the better option.

So what should you do? You get the warning because you construct a QString from a QLatin1String, thus implicitly calling the constructor and allocating. Change all use of QLatin1String to QStringLiteral, (where we don't have an overload for QLatin1String, nearly all cases).

[andesfreedesign]

@andesfreedesign my version of f3d, 2.2.1, does not understand the --config=thumbnail or --verbose=quiet arguments. Which version are you using in your tests?

Sorry, but I was on vacation...
My F3D version is 2.5.0

[adrianinsaval]

does --filename=0 option work on your version @chennes ?

[adrianinsaval]

--filename=0 --grid=0 --no-background gives a more similar result to --config=thumbnail, we should also set --max-size value, thumbnail config has it at 100Mib on my version, might make sense to make it even lower. For reference here's what I think is used when we use --config=thumbnail option

cat .\thumbnail.d\05_all.json

[
{
  "options":
  {
    "anti-aliasing": true,
    "camera-direction": "-1,-0.5,-1",
    "hdri-ambient": true,
    "max-size":100,
    "no-background": true,
    "verbose": "quiet",
    "tone-mapping": true,
    "translucency-support": true
  }
}
]

cat .\thumbnail.d\10_occt.json

[
{
  "match": ".*(step|stp|iges|igs|brep|xbf)",
  "options":
  {
    "scalar-coloring": true,
    "load-plugins": "occt",
    "up": "+Z",
    "ambient-occlusion": true,
    "coloring-component": "-2",
    "coloring-by-cells": true,
    "camera-direction": "-1,1,-0.5"
  }
}
]

while this is what is used normally:

cat .\config.d\05_all.json

[
{
  "options":
  {
    "axis": true,
    "tone-mapping": true,
    "grid": true,
    "loading-progress": true,
    "anti-aliasing": true,
    "filename": true,
    "camera-direction": "-1,-0.5,-1",
    "hdri-ambient": true,
    "translucency-support": true,
    "animation-progress": true
  }
}
]

cat .\config.d\10_occt.json

[
{
  "match": ".*(step|stp|iges|igs|brep|xbf)",
  "options":
  {
    "scalar-coloring": true,
    "load-plugins": "occt",
    "up": "+Z",
    "ambient-occlusion": true,
    "coloring-component": "-2",
    "coloring-by-cells": true,
    "camera-direction": "-1,1,-0.5"
  }
}
]

we could also consider enabling --camera-orthographic depending on user's view preference

[chennes]

Thanks, @adrianinsaval -- can you add that as a GitHub suggestion so I can just adopt the commit?

[adrianinsaval]

I just tested a 600mb file, freecad doesn't freeze but I eventually get this warning:
17:28:52 QProcess: Destroyed while process ("C:\\Program Files\\F3D\\bin\\f3d.exe") is still running.
the warning is shown after some time every time I start freecad or more specifically every time Start is loaded
setting max-size will likely greatly reduce how often users can encounter this warning but it might still a good idea to redirect those to log level or straight up ignore them

[chennes]

f3d turns out to be something of a moving target: I just tried it on Ubuntu 22.04LTS and it's only version 1.2 there, which doesn't have a "--load-plugins" argument (for example). How much maintenance are we committing ourselves to here?

[adrianinsaval]

f3d turns out to be something of a moving target: I just tried it on Ubuntu 22.04LTS and it's only version 1.2 there, which doesn't have a "--load-plugins" argument (for example). How much maintenance are we committing ourselves to here?

what where you testing on before? I don't think we need to support every single feature in 22.04 IMO it's okay to ask for a higher minimum f3d version for this non critical QoL feature. So perhaps it makes sense to have a version check before attempting to run f3d. On the other hand I don't know if the --load-plugins argument is even required, I never needed it when calling f3d 2.5.0 from command line.

[chennes]

f3d turns out to be something of a moving target: I just tried it on Ubuntu 22.04LTS and it's only version 1.2 there, which doesn't have a "--load-plugins" argument (for example). How much maintenance are we committing ourselves to here?

what where you testing on before? I don't think we need to support every single feature in 22.04 IMO it's okay to ask for a higher minimum f3d version for this non critical QoL feature. So perhaps it makes sense to have a version check before attempting to run f3d. On the other hand I don't know if the --load-plugins argument is even required, I never needed it when calling f3d 2.5.0 from command line.

I needed the --load-plugins=occt to generate previews for STEP files (which I'd guess is a pretty common use-case).

[chennes]

OK: the documentation for pre-2.0 releases is pretty sketchy, and it looks like I can write support for 2.x and 3.x, so I will add a bit of code to do a version check, and cut off at 2.0 as the oldest we support. That was only two years ago, but this software is undergoing quite rapid development.

[chennes]

@FreeCAD/code-quality-working-group could you please take a look at ThumbnailSource.*? Particularly to make sure I'm thread-safe there, but anything else that catches your eye as well. Thanks!

[chennes]

@pieterhijma your comment about the timeout reminded me that we don't actually properly handle that case, so I've added the code to do so (the timeout doesn't actually kill the process, but the code assumed that after the wait the process had completed, which is not necessarily true). I've also added a timeout to stop the job after 30 seconds so it doesn't just consume this thread forever.

[3x380V]

FreeCAD is going to show different file size than Windows File Explorer, Gnome Nautilus, MacOS Finder, etc. Is there particular reason for this behaviour?

More about this imlementation in separate #17817 (comment)

[hyarion]

Remember this comment? #17817 (comment)

[3x380V]

Not sure how does it answer the above question, so again: why should FreeCAD display different numbers (for sizes up to 1000B numbers would be the same) than each and every file manager ever used?

[hyarion]

On your question

Is there particular reason for this behaviour?

The comment answers why it was done like that before

I originally went with the 1000 divisor because I personally prefer the MB to MiB etc. -- the MiB always felt overly pedantic to me (and they are awkward to pronounce :) ). But that is pure personal preference, I'm not going to hold this PR up for that.

I don't think anyone would mind much if you like it to be 1024 and still use MB etc. instead - like in Finder etc.

[3x380V]

It is not what I would like, it is common practice. FreeCAD is just showing different size than everything else. I'm not aware about any software using 1000 base. To make it clear, I would of course like it to be denoted according to IEC 80000-13 (kiB, MiB...), just for a fun of educating users, but using kB for 1024, MB for 1024×1024, etc. as semiconductor memory industry does for ages seems appropriate. The key point is to avoid question like why does FreeCAD shows different file size than my file manager.

[hyarion]

It looks like I'm wrong, sorry.

Finder does use 1000, not 1024 (since 10.8). The same thing applies for all software on mac which uses the foundation framework for converting number of bytes to a string and doesn't explicitly specify to use 1024: https://developer.apple.com/documentation/foundation/bytecountformatter/countstyle (it always uses MB never MiB)

According to IEC 80000-13, MB and KB is valid to use when using 1000 as factor (look at the examples). It does specify that if 1024 is used, it should be named KiB, MiB etc.

By your arguments (follow standard + consistency with os file explorer), I suppose the current version actually is preferred on macOS?

[3x380V]

Ouch! I wasn't aware Apple changed to SI standard in 10.8. Anyway, IEC 80000-13 states in the second paragraph The following referenced documents are indispensable for the application of this document but, duh!, referenced ISO-IEC-2382-16 is no longer valid and I have not purchased current norm.

So indeed, current implementation matches current MacOS.

[hyarion]

https://www.iso.org/standard/7259.html has been replaced by https://www.iso.org/standard/63598.html which is available for free ($0 but need to go though the checkout process) or accessed from the online platform: https://www.iso.org/obp/ui/en/#!iso:std:63598:en

[chennes]

Sorry for the delay on this, it turned out that I had an error in the logic about when the new thumbnail signal was emitted. I fixed that, and also added the code to correctly emit the dataChanged() signal from the model when the new thumbnails become available, so that they appear as soon as possible without needing any sort of user-driven refresh.

[maxwxyz]

@chennes could you take a look at the CI fails?

[mwestphal]

Hi, F3D maintainers here! Do not hesitate to ping me if you need explanation about our options or our doc :)

[oursland]

The f3d gallery is pretty impressive! OCCT is in the process of adding support to representing animations in their data structures (Open-Cascade-SAS/OCCT#492), which will eventually feed into being able to export animations via glTF or other formats. I can see further integration of f3d to generate renders of assemblies, exploded parts, and animations.

[maxwxyz]

^ @FEA-eng @marioalexis84 heads up for FEM animations and @mfasano727

[pieterhijma]

Looks great to me.

Just one little thing but I'm nitpicking now: For concurrent code, I would always like to see 1) a motivation for the locking mechanism and 2) a comment on the granularity of the locking. The latter is not in this code, whereas setupF3D() is only called when a cached thumbnail is not available, therefore limiting the contention on the lock. I think this is interesting to note here (and educational for anyone who takes a look at this code), but if too much trouble, I'm fine with how it is.

[chennes]

@pieterhijma maybe use the "suggestion" feature here to show what you mean, and I can just merge it in?