Consider to bundle appstreamcli in the appimagetool AppImage

[kapitainsky]

It works but only when I remove metadata xml file. Otherwise it fails as below:

Found appimagetool: /tmp/.mount_linuxdCqqmwp/usr/bin/appimagetool
Running command: /tmp/.mount_linuxdCqqmwp/usr/bin/appimagetool "AppDir" "-g"

appimagetool, continuous build (commit fef038a), build 2093 built on 2019-07-07 12:07:34 UTC
WARNING: appstreamcli command is missing, please install it if you want to use AppStream metadata
Using architecture x86_64
/dev/shm/rclone-browser-1.6.0-d9e3ebe.AppImage/AppDir should be packaged as Rclone_Browser-1.6.0-d9e3ebe-x86_64.AppImage
Deleting pre-existing .DirIcon
Creating .DirIcon symlink based on information from desktop file
AppStream upstream metadata found in usr/share/metainfo/rclone-browser.appdata.xml
Trying to validate AppStream information with the appstream-util tool
In case of issues, please refer to https://github.com/hughsie/appstream-glib
/dev/shm/rclone-browser-1.6.0-d9e3ebe.AppImage/AppDir/usr/share/metainfo/rclone-browser.appdata.xml: FAILED:
• url-not-found         : <screenshot> failed to connect: SSL handshake failed [https://github.com/kapitainsky/RcloneBrowser/wiki/images/IMGdefault.png]
• url-not-found         : <screenshot> failed to connect: SSL handshake failed [https://github.com/kapitainsky/RcloneBrowser/wiki/images/IMGtransfer.png]
Validation of files failed
Failed to validate AppStream information with appstream-util
rename: Rclone_Browser*: rename to rclone-browser* failed: No such file or directory
cp: cannot stat ‘*AppImage’: No such file or directory

the same metadata xml file works perfectly on Ubuntu 16.04.

Both urls it complains about are valid.

The issue is in reference to linuxdeploy/linuxdeploy-plugin-appimage#8

[probonopd]

As the message says,

In case of issues, please refer to https://github.com/hughsie/appstream-glib

[kapitainsky]

It is not so simple:) what about to bundle working appstreamcli within your tool?

[kapitainsky]

Isn’t all AppImage idea about providing working bundle? Not just telling me that your version of OS cli whatever tool is not what we like. Tough.

[TheAssassin]

Not just telling me that your version of OS cli whatever tool is not what we like. Tough.

Well, then you have a really old version, and even if your file is valid with the bundled appstream-cli version, it might break on newer systems.

AppStream is broken by design...

[kapitainsky]

Yes it is old because I try to follow AppImage guidance and build my app on the oldest still supported distro. And I prefer this way. I want to make my AppIamge app working on as many Linux versions as possible. I am not criticising here. Just looking for solution.

…

On 25 Nov 2019, at 21:41, TheAssassin ***@***.***> wrote:  Not just telling me that your version of OS cli whatever tool is not what we like. Tough. Well, then you have a really old version, and even if your file is valid with the bundled appstream-cli version, it might break on newer systems. AppStream is broken by design... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[TheAssassin]

Well, you're not the only one to have problems with this stuff... xournalpp/xournalpp#1618

[probonopd]

Agree with @kapitainsky - the tools to validate AppStream are not easy to use because a) there are two different tools which don't always agree, and b) it's hard to get a version that validates AppStream against a recent spec to run on systems that are less than recent.

Something needs to happen indeed. I am considering to build appstreamcli in a way that uses nothing from the target system (e.g., build with musl like the one linked below) and perhaps include it in the appimagetool AppImage?

In the meantime, here is a build that should be able to do the trick:
https://github.com/probonopd/static-tools/releases/download/continuous/appstreamcli.tar.gz

[kapitainsky]

Ta. Obviously it is something you should find the way forward. What’s the point of having something which sounds like is not going to work. In my small case not so important. I just drop all these meta info and will carry on:)

…

On 25 Nov 2019, at 21:49, TheAssassin ***@***.***> wrote:  Well, you're not the only one to have problems with this stuff... xournalpp/xournalpp#1618 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[probonopd]

Please check out the appstreamcli that I linked. It should do the trick.

[kapitainsky]

In the meantime, here is a build that should be able to do the trick:
https://github.com/probonopd/static-tools/releases/download/continuous/appstreamcli.tar.gz

Thank you for trying but on CentOS it does not work:

[db@localhost appcli]$ ./appstreamcli 
bash: ./appstreamcli: /lib/ld-musl-x86_64.so.1: bad ELF interpreter: No such file or directory

[TheAssassin]

Using the latest version of this tool doesn't make any sense. Neither does bundling. It seems it's possible to bundle an extremely old version, files it validates might be used with newer AppStream tools as well. The other way round is apparently broken. See the Xournal++ issue I linked. That file validates just fine with the latest available version, but broke on the not so old Ubuntu bionic.

Edit: this is just an observation I made recently for some very specific projects. It doesn't mean that it applies globally. But, thinking about it, it makes some sense, right? One should assume that newer versions of the tool remain backwards compatible, i.e., once a file can be validated with a specific version, it'll be able to validate it with any newer version.

[kapitainsky]

AppImage tools should have 'reference' platform they are build to work on. Oldest supported LTS. Now it is either CentOS 7 or Ubuntu 16. I prefer former as appimage generated on CentOS work on Ubuntu not vice versa + CentOS still have many years of support left (till 2024). Now if developers use this platform to compile and package their apps they would expect that it works there. I am not computer expert and frankly even not entirely sure what these metadata is used for but building my first AppImage I wanted to utilize all available functionality (metadata, signing etc.) so step by step trying to follow online AppImage documentation.

[TheAssassin]

Actually, CentOS 6 will continue to receive "maintenance updates" until Nov 2020 (however, we don#t know how well they track security issues and fix them...).

As said, we've had cases when AppStream data which worked on CentOS 6 broke in higher releases, and vice versa. That's the big problem. Sometimes it seems the AppStream people retrospectively fixed bugs in their format, which caused their validator to fail. That means we cannot just say "whatever works on CentOS 6 will work everywhere", that's why everyone has issues with this AppStream stuff.

It is always bad if a format changes retrospectively, but isn't properly versioned, so that newer validators just reject a format that worked in the good old days. Warnings would be okay, but the AppStream validators don't have an option "ignore warnings" or anything.

Check out #603 for more information.

Edit: I haven't checked in a while, the information in this post might be a little outdated. Please correct me if I'm wrong!

[kapitainsky]

from practical perspective reading #603 it looks for me that it is hopeless atm. Chances that Linux world agrees on some standard are close to zero. Simply I package my apps without metadata.

[TheAssassin]

Practically speaking this seems like the best idea, yeah.

It's not really about agreeing on one standard. They have! But the standard doesn't seem to be maintained properly, or the tools aren't properly implemented. I don't know for sure...

[probonopd]

Using the latest version of this tool doesn't make any sense. Neither does bundling. It seems it's possible to bundle an extremely old version, files it validates might be used with newer AppStream tools as well. The other way round is apparently broken. See the Xournal++ issue I linked. That file validates just fine with the latest available version, but broke on the not so old Ubuntu bionic.

@TheAssassin I don't understand why you say "Using the latest version of this tool doesn't make any sense. Neither does bundling". Let's assume we can find a good way to bundle their latest version. Why would that not be sufficient? Maybe older base systems can't use the latest AppStream format, but so what - we are not handing AppStream metainfo to the base system, we are just using it internally in AppImage tools.

If I am not mistaken, bundling the most recent appstreamcli would allow the Xournal++ AppStream metainfo file to pass verification.

Thank you for trying but on CentOS it does not work

Ah right, I forgot to explain how to use it:

wget https://github.com/probonopd/static-tools/releases/download/continuous/appstreamcli.tar.gz
tar xfv appstreamcli.tar.gz 
./ld-musl-x86_64.so.1 ./appstreamcli

If we decide to bundle it inside the AppImage, of course this would all happen automatically.

[kapitainsky]

on CentOS 7

$ ./ld-musl-x86_64.so.1 ./appstreamcli validate ./rclone-browser.appdata.xml
Segmentation fault (core dumped)

but when I run only ./ld-musl-x86_64.so.1 ./appstreamcli it works listing all possible options

[probonopd]

Argh. Can you experiment a bit with the options of this tool and see which ones crash and which don't? I don't know what causes this bug...

[kapitainsky]

sort of all works... but validate correct xml file. If file to be validated is incorrect validation shows error

E: ~:~: xml-markup-invalid Could not parse XML data: Entity: line 1: parser error : Start tag expected, '<' not found
ELF
^
Validation failed: errors: 1

so looks like when parsing XML it crashes

[probonopd]

Are you 100% sure your XML validates correctly, e.g., in https://jsonformatter.org/xml-formatter?

[kapitainsky]

[kapitainsky]

The file is here

https://github.com/kapitainsky/RcloneBrowser/blob/master/assets/rclone-browser.appdata.xml

[kapitainsky]

I pretty much used Subsurface appdata.xml as my template

[kapitainsky]

on Ubuntu 16.04 built in appstreamcli does not like my id tag (i don't worry about it now)

$ appstreamcli validate ./rclone-browser.appdata.xml 
W - rclone-browser.appdata.xml:org.kapitainsky.rclone-browser
    Component id belongs to a desktop-application, but does not resemble the .desktop file name: "org.kapitainsky.rclone-browser"

Validation failed.

but when I remove id it passes:

$ appstreamcli validate ./rclone-browser.appdata.xml 
Validation was successful.

[probonopd]

Indeed I can reproduce the segfaults. Maybe it doesn't like to be compiled with musl. We should ask @ximion what to do.

[ximion]

Odd. Is there any way to build AppStream with glibc? It isn't really tested with musl, but I also don't see a reason why it wouldn't work - as long as glib can build with musl, at least.
First of all though, it looks like the version of AppStream you are using is ancient, or there is something else wrong which doesn't make it produce all validation hints.
This is the output I get when validating the file:

I: org.kapitainsky.rclone-browser:8: summary-has-dot-suffix Simple cross platfrom GUI for rclone command line.
I: org.kapitainsky.rclone-browser:4: cid-contains-hyphen org.kapitainsky.rclone-browser
E: org.kapitainsky.rclone-browser:5: metadata-license-invalid Unlicense

Validation failed: errors: 1, infos: 2, pedantic: 2

Maybe update AppStream first, just in case there's an old bug in whatever version you are using?
In any case, to say anything for sure I would need a backtrace of this crash - the only thing I can say for now is that the file validates without crash on a glibc-compiled appstreamcli Git master version.

[probonopd]

Integration tools do.

Yes, and the integration tools can use the latest and greatest to parse AppStream. Not the outdated stuff that comes with distributions.

I have great doubts that the tool requires any recent glibc features.

Me too, but if I remember correctly then appstreamcli can only be built on very recent systems. (They are using a Docker container to build on Travis CI rather than the normal Travis CI provided build environments.)

Which is why the AppImage I made is an AppImage that bundles everything (down to, and including, ld-linux).

[TheAssassin]

Integration tools might come with your distribution and these likely don't support the ever latest AppStream, so your comment doesn't make any sense. Please go for the least common denominator and don't ignore the rest of the world just because you personally don't experience that issue.

</offtopic>

Please make sure your AppImage runs on CentOS 6. Then the app can be included in appimagetool.

[probonopd]

It is an ongoing issue that AppStream (the spec) seems to be a moving target, so what distributions ship never can handle the latest AppStream.

My learning: Never let anything that came with the distribution handle AppStream.

[kapitainsky]

Integration tools do.

Yes, and the integration tools can use the latest and greatest to parse AppStream. Not the outdated stuff that comes with distributions.

I have great doubts that the tool requires any recent glibc features.

Me too, but if I remember correctly then appstreamcli can only be built on very recent systems. (They are using a Docker container to build on Travis CI rather than the normal Travis CI provided build environments.)

Which is why the AppImage I made is an AppImage that bundles everything (down to, and including, ld-linux).

what about other platforms? arm? is it something you guys care about?

[TheAssassin]

It is an ongoing issue that AppStream (the spec) seems to be a moving target, so what distributions ship never can handle the latest AppStream.

That is not the point, and you probably know that... Show me one example of a feature added in AppStream that is really essential and must be supported.

[probonopd]

That'd be @ximion to answer ;-)

[probonopd]

what about other platforms? arm? is it something you guys care about?

Absolutely. But since appstreamcli is not written in Go, it's more cumbersome than necessary to produce binaries for other architectures as well.

[ximion]

Absolutely. But since appstreamcli is not written in Go, it's more cumbersome than necessary to produce binaries for other architectures as well.

What? Being written in plain C actually makes it easy to support pretty much all architectures, as a C compiler is always present.
It also is a myth that only the latest distribution will compile AppStream - the CI uses the most recent versions, because that's the point of a CI: Ensuring that stuff builds with newer dependencies.
That said, recent changes in GLib may make it necessary to bump the dependency on GLib to something more recent soon.

The AppStream specification is 100% backwards compatible, but of course if you use newer features than the version your software center supports, you won't see them. There is one exception to the compatibility rule: Formatting in description paragraphs (<em/>, <code/>) will produce funny results when used with an older libappstream - that's why we currently don't recommend using it, until implementations of this feature are widely available in stable distros. All other newer features will not cause issues when used.
Of course, the validator is evolving constantly, things that were failing to validate with older validators may be permitted with newer versions of the specification. Validating with the latest version is therefore pretty much always a good idea, as a metainfo file that passes validation is also backwards-compatible with older implementation.

As for what new features you may want, besides the formatting options for paragraphs mentioned above, newer versions also support videos as screenshots, release artifacts (to point to the source code of a release, for example, or link an AppImage file), newer content_rating tags, etc.

If you're aware of any issue that distro tools have with metainfo files that pass validation, let me know (unless it involves <em/> etc., as that is intentional, and only appstream-glib messes up when encountering those at the moment)

[probonopd]

Thanks @ximion for the explanation. So I think that using an AppImage of the latest and greatest appstreamcli to do the validation is the way to go.

What? Being written in plain C actually makes it easy to support pretty much all architectures, as a C compiler is always present.

Yes, but if we want Intel and ARM, 32-bit and 64-bit, then the setup in Travis CI becomes a bit more cumbersome, doesn't it?

[ximion]

Yes, but if we want Intel and ARM, 32-bit and 64-bit, then the setup in Travis CI becomes a bit more cumbersome, doesn't it?

Yeah, but it would be the same with Go. In theory, meson should be able to reliably crossbuild things, but I actually never tested that. Best would be a proper builder network with dedicated arm64/amd64 machines.

Thanks @ximion for the explanation. So I think that using an AppImage of the latest and greatest appstreamcli to do the validation is the way to go.

I have no idea about the AppImage creation process, but that feels like the sensible thing to do and the easiest way to go - afterall, isn't the point of AppImage to create portable executables? ;-)

[TheAssassin]

@ximion your overall point is, if I understand it correctly: we should build and ship the latest available appstreamcli for validation. Older implementations of libappstream should ignore newer features. The point of validation is to make sure nothing completely weird is happening (missing XML tags, broken XML, etc.), and that every feature used is valid. If newer features are used, they should be accepted, not rejected, and it's not an issue for older implementations.

Is that correct so far? If yes, I'd probably just go integrate appstreamcli into our building process (i.e., figure out how to build it on CentOS 6) and include it in appimagetool. That'll make a lot of things easier.

[ximion]

we should build and ship the latest available appstreamcli for validation

That would be ideal, as it means your upstream projects can use whatever AppStream feature they like and you don't limit them to a particular version of the spec.

Older implementations of libappstream should ignore newer features

Yes

The point of validation is to make sure nothing completely weird is happening (missing XML tags, broken XML, etc.), and that every feature used is valid.

Yes. Of course, older validators don't know about the new features yet, so they fail validation in case they encounter one. If you're not validating though, those things simply get ignored.
Errors and warnings returned by the validator usually mean something bad is happening, while infos and pedantic hints are just hints for the developer on how to improve the metadata (and won't fail validation).

If newer features are used, they should be accepted, not rejected, and it's not an issue for older implementations.

Yes.

Is that correct so far? If yes, I'd probably just go integrate appstreamcli into our building process (i.e., figure out how to build it on CentOS 6) and include it in appimagetool. That'll make a lot of things easier.

Yes, your description is correct. CentOS 6 probably has a really old GLib, that will give you problems unless you build it as part of building AppStream and then bundle it with the appstreamcli tool.

[TheAssassin]

We already build a newer GLib version, 2.44. Do you know if that's recent enough? Does your meson build system contain a check we could lookup to see if we have to upgrade?

Edit: found it, https://github.com/ximion/appstream/blob/master/meson.build#L63. As long as 2.56 builds on CentOS 6, we should be able to upgrade in our containers. Then we can build and ship appstreamcli easily.

[ximion]

Git master currently requires 2.62 even. I am thinking about how to lower that requirement, but if it's too much work, 2.62 would be the new baseline for a while.
All GLib versions should build on CentOS 6, I don't think that would be much of an issue (unless the GCC version and glibc there is really, really ancient)
I also have older systems to support with newer AppStream (always the current Debian stable release), so I do have some interest in not escalating the version requirements too much.

[probonopd]

2.62 is from 2019. How on earth are still-supported distribution like CentOS 6 or Ubuntu Xenial supposed to use the latest appstreamcli then? Ah right, an AppImage that bundles everything, down to and including ld-linux...?

[TheAssassin]

Absolute nonsense. GLib has always been bundled. And we have needed a newer version than whatever CentOS 6 implements due to a bug fix or something ever since I introduced AppImageBuild years ago. It doesn't really matter which version we use/bundle. The point is, I need to be able to build it in CentOS 6. That's what matters. I don't care at all what version we ship exactly.

I hope @ximion can keep using whatever version is in use now for a while at least. I can't imagine that there's so many GLib innovations you'd have to upgrade every couple of weeks.

[probonopd]

You are right @TheAssassin

[ximion]

I hope @ximion can keep using whatever version is in use now for a while at least. I can't imagine that there's so many GLib innovations you'd have to upgrade every couple of weeks.

It's unlikely to get bumped higher than 2.62, in fact, it is quite likely that I reduce the dependency down to 2.56 before releasing the current version (all releases only need 2.56 so far).

[TheAssassin]

This downgrade would be highly appreciated. The only thing I struggle with on CentOS 6 right now is that it can't find libmount, and I highly doubt either appstream or we need that, so once this autotools stuff finally accepts --disable-libmount resp. --enable-libmount=no, we'd be good to go...

[TheAssassin]

gfile.c: In function 'splice_stream_with_progress':
gfile.c:3017:35: error: 'F_SETPIPE_SZ' undeclared (first use in this function)
   buffer_size = fcntl (buffer[1], F_SETPIPE_SZ, 1024 * 1024);

Seems the kernel in CentOS 6 is too old, doubt we can get GLib 2.56 to build there. Unless we identify the section that requires that flag.

[probonopd]

Please make sure your AppImage runs on CentOS 6. Then the app can be included in appimagetool.

It does:

[root@host /]# cat /etc/*-release
CentOS release 6.7 (Final)

[root@host /]# ./appstreamcli-266eccd-x86_64.AppImage --version
(...)
AppStream version: 0.12.10

Then the app can be included in appimagetool.

Well, it's a bit large for that... 23 MB.

[TheAssassin]

You know, building your own stuff on an older distro isn't that hard. I've managed to hack around the issue mentioned above by adding CFLAGS="... -DF_SETPIPE_SZ=1024+7 -DF_GETPIPE_SZ=1024+8. I'm building the remaining containers now. Then I'm going to check we can build appstreamcli in them.

[ximion]

@TheAssassin The absolute minimum version of GLib to build AppStream is now 2.58, and highly likely will remain at that level for quite a while. This is the version that current Debian stable (buster) satisfies, which is a reasonable requirement (and also happens to be a target that I have to support anyway).
Ubuntu 20.04 LTS will have at least GLib 2.62. So this should be buildable on any distro unless it's really old.

[benrob0329]

Unsure if I should open a new issue or not, but appstreamcli exits with a symbol lookup error on Fedora 31 when I call appimagetool, but works fine when I call appstreamcli validate directly.

[benrob0329@aleph appimage]$ ~/Applications/appimagetool-x86_64.AppImage AppDir
/usr/bin/AppImageLauncher: /lib64/libcurl.so.4: no version information available (required by /usr/bin/../lib/x86_64-linux-gnu/appimagelauncher/libappimageupdate.so)
appimagetool, continuous build (commit 64321b7), build 2111 built on 2019-11-23 22:20:53 UTC
/home/benrob0329/src/minetest/build/appimage/AppDir/net.minetest.minetest.desktop: warning: key "Keywords" in group "Desktop Entry" is a reserved key for KDE
Using architecture x86_64
/home/benrob0329/src/minetest/build/appimage/AppDir should be packaged as Minetest-x86_64.AppImage
AppStream upstream metadata found in usr/share/metainfo/net.minetest.minetest.appdata.xml
Trying to validate AppStream information with the appstreamcli tool
In case of issues, please refer to https://github.com/ximion/appstream
/usr/bin/appstreamcli: symbol lookup error: /lib64/libappstream.so.4: undefined symbol: g_ptr_array_steal_index_fast
Failed to validate AppStream information with appstreamcli

[benrob0329@aleph appimage]$ appstreamcli validate AppDir/usr/share/metainfo/net.minetest.minetest.appdata.xml 
I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:57
    Consider using a secure (HTTPS) URL for 'http://wiki.minetest.net/FAQ'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:38
    Consider using a secure (HTTPS) URL for 
    'http://www.minetest.net/media/gallery/1.jpg'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:54
    Consider using a secure (HTTPS) URL for 
    'http://www.minetest.net/development/#reporting-issues'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:56
    Consider using a secure (HTTPS) URL for 
    'http://www.minetest.net/development/#donate'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:55
    Consider using a secure (HTTPS) URL for 'http://dev.minetest.net/Translation'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:15
    First 'description/p' paragraph might be too short (< 80 characters).

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:58
    Consider using a secure (HTTPS) URL for 'http://wiki.minetest.net'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:53
    Consider using a secure (HTTPS) URL for 'http://minetest.net'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:41
    Consider using a secure (HTTPS) URL for 
    'http://www.minetest.net/media/gallery/3.jpg'

I - net.minetest.minetest.appdata.xml:net.minetest.minetest.desktop:44
    Consider using a secure (HTTPS) URL for 
    'http://www.minetest.net/media/gallery/5.jpg'

Validation was successful: infos: 10, pedantic: 3

[ximion]

symbol lookup error: /lib64/libappstream.so.4: undefined symbol: g_ptr_array_steal_index_fast
Looks like AppStream was built against a newer GLib and used with an older GLib - so the newer version was either not bundled, or AS was compiled against newer GLib by accident.

[probonopd]

@ximion do you know a way to link GLib (or better: everything) statically?

[ximion]

Besides #1011 (comment) no. I see no reason why this wouldn't link GLib statically though, provided static libs are available at build time.
If that doesn't work, LD_LIBRARY_PATH/LD_PRELOAD with dynamic libraries will definitely do the trick.

[probonopd]

Thanks to the help of @Tachi107 we can now build a static version of appstreamcli, which means that we can use a new appstreamcli version and it will still run on older systems.

This makes bundling appstreamcli in the appimagetool AppImage a real possibility once we have aarch and aarch64 builds, too.

Here you see the static Linux binary running happily on FreeBSD thanks to the Linuxulator:

FreeBSD% file /tmp/appstreamcli-x86_64   
/tmp/appstreamcli-x86_64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

Man do I love static linking. Well worth the 4,2 MB of disk space (uncompressed).

[ximion]

Neat! :-) But you will not need to build lmdb, AppStream hasn't needed that for a while ;-)
Also, on Alpine there apparently might be extremely weird bugs (ximion/appstream#395) possibly related to Musl libc, so it may be useful to at least build the tests (and ideally run them) just to make sure noting major is broken ;-)