[feature] apt-update for exist-db

[eduarddrenth]

For example when building an exist-db docker image it would be good if packages could be updated to the latest version.

See: https://exist-db.slack.com/archives/CG2MRUZ35/p1662555892413129

See: https://gist.github.com/line-o/8bab558790a532939432da5cd3ad8ef7

The gist can be a starting point, it should be reviewed and tested of course.

[line-o]

@eduarddrenth where or how would this be called?

[eduarddrenth]

Well, you could call it from your Dockerfile (bit complex example, sorry):

RUN --mount=type=secret,id=commands,required,target=/exist/commands [ "java", "org.exist.start.Main", "client", "--no-gui", "-l", "-u", "admin", "-P", "", "-F", "/exist/commands" ]

[line-o]

@eduarddrenth are there existing commands in /exist/commands or would this be the first one?

[line-o]

I was able to remove the dependency from package-service completely in the updated gist.
Remaining ones are util, repo, http-client and semver. semver is the only library package, the others are built-in modules.
semver is bundled with eXist by default.

[eduarddrenth]

sounds good... and the hard coded package-repo?

[eduarddrenth]

OK, the file commandos holds arbitrary xquery, isn't part of exist but can be supplied by any exist developer to be executed as part of a docker build process. In this file I would like to call update-all Op do 8 sep. 2022 17:02 schreef Juri Leino ***@***.***>:

…

@eduarddrenth <https://github.com/eduarddrenth> are there existing commands in /exist/commands or would this be the first one? — Reply to this email directly, view it on GitHub <#4553 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACN2KKF7ETK57WKAUBZH7BTV5H5ZJANCNFSM6AAAAAAQHTGOXY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[line-o]

Well, I will make it external but that likely won't help you much.
Would a request parameter help you?

[eduarddrenth]

Well, I will make it external but that likely won't help you much. Would a request parameter help you?

You refer to the hard coded url I suppose? I don't know, can be in a global property/config? Otherwise perhaps a parameter (not a request parameter I would say) to the function...

[line-o]

OK, the file commandos holds arbitrary xquery, isn't part of exist but can
be supplied by any exist developer to be executed as part of a docker build
process. In this file I would like to call update-all

@eduarddrenth is the commands file documented somewhere? When is this called? At which point in the lifetime of the container? At startup? Before autodeploy is read?

Why is it not commands.xq - is that an API endpoint?

Assuming there is an XQuery main module "commands" that is evaluated each time the docker container starts.

1. If it is read before packages in autodeploy are installed this will fail as semver will not be available reliably. So at least the call to the function must be conditional.
2. If you want to call a function with packageUri as a parameter, then the gist code needs to be refactored into a library module.

Looking at the above you will likely be better off just using the functions that packageService already exposes.

[line-o]

see also eXist-db/xst#33

[eduarddrenth]

OK, the file commandos holds arbitrary xquery, isn't part of exist but can
be supplied by any exist developer to be executed as part of a docker build
process. In this file I would like to call update-all

@eduarddrenth is the commands file documented somewhere? When is this called? At which point in the lifetime of the container? At startup? Before autodeploy is read?

Why is it not commands.xq - is that an API endpoint?

Assuming there is an XQuery main module "commands" that is evaluated each time the docker container starts.

1. If it is read before packages in autodeploy are installed this will fail as semver will not be available reliably. So at least the call to the function must be conditional.

2. If you want to call a function with packageUri as a parameter, then the gist code needs to be refactored into a library module.

Looking at the above you will likely be better off just using the functions that packageService already exposes.

This is not part of exist-db at all, it is my own use-case. The docker RUN command is executed build time, not every time the container starts, it is the very last command executed at build time, so after autodeploy etc.

I do not understand your last remark "better off...", packageService does not contain an update-all function I would like to call.

Feels like we do not understand eachother, perhaps we should call....

[line-o]

OK, if you are sure that packageservice is installed, then

• gather installed packages using packages:get-installed()
• feed that list to packages:get-remote()
• filter the list (each item that has both available and installed)
• for each call packages install

[eduarddrenth]

see also eXist-db/xst#33

good to know, but will not work within a (distroless) docker build of exist

[eduarddrenth]

OK, if you are sure that packageservice is installed, then

* gather installed packages using packages:get-installed()

* feed that list to packages:get-remote()

* filter the list (each item that has both available and installed)

* for each call packages install

Isn't that what your update-all gist does? Or do you suggest a different approach here?

[line-o]

see also eXist-db/xst#33

good to know, but will not work within a (distroless) docker build of exist

With xst installed on the host machine you can "remote-control" an eXistdb instance running in docker.

[line-o]

Isn't that what your update-all gist does? Or do you suggest a different approach here?

Not anymore as the gist does no longer depend on package service being installed.
But for you and for know you could just use the gist and modify it to your needs, right?
Or is it missing something?

[eduarddrenth]

see also eXist-db/xst#33

good to know, but will not work within a (distroless) docker build of exist

With xst installed on the host machine you can "remote-control" an eXistdb instance running in docker.

Nice!

[eduarddrenth]

Isn't that what your update-all gist does? Or do you suggest a different approach here?

Not anymore as the gist does no longer depend on package service being installed. But for you and for know you could just use the gist and modify it to your needs, right? Or is it missing something?

Thanks, in due time I'll see how to integrate in my build/deploy pipelines

[eduarddrenth]

One more question, is the idea now to let "package update fucntionality" be a part of xst and not to add an update-all function to the package-service? This would somewhat force users to implement update functionality on docker hosts, which may be undesirable.

[line-o]

@eduarddrenth I am not against adding a update-all feature to package-service or the util or maybe even the repo- module. I will not be doing this myself anytime soon, though. That's why I wanted to provide my findings here, if somebody else wants to pick up that task.

[adamretter]

when building an exist-db docker image it would be good if packages could be updated to the latest version

When you build the Docker image of eXist-db it is always the case that the latest versions of both:

1. Linux packages for Debian are downloaded ands used
2. The latest version of Xar files, for the version of eXist-db that you are building, are downloaded and used

[line-o]

This feature is not implemented in exist-db (at runtime) yet.

[adamretter]

@line-o But I think the issue was about build time and not runtime. @eduarddrenth said this in the original enquiry:

when building an exist-db docker image

[eduarddrenth]

when building an exist-db docker image it would be good if packages could be updated to the latest version

When you build the Docker image of eXist-db it is always the case that the latest versions of both:

1. Linux packages for Debian are downloaded ands used

2. The latest version of Xar files, for the version of eXist-db that you are building, are downloaded and used

Yes, I know, but I build my own docker apps based on an exist-db build from docker-hub (FROM exist-db...). In that situation updates aren't downloaded and that cannot be done in an easy way (exist-apt upgrade).

[adamretter]

@eduarddrenth If you use Maven to build your images from the official eXist-db images, it is fairly trivial to have it download the latest Xar files for you.

[line-o]

The feature, to update all currently installed packages to their newest version within an instance of exist - at runtime, is useful for a large number of users. This is regardless, if @eduarddrenth's use-case can be achieved at build time or not. This is why I would like to keep this issue open.

[eduarddrenth]

Good to hear, I'll look into that, perhaps it's an option for me Op zo 17 sep. 2023 13:49 schreef Adam Retter ***@***.***>:

…

@eduarddrenth <https://github.com/eduarddrenth> If you use Maven to build your images from the official eXist-db images, it is fairly trivial to have it download the latest Xar files for you. — Reply to this email directly, view it on GitHub <#4553 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACN2KKENYHGOTNKGSIPQKLTX23PUNANCNFSM6AAAAAAQHTGOXY> . You are receiving this because you were mentioned.Message ID: ***@***.***>