[BUG] missing dependency teipublisher

[eduarddrenth]

To be able to better understand you problem, please add as much information as possible to this ticket. Always test your bugs against the latest stable release of exist. We cannot provide support for older versions here on GitHub. If the version of eXist that is experiencing the issue is more than 1 major version behind the most recent release, please consider posting a question on our mailing list.

Describe the bug
During docker build teipublisher dependency cannot be found:

• package requiring it is not installed
• build succeeds

Expected behavior

• http://exist-db.org/exist/apps/public-repo/packages/tei-publisher to be found during docker build
• and, perhaps not easy, a failing build when packages fail to install for whatever reason

To Reproduce

• create Dockerfile:

FROM existdb/existdb:6.2.0
COPY *.xar $EXIST_HOME/autodeploy/
RUN [ "java", "org.exist.start.Main", "client", "--no-gui", "-l", \
    "-u", "admin", "-P", "", "-x", "system:get-version()" ]

• use this jar:
  corpora-1.0.xar.zip
• issue docker build:
  docker build --progress plain -t fails:1 .
• the output shows tei-publisher isn't found

Context (please always complete the following information)

• Build: [eXist-6.2.0]
• Java: [1.8.0_352]
• OS: [linux/docker]

[joewiz]

Based on that <dependency>, the request issued by the build process to the public-repo should be https://exist-db.org/exist/apps/public-repo/find?abbrev=tei-publisher&semver=7. This request currently returns tei-publisher-7.1.1.xar.

To troubleshoot further, it would be great to know:

1. What actual request URL does the build process send to the public-repo?
2. Does the public-repo provide the expected result, or does it return an error? If the latter, what error code and description?

[eduarddrenth]

The logging doesn't show the URL.
The install of the package via dashboard succeeds after docker build/deploy

[line-o]

This seems to be common misconception. @eduarddrenth @joewiz Autodeploy does not resolve dependencies, by design. For autodeploy to succeed to install a package with declared dependencies all of them need to be:

• in the autodeploy folder
• ensure that they are installed first as they will be installed in alphabetical order

If you want to have declared dependencies be resolved from a package repository (being it the public package repository or a private one) you have to use other means of installing the package (upload via dashboard, curl, xst, gulp-exist, ...).

[eduarddrenth]

Thanks, may I ask why this is? It seems quite logical that dependencies for a package are being resolved upon installation, whether via autodeploy or otherwise.
The way it works now forces me to manually resolve dependencies or script that somehow in my docker build

[line-o]

The rationale why autodeploy does not resolve dependencies was explained to me to allow offline installations that do not require any internet connection.

Installation via xst is scriptable and will resolve dependencies. At the moment this will only work for packages that are available on the public package repository.

[eduarddrenth]

good to know! Is xst by default available in exist-db docker images (which are distroless)?

[line-o]

No, it's an external tool to work with any number of exist-db instances. Typically you would install it globally on your development machine. But you can declare it as a dependency in package.json and use it in an npm script as well.

[line-o]

@eduarddrenth can we close this issue?

[eduarddrenth]

xst works, but I need a running exit-db for it. This issue gives me a lot of trouble, I need to be able to build an exist-db image that includes my own xar and the dependencies it needs. Autodeploy doesn't work, there is no "apt install", xst is a runtime option for which I need to setup a separate "init" environment/container, xst install via multistage also isn't an option. Why can't autodeploy just retrieve dependencies?

[eduarddrenth]

moreover because of distroless curl, wget, shell are also no options.

[line-o]

@eduarddrenth at the moment this is the case. I will bring this up in next weeks community call. We could change autodeploy to attempt to retrieve dependencies. This would then also mean that the repository can be, or better has to be, set in conf.xml. Otherwise the system has no way of knowing from where to resolve them.

For the time being you still do have the option to provide all dependencies of your app in autodeploy in a multi-stage docker build.

[line-o]

@eduarddrenth I just read through this issue and was surprised I did not include that information before.

1. download all dependencies of your app as XARs
2. copy them to your autodeploy
3. ensure order of installation by renaming the contents of autodeploy (they will be installed in alphabetical order)

I know this is far from ideal. One of the reasons why I use xst instead. But the above can be coded into the docker file.

[eduarddrenth]

Thanks, I know this option and skipped it sofar because I do not want to manually figure out and download dependencies, manually manage versions of dependencies or manually ensure correct install order.

But perhaps this is my best option for now. I am also considering an xst init script in a dedicated docker container.

[eduarddrenth]

I choose the option to prepare libs manually because it gives me the advantage that the image is ready to run.

Perhaps I'll invest some time in #4553 to properly solve this issue. Shouldn't be too hard, because the functionality is already there in the dashboard/package manager.