Discussion: Breakup and Slimming of Docker/Singularity Dependencies

[gdevenyi]

Currently, nf-neuro relies heavily on the single container developed by scilus, this introduces a number of potential problems

• large download regardless of use case
• upgrade of individual tools is difficult
• adding more tools requires update/rebuild of container

I am investigating breakuing up the container with the following goals

• use upstream containers wherever possible
• move container version/details into a centrally managed configuration instead of in individual pipeline steps

Problems I see:

• QC generation for many steps appears to be integrated into the main container using tools other than the "main tool" I'd like to breakout
• Q: can we make a slimmer "QC" container which does this?
• What to do about tools without upstream containers
  • contribute a container to upstream?
  • how to standardize containers for pythonic or other packages where upstream "shouldn't" make containers

[gagnonanthony]

Hi @gdevenyi !

Thanks for this thread. I agree that using upstream containers is the preferred option, as it frees us from updating versions, etc. What did you have in mind for a centrally managed configuration for Docker versions? Like a centralised list of modules with their respective containers?

For the problems you listed, I might have some possible solutions:

• QC steps can be moved outside, which will increase the number of modules run during a pipeline execution, but I don't think this would be a major issue (what do you think @AlexVCaron ?). Also, most QC steps have the same backbone in terms of steps and operations and could be generalized in a Python script that does it all.
• Making a slimmer QC container that contains everything needed would be great, and linked with problem 3, a pythonic package can be built using Seqera Containers if it is available through pypi or conda. This might be a viable way to create a container with fewer downsides?

Anyway, those are my thoughts! Happy to discuss further

[gdevenyi]

What did you have in mind for a centrally managed configuration for Docker versions? Like a centralised list of modules with their respective containers?

Something like this. Like a config file for the nf-neuro collection, so that its clear the versions of each tool used in a more effective way than having to examine each main.nf file.

[AlexVCaron]

As @gdevenyi, I believe we have to piggyback on upstream images to solve anything, just considering the size of the scilus artefact, but it comes with its own limitations :

• image availability
• image ownership
• efficiency and size
• potentially limited tooling (like basic image math)

Here is an excerpt of my exploration in the matter up to now. I think solving them completely would require hands-on development and us to maintain some kind of software and/or image repository. I think we'd all prefer to avoid this. There are hopeful developments in communities like Neurodocker, use by neurodesktop for example, but they won't manage images for us. We could start with putting in place practices and recommendations on container definition, with little added work for us :

1. If an image by the main development group is available, use it. We can maintain a list on the doc website.
2. If not, ask for one. Groups might be open to start developing their own.
3. For pythonic (with pip) and conda dependencies, use Seqera Containers to generate the image.
4. Ask us to build one ? Or list requirements for contributors to build one themself.

That doesn't solve QC. To tackle this, I believe we should remove QC from modules. There are plenty of benefits in doing so :

• The same QC module is used for multiple modules anyway.
• We could pass data to QC modules using the same logic as for the version file. As long as naming is right, multiqc will work.
• It is simple, demands less knowledge from contributors and doesn't add much maintenance.

It is even better if we consider the topic channels that just released out of preview in nextflow 25. With it, QC becomes childishly easy to add to a module. You add a topic channel to the module that pushes data to QC to a correctly named channel. The QC module listens to that topic and produces QC output on demand, when included in the pipeline. I can produce a quick PoC for this.

[AlexVCaron]

What did you have in mind for a centrally managed configuration for Docker versions? Like a centralised list of modules with their respective containers?

Something like this. Like a config file for the nf-neuro collection, so that its clear the versions of each tool used in a more effective way than having to examine each main.nf file.

I think implementing this in config is feasible, but we need to check if nf-core linting tools supports it. It didn't in the past when we tried with versions prior to 3.2.0. If not, we should consider opening an issue to add support for it.

[gdevenyi]

Related to the config idea, here's a case where "1.7" would be specified "somewhere else" and the module would pull in the version, making the module more of a template
#153