Use conda packages cache as we do with pip

[humitos]

At the moment our conda command uses these settings,

docs@cec884de0fdf:~$ conda info

SNIP

       root environment : /home/docs/.conda  (writable)
    default environment : /home/docs/.conda
       envs directories : /home/docs/.conda/envs
          package cache : /home/docs/.conda/pkgs

SNIP

For pip we use the --cache-dir attribute so we save all the packages in the project's directory and we avoid to re-download them again.

In conda we are doing nothing. I found that we can use CONDA_PKGS_DIRS env variable for this (I didn't find an attribute yet), so:

docs@cec884de0fdf:~$ CONDA_PKGS_DIRS=/tmp conda info

SNIP

       root environment : /home/docs/.conda  (writable)
    default environment : /home/docs/.conda
       envs directories : /home/docs/.conda/envs
          package cache : /tmp

SNIP

We could use this env variable to point to something like ./user_builds/<project-slug>/.cache/conda

[shoyer]

My project's builds on RTD have been timing-out recently. The biggest contributor (likely ~300 seconds) appears to be time spent downloading packages in packages as part of conda env create.

So if we could setup caching, that would make a big difference for us.

[jorisvandenbossche]

I want to second this. I am having similar problems recently (#4071), and it looks the download + install of the conda packages varies to take between 500- 800 seconds (once even up to 1500 seconds), of which the download is a significant part I suspect.
Running the docs itself once everything is installed does not take that long, but we have some quite heavy dependencies (eg the full gdal and deps stack). So caching the conda packages would also help a lot.

[stsewd]

Just to clarify, if this is implemented you will still have the timeout problem in your first recent builds. How is this?

1. The build is triggered
2. You get the timeout (but rtd cache the downloaded Conda packages)
3. The second time probably there is no timeout
4. After some time rtd will clear the cache to save disk space
5. Back to 1)

[humitos]

Once I thought that maybe a global cache for conda/pip would be great, but then I realize that this has two main problems:

• the packages cache need to be replicated on each builder server
• if a package is malicious (for some reason) and is saved in the cache all the other people will use that one

Then, we talked about devpi (which is great!) and we opened an issue for using it under development on intermittent internet connections: #3553

Although, devpi could be used to solve this problem of a shared cache because:

• we need to store only the packages that were used at least once
• we could use management commands to keep it clean
• we could share all those packages within our intranet and do not repeat the storage (repointing the index URL to our devpi server)

Anyway, these are just ideas and unfortunately, they are not on our current roadmap.

[shoyer]

Just to clarify, if this is implemented you will still have the timeout problem in your first recent builds.

This is true -- but it would still be better than our current state of affairs!

[humitos]

Unfortunately, I didn't find a proxy for conda channels similar to devpi for PyPI.

I found conda-mirror (https://github.com/Valassis-Digital-Media/conda-mirror) which creates a mirror of an upstream channel. Although it's not exactly what we need, it could be useful to link it here. There is possibles solution around using a custom mirror, but they are too complex.

[stsewd]

We no longer cache things, and we have bigger builders for users using conda