Use devpi as PyPI cache

[humitos]

This is a working POC to use a devpi-server on development. Anyways, do not merge! This is more a way to share the diff for now and be able to apply it when needed.

Use devpi-server inside docker (when running under docker-compose) to speed up python package installation when triggering builds.

Adds a new service for docker-compose that spin up a devpi-server to cache all the Python packages used by the builders.

If configured, --index-url and --trusted-host is added to all the pip commands.

Closes #3553

[ericholscher]

Is this meant to be used in prod, or only on dev? It could perhaps speed up builds if we had it in prod, but I don't know if we want to maintain it, and fastly is likely fast enough already to minimize this being a huge install-time issue.

[humitos]

The original idea was to use when developing since if you are doing some work related to the build process, it takes a lot of time to download each package over and over again (even more now that we are cleaning up after build ends). Also, when working with limited internet access.

Although, I'm not opposite to use this in production if there is a real difference between downloading from internal cache than from internet.

[humitos]

Marking as WIP for now since the docker file could be moved to common repository now and we want to integrate with invoke commands as well.

[humitos]

This PR is updated and ready for review. Some code was moved to the linked common PR.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

This PR needs to be updated with latest changes con common repository for docker compose setup. I will try to do this when I have some time.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

This is ready for review. It should work in a transparent way on development environment. It can be disabled with --no-devpi

[stsewd]

No really sure this is a big improvement, if I'm having a slow connection I just set this setting to false

readthedocs.org/readthedocs/settings/docker_compose.py

Line 51 in b53ebff

RTD_CLEAN_AFTER_BUILD = True

Also, this requires to spin another container, docker is already slow for some people.

[humitos]

No really sure this is a big improvement, if I'm having a slow connection I just set this setting to false

This requires to re-download all the packages again for every different project.

Also, this requires to spin another container, docker is already slow for some people.

You can decide to not use this just by passing --no-devpi. Which is easier than modifying a settings file.

[humitos]

Any objection on merging this?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

I'll try to tackle this next week and merge it.

[humitos]

This didn't have too much traction and nobody has complained about slow connections in a while. Let's come back to this when we are allowed to travel and be behind a cell connection 😄