[question] conan-center-backup-sources artifactory repo no longer being updated?

[samuel-emrys]

What is your question?

Hi Team,

I noticed that in the recent shift to the new CI system, the configuration regarding source backup was removed in 2774bc9#diff-0595100739eb70cf48980960610cd14384c1caa88dc1198b3d437145812a2f77L8.

I'm currently mirroring the conan-center-backup-sources repo to make sources available for my internal and offline builds, and I've noticed recently an increase in build failures due to missing sources. Has the upload/download policy for this artifactory repo changed? Are you still using it as part of your CI process?

Cheers

[memsharded]

Hi @samuel-emrys

Thanks for your question.

Indeed, in the recent change of the new CI pipelines, with Conan 2 support only, the backup sources was not implemented, the priority was to get the core building pipeline there, and some non critical things were postponed for later development, like the backup sources. It will be added back when possible, likely soon.

In any case, please note that the backup sources feature is intended as a last-resort fallback if the original sources are unreachable - and only as a convenience. It must not be used as the primary source, the original URL should always be tried first. And much like the rest of Conan Center, it is not intended to be mirrored nor copied - attempts to do massive copies of the repository or its resources might be flagged as attacks or abuse automatically and blocked. There is also no guarantee that this resource is permanently available - for example we might remove artifacts from the backup sources repo for unmaintained packages (those which recipes or versions are removed from the conan-center-index repository).

Then, the recommendation would be:

• When using our remote for backup-sources (again with the note that there is no guarantee resources are always available), make sure that it is configured to be attempted last, after all the original URLs and mirrors
• For populating your own backup-sources private repository, it is possible to parse the conandata.yml to retrieve the URLs and sha256 checksums to retrieve the files, and then upload them to the private repo (with just a couple of changes, like using the sha256 as filename, and recommended to submit the accompanying .json file)
• If the backup sources feature is desired and used in production, it would be discouraged to try to mirror the ConanCenter repository.
• The way to implement it would be to configure it for the on-prem build of packages from source from conan-center-index repo
• Recall that one of the recommendations for many production cases for consuming ConanCenter packages is building them from source (https://docs.conan.io/2/devops/using_conancenter.html). It seems to me that a project that implements its own "backup sources" repository in their own private server would easily fall in this category.