Switch Conda packages to .conda (instead of .tar.bz2)

[jakirkham]

Currently RAPIDS builds and publishes packages using .tar.bz2. However this was revamped in the newer .conda packages. The make a few important changes:

1. Use a top-level uncompressed .zip file
2. Means all .conda packages can be renamed to .zip and extracted
3. As .zip files allow random access, specific items can be retrieved
4. Metadata is placed in the top-level
   1. Contains info about the .conda metadata itself
   2. Package metadata (how it was built, dependencies needed, etc.)
   3. Compressed package contents (currently using Zstd)

This can help with solve times (no need to decompress .tar.bz2 to find metadata first). It can also help with download sizes & times (we noticed ~30% reduction in size of the legacy cudatoolkit when transitioning conda-forge)

---

To make the change, we would simply need to update our condarc at build time to include

conda_build:
  pkg_format: '2'

This can also be done with the following command:

conda config --set conda_build.pkg_format '2'

---

Also as pointed out by James in comment ( rapidsai/ci-imgs#176 (review) ) there are a few places where .tar.bz2 shows up in the RAPIDS org. Of these noted that some were:

• Data files (so not Conda packages)
• Archived repos (so not active)
• Dormant repos (haven't seen activity in years; candidates for archival)
• Forks of upstream repos (in some cases these overlap with the above categories)

Decided to skip the above. In the event we want to revitalize one of these projects, likely this is one of many changes that will be needed.

Of the remainder saw:

• Infrastructure tools
• Shared CI content
• One-off projects

Made a best effort to update these. With some one-off projects, they don't run CI yet; so, we can likely move ahead without those

[jakirkham]

It is worth noting that XGBoost Conda packages in RAPIDS are already built in the .conda format (for example. So we are already making some use of these today

[jakirkham]

Think this is all we need: rapidsai/ci-imgs#176

Would be good if others can confirm though

[jameslamb]

think this is all we need

We'll also need to update any places where *.tar.bz2 or similar is being used to list conda packages. See rapidsai/ci-imgs#176 (review)

[jakirkham]

Have made a best effort to submit PRs where appropriate. Included notes on this above. Would be good if someone else can recheck whether we got everything we deem relevant

[jameslamb]

I think you got everything, and I've reviewed them all: rapidsai/ci-imgs#176 (comment)

[jakirkham]

Think we have what we need in for PR: rapidsai/ci-imgs#176

AIUI gpuci-tools should no longer be used. Though James mentioned offline there might still be a few spots where it is used

If we do want to update gpuci-tools, we have PR: rapidsai/gpuci-tools#37

Will double check next week (after the long weekend) if we want to include that PR too

The others still open we concluded don't need to merge as they are to projects planned for archival

[jakirkham]

Have confirmed we don't need PR: rapidsai/gpuci-tools#37

More details in comment: rapidsai/gpuci-tools#37 (comment)

So have closed it out

Think we are ready for PR ( rapidsai/ci-imgs#176 ). Would recommend we do that after 24.10 ships

[jakirkham]

We have just shipped the final piece 🥳

Let's see how this goes

[jakirkham]

There were some hiccups due to a couple spots that still needed updates

These were resolved with the help of James and Bradley

More details in issue: rapidsai/shared-workflows#250