document why and when --no-build-isolation is used in wheel-building CI

.pre-commit-config.yaml

@@ -0,0 +1,7 @@
+---
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: end-of-file-fixer
+      - id: trailing-whitespace

docs/docs/building.md

@@ -43,12 +43,14 @@ Some RAPIDS packages are pure Python and simply depend on other packages for com
 Those packages do not contain any Cython or CMake, and instead using `setuptools`.
 
 To build and install Python packages, you follow the same approach used to build any normal Python package:
+
+```shell
+python -m pip install /path/to/project
 ```
-python -m pip install --no-build-isolation /path/to/project
-```
+
 where `/path/to/project` is the path to a directory containing a `pyproject.toml` file.
-Note the specification of `--no-build-isolation` above; this assumes that you have already installed all the Cython and C++ build dependencies of your project into your environment.
-Currently this is a requirement of RAPIDS builds, which are not able to correctly specify the Python package requirements, [but this will be fixed in the future with the rapids-build-backend](packaging.md#pyproject).
+
+By default, this will create an isolated build environment, and `rapids-build-backend` will ensure that the necessary build-time dependencies are installed in that environment.
 
 ## build.sh scripts
 

docs/docs/ci.md

@@ -20,6 +20,47 @@ While each repository is free to define whatever scripts it requires, most RAPID
 Some repositories will include helper scripts in this directory to share logic between the above scripts.
 This practice is especially common if there are multiple wheels to build since the logic for building various wheels is usually similar.
 
+### Wheel-building scripts
+
+Some RAPIDS wheels rely on other wheels to provide build-time dependencies.
+For example, `pylibcudf-cu12` wheel builds install `libcudf-cu12` wheels to get `libcudf.so` and its headers.
+
+By default, `pip wheel` or `python -m build` will install such build-time dependencies into a temporary directory (see ["Python Packages"](build.md#python-packages) for details).
+The path to that directory is randomly chosen, and so the path that those build dependencies are installed to changes on every build.
+The filepaths for files used in the build are considered as part of `sccache`'s cache key ([see the `sccache` docs](https://github.com/mozilla/sccache/blob/main/docs/Caching.md)), and so this use of build isolation leads to a high rate of cache misses.
+
+To minimize the impact of this, since https://github.com/rapidsai/build-planning/issues/108 RAPIDS wheel-building scripts in CI have followed these rules:
+
+* pure Python (e.g. `dask-cuda`): build isolation
+* Python & Cython (e.g. `ucxx`): build isolation
+* mostly C++ (e.g. `libcudf`): no build isolation
+
+This pattern is followed to balance these two competing concerns:
+
+* improved cache hit rates for `sccache`, and therefore faster and less resource-intensive builds
+* testing that `rapids-build-backend` correctly generates package metadata
+* ensuring that the default approach for local builds from source (with isolation) will work out of the box for the non-C++ packages
+
+Without build isolation, all build-time dependencies have to be installed manually before invoking a wheel-building backend.
+That is why in `build_wheel_{lib}.sh` scripts, you'll often see a pattern similar to this:
+
+```shell
+rapids-dependency-file-generator \
+  --output requirements \
+  --file-key "py_build_${package_name}" \
+  --file-key "py_rapids_build_${package_name}" \
+  --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \
+| tee /tmp/requirements-build.txt
+
+python -m pip install \
+    -v \
+    --prefer-binary \
+    -r /tmp/requirements-build.txt
+
+# build with '--no-build-isolation', for better sccache hit rate
+export PIP_NO_BUILD_ISOLATION=0
+```
+
 ### Versioning
 
 RAPIDS libraries are versioned using [CalVer](https://calver.org/).

docs/docs/packaging.md

@@ -65,13 +65,9 @@ These suffixes are part of the package name, but they don't change the import na
 Each package will contain roughly the same contents, just built with different CUDA versions.
 In other words, `cudf-cu11` and `cudf-cu12` should not be installed together in the same environment, because they will clobber each other and cause unpredictable results.
 
-This difficulty is the reason why [the section on building Python packages recommended turning off build isolation](building.md#python-packages).
 The default dependency list specified in `pyproject.toml` for our packages does not contain the necessary suffix, and will therefore be incorrect.
-However, the correct dependency list may be generated using DFG, and that `requirements.txt` file can be installed into the environment by the developer.
-When we build wheels for distribution, we modify the metadata in `pyproject.toml` directly before building the package to address the above concerns.
-These modifications can be seen in our CI scripts: for instance, [you can see that cudf changes `rmm` to `rmm-cu11` or `rmm-cu12` with sed in its wheel building scripts](https://github.com/rapidsai/cudf/blob/branch-24.06/ci/build_wheel.sh#L42).
-If developers wish to build and/or install a specific RAPIDS wheel locally, they will have to make similar modifications.
-In the future, this will be fixed with the [`rapids-build-backend`](https://github.com/rapidsai/rapids-build-backend/) a PEP 517 build backend wrapper that essentially provides the glue between DFG and the underlying build backend (typically `scikit-build-core` or `setuptools`), ensuring that when a Python package is built the appropriate suffixes are added.
+However, the correct dependency list is generated at wheel-building time using [`rapids-build-backend`](https://github.com/rapidsai/rapids-build-backend/), a PEP 517
+build backend wrapper that provides the glue between `rapids-dependency-file-generator` and the build backend (typically `scikit-build-core` or `setuptools`).
 
 ## Nightlies