Add a remote repo contents cache

[fmeum]

Repositories are cached in a regular remote cache as AC entries for a synthetic command with the predeclared input hash as the salt. The contents are represented as an output file for the marker file and an output directory for the contents.

Upon a cache hit, the metadata of the files comprising the repository is downloaded and injected into an in-memory file system that is overlaid on the external directory on the native file system. Downloads of file contents only occur when Bazel needs to read a file (e.g., a BUILD or .bzl file) or if a file is an input to an action executed locally. This can save time taken to execute repo rules and compute file digests and disk space required to store the contents of external repositories.

The output of du -h $(bazel info output_base) after bazel build //src:bazel-dev and a fully up-to-date remote cache

• without the flag:

$ bazel info peak-heap-size used-heap-size-after-gc
peak-heap-size: 271MB
used-heap-size-after-gc: 123MB
$ du -h $(bazel info output_base) | tail -1
1.3G	/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722

• with the flag:

$ bazel info peak-heap-size used-heap-size-after-gc
peak-heap-size: 266MB
used-heap-size-after-gc: 120MB
$ du -h $(bazel info output_base) | tail -1
380M	/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722

Some repos are still materialized eagerly, which may not be necessary. Patched http_archive also can't be cached yet, so these numbers are likely to improve with further work on this feature.

TODO:

• Should the debug events be removed?
• Remove repo cache entries without a matching recorded inputs file during GC

Fixes #6359
Fixes #22366

RELNOTES[NEW]: The results of reproducible repository rules without dependencies added at runtime (e.g., via repository_ctx.watch or .getenv) can now be cached in a regular HTTP or gRPC remote cache if the new --experimental_remote_repo_contents_cache startup option is provided.

[Wyverald]

some initial comments. will review deeper later, probably after the PR split (see below)

[fmeum]

Looks like I broke a test. I'll investigate that and also improve the recovery in case of a failed materialization.

[fmeum]

@Wyverald Should be fixed. Failing materialization should not have written out the marker file, so this should be fine as well.

[Wyverald]

Nice. I'll start the import myself.

[iancha1992]

@fmeum Bazel 9.0.0 rc1 cut is scheduled for Thursday. Please merge your fix by then to include it in rc1, or it will be cherry-picked into a later RC. If the fix won't make the Bazel 9 release deadline, then please remove this issue from the milestone. Thanks!

cc: @sluongng @meisterT @Wyverald

[iancha1992]

@fmeum Bazel 9.0.0 rc1 cut is scheduled for Thursday. Please merge your fix by then to include it in rc1, or it will be cherry-picked into a later RC. If the fix won't make the Bazel 9 release deadline, then please remove this issue from the milestone. Thanks!

cc: @sluongng @meisterT @Wyverald

Please note: The rc1 cut date has been rescheduled to Monday, November 3rd.

[iancha1992]

@bazel-io fork 9.0.0