Revise DR-005-infra design document #2262
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,242 @@ | ||||||
| DR-005-infra: Directory Structure for S-CORE Bazel C++ Toolchain Configuration Repository | ||||||
| ==================================================================================== | ||||||
|
Member
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| Overview | ||||||
| -------- | ||||||
|
|
||||||
| This repository acts as a **pure configuration layer** for C++ toolchains used in Bazel builds. | ||||||
| It contains **only configuration files** (``BUILD.bazel``, ``.bzl``), while all **toolchain | ||||||
| binaries** are downloaded from external repositories via ``http_archive``. | ||||||
|
|
||||||
| Platform definitions are **not** part of this repository—they are provided separately by the | ||||||
| S-CORE Bazel Platforms module. | ||||||
|
|
||||||
| This structure supports: | ||||||
|
|
||||||
| - Multiple compiler versions | ||||||
| - Multiple OS/CPU combinations | ||||||
| - Clean layering following Bazel best practices | ||||||
| - Bzlmod-compatible module extensions | ||||||
| - Reusability and long-term maintainability | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| High-Level Goals | ||||||
| ---------------- | ||||||
|
|
||||||
| - Provide a **central place** for all C++ toolchain configuration logic. | ||||||
| - Separate toolchain **binaries** from **configurations**. | ||||||
| - Provide scalable support for: | ||||||
|
|
||||||
| - Linux (x86_64, arm64) | ||||||
| - QNX (x86_64, arm64) | ||||||
|
There was a problem hiding this comment. Given that Linux has multiple distributions and S-core currently uses three different ones (Ubuntu for dev containers, EB-Linux and Red Hat Linux for reference integration), should we consider creating separate platform and tool chain definitions for each (e.g., linux-ubuntu, linux-eb, linux-redhat) or as the toolchain contains only rules/config it should be compatible for different Linux flavours?
Member
Author
There was a problem hiding this comment. These are distributions and as such should not depend on platform constraints (OS and CPU). We can add flags to platform if we want to have some separation between them, but I would like first to know what is delta between these 3 in terms of configuration. |
||||||
| - Multiple compiler variants (GCC, Clang, QCC) | ||||||
| - Multiple versions per compiler | ||||||
|
|
||||||
| - Allow all consuming repositories to pull toolchains via **Bzlmod** with simple configuration. | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| Proposed Directory Structure | ||||||
| ---------------------------- | ||||||
|
|
||||||
| .. code-block:: text | ||||||
|
|
||||||
| <repo_root>/ | ||||||
| ├── MODULE.bazel | ||||||
| ├── BUILD.bazel | ||||||
| │ | ||||||
| ├── toolchains/ | ||||||
| │ ├── linux/ | ||||||
| │ │ ├── x86_64/ | ||||||
| │ │ │ ├── gcc/ | ||||||
| │ │ │ │ ├── 12.2.0/ | ||||||
| │ │ │ │ │ ├── BUILD.bazel | ||||||
| │ │ │ │ │ ├── cc_toolchain_config.bzl | ||||||
| │ │ │ │ │ └── third_party/ | ||||||
| │ │ │ │ │ └── gcc.BUILD | ||||||
| │ │ │ │ └── ... | ||||||
| │ │ │ └── llvm/ | ||||||
| │ │ │ ├── 16/ | ||||||
| │ │ │ └── ... | ||||||
| │ │ │ | ||||||
| │ │ └── arm64/ | ||||||
| │ │ ├── gcc/ | ||||||
| │ │ │ ├── 12.2.0/ | ||||||
| │ │ │ └── ... | ||||||
| │ │ └── llvm/ | ||||||
| │ │ ├── 16/ | ||||||
| │ │ └── ... | ||||||
| │ │ | ||||||
| │ ├── qnx/ | ||||||
| │ │ ├── x86_64/ | ||||||
| │ │ └── arm64/ | ||||||
| │ │ └── qcc/ | ||||||
| │ │ ├── 7.1/ | ||||||
| │ │ ├── 8.0/ | ||||||
| │ │ └── ... | ||||||
| │ │ | ||||||
| │ ├── rules/ | ||||||
| │ │ ├── BUILD.bazel | ||||||
| │ │ ├── gcc.bzl | ||||||
| │ │ ├── llvm.bzl | ||||||
| │ │ └── qcc.bzl | ||||||
| │ │ | ||||||
| │ ├── extentions/ | ||||||
| │ │ ├── BUILD.bazel | ||||||
| │ │ ├── gcc.bzl | ||||||
| │ │ ├── llvm.bzl | ||||||
| │ │ └── qcc.bzl | ||||||
| │ │ | ||||||
| │ └── common/ | ||||||
| │ ├── cc_config_common.bzl | ||||||
| │ ├── toolchain_features.bzl | ||||||
| │ └── version_matrix.bzl | ||||||
| │ | ||||||
| ├── tests/ | ||||||
| │ └── ... | ||||||
| │ | ||||||
| └── docs/ | ||||||
| ├── repo_overview.md | ||||||
| ├── naming_convention.md | ||||||
| ├── how_to_add_toolchain.md | ||||||
| └── version_policy.md | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| Structure Rationale | ||||||
| ------------------- | ||||||
|
|
||||||
| 1. OS → Architecture → Compiler → Version Hierarchy | ||||||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| This structure provides: | ||||||
|
|
||||||
| - Clear separation of toolchains | ||||||
| - Easy extension when adding: | ||||||
|
|
||||||
| - New compilers | ||||||
| - New versions | ||||||
| - New platforms | ||||||
|
|
||||||
| Example: | ||||||
|
|
||||||
| .. code-block:: text | ||||||
|
|
||||||
| toolchains/linux/x86_64/gcc/13.1.0/ | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| 2. Clean Separation from Binaries | ||||||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| All binaries remain outside: | ||||||
|
|
||||||
| - Toolchain binaries provided by vendor archives | ||||||
| - Retrieved via ``http_archive`` | ||||||
| - Only configuration logic lives here | ||||||
|
|
||||||
| Each versioned toolchain directory pulls paths from package repository | ||||||
| ``toolchain_cpp_packages`` (full name: ``S-CORE Bazel CPP Toolchain packages``) such as: | ||||||
|
|
||||||
| :: | ||||||
|
|
||||||
| @score_toolchain_linux_x86_64_gcc12_2_0 | ||||||
|
|
||||||
| or: | ||||||
|
|
||||||
| :: | ||||||
|
|
||||||
| @score_toolchain_linux_arm64_gcc13_1_0 | ||||||
|
|
||||||
| This ensures the configuration repo remains lightweight and portable. | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| 3. Shared Common Logic | ||||||
| ~~~~~~~~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| All shared logic lives under: | ||||||
|
|
||||||
| :: | ||||||
|
|
||||||
| toolchains/common/ | ||||||
|
|
||||||
| Contained files: | ||||||
|
|
||||||
| +------------------------------+-----------------------------------------------+ | ||||||
| | File | Purpose | | ||||||
| +==============================+===============================================+ | ||||||
| | ``cc_config_common.bzl`` | Common flags and default configurations | | ||||||
| +------------------------------+-----------------------------------------------+ | ||||||
| | ``toolchain_features.bzl`` | Helper functions for constructing features | | ||||||
| +------------------------------+-----------------------------------------------+ | ||||||
| | ``version_matrix.bzl`` | Single source of truth for supported versions | | ||||||
| +------------------------------+-----------------------------------------------+ | ||||||
|
|
||||||
| This avoids duplication across the full matrix of platform/compiler/version combinations. | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| 4. Bzlmod Extension | ||||||
| ~~~~~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| The repository exposes module extensions defined in the ``extentions`` directory. | ||||||
|
|
||||||
| Example: | ||||||
|
|
||||||
| :: | ||||||
|
|
||||||
| module_extension( | ||||||
| name = "gcc", | ||||||
| extension_file = "//toolchains/extentions:gcc.bzl", | ||||||
| ) | ||||||
|
|
||||||
| or: | ||||||
|
|
||||||
| :: | ||||||
|
|
||||||
| module_extension( | ||||||
| name = "llvm", | ||||||
| extension_file = "//toolchains/extentions:llvm.bzl", | ||||||
| ) | ||||||
|
|
||||||
| The extension: | ||||||
|
|
||||||
| - Loads toolchain definitions | ||||||
| - Provides interface for root module to set desired version and options | ||||||
| - Maps toolchains to the Bazel Platforms module | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| 5. Documentation | ||||||
| ~~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| A dedicated ``docs/`` directory ensures maintainability: | ||||||
|
|
||||||
| - **repo_overview.md** — high-level purpose | ||||||
| - **naming_convention.md** — toolchain naming scheme | ||||||
| - **how_to_add_toolchain.md** — step-by-step contributor guide | ||||||
| - **toolchain_testing_and_validation** — integration-gate validation specification | ||||||
| - **version_policy.md** — rules for adding/deprecating versions | ||||||
|
|
||||||
| ---- | ||||||
|
|
||||||
| Benefits Summary | ||||||
| ---------------- | ||||||
|
|
||||||
| +-------------------+--------------------------------------------------------------+ | ||||||
| | **Benefit** | **Explanation** | | ||||||
| +===================+==============================================================+ | ||||||
| | Scalable | Clean hierarchy allows adding platforms and compilers | | ||||||
| | | easily. | | ||||||
| +-------------------+--------------------------------------------------------------+ | ||||||
| | Modular | Fully compatible with Bzlmod. | | ||||||
| +-------------------+--------------------------------------------------------------+ | ||||||
| | Lightweight | No binaries stored in the repository. | | ||||||
| +-------------------+--------------------------------------------------------------+ | ||||||
| | Consistent | Shared logic centralized under ``toolchains/common``. | | ||||||
| +-------------------+--------------------------------------------------------------+ | ||||||
| | Traceable | Version matrix gives a clear overview of supported versions. | | ||||||
| +-------------------+--------------------------------------------------------------+ | ||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nradakovic am I right that this would go into the direction of implementing all the toolchains on our own? What about available opensource toolchains for GCC and LLVM?
Or would it be viable to use these available toolchains as part of this setup?
I would like an evaluation of the different options. This helps in the future if we ever want to reevaluate or understand why we decided this way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GCC doesn't exist in OS world (at least I'm not aware) and LLVM seems to be a bit heavy on resource usage (not optimized for CI). That's why BMW has it's own implementation of LLVM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would still be great to have this as part of the document.
GCC does exist here: https://github.com/f0rmiga/gcc-toolchain
It works as advertised, but I can't judge the quality of the implementation.
My proposal is to then upstream the necessary changes. I'm sure that other people also use the toolchain in a CI environment. So there should be interest in having this in the OS world.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, it's not my call when LLVM will be outsourced from SWH end, I cannot enforce this. For now we use the one that is available. Once it's available we can add it into repo without too much effort.
The other GCC solution doesn't really fit our needs. The interface is quite close, we cannot inject additional features (my implementation will support that) and there is no way of redefining existing features. Also I cannot point to different package. This config only support certain versions of GCC.