Skip to content

build(docker): add build-tools sysroot image and runner.topology config for arc-dind#5696

Merged
lpcox merged 3 commits into
mainfrom
arc-dind-build-tools-sysroot
Jun 30, 2026
Merged

build(docker): add build-tools sysroot image and runner.topology config for arc-dind#5696
lpcox merged 3 commits into
mainfrom
arc-dind-build-tools-sysroot

Conversation

@lpcox

@lpcox lpcox commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds support for ARC/DinD deployments where the runner and Docker daemon have separate filesystems and no root access is available in the workflow.

Closes #5693
Refs: github/gh-aw#42368, #5541, #5591

Key changes

1. Build-tools sysroot image (containers/build-tools/Dockerfile)

Ubuntu 22.04 image with system-level build infrastructure:

  • Compilers & linkers: gcc, g++, make, cmake, autoconf, binutils
  • Dev libraries: libssl-dev, libc6-dev, libicu-dev, zlib1g-dev
  • System utilities: bash, coreutils, git, curl, wget, jq
  • Agent dependencies: libcap2-bin (capsh), gosu, gnupg, gh

These packages require root to install and cannot be added at workflow runtime on ARC. The image is used as an init container that copies its filesystem into a named volume mounted at /host by the agent.

2. runner.topology config field

New config section in the AWF JSON schema:

{
  "runner": {
    "topology": "arc-dind",
    "sysrootImage": "ghcr.io/github/gh-aw-firewall/build-tools:latest"
  }
}

When topology is "arc-dind", AWF:

  • Emits a sysroot-stage init service in the compose file
  • Adds a sysroot named volume and mounts it on the agent at /host:ro
  • Warns if RUNNER_TOOL_CACHE is under /opt (invisible to DinD daemon)

This is the single stable contract between gh-aw and AWF for ARC deployments (see github/gh-aw#42368).

3. Release workflow

New build-build-tools job in .github/workflows/release.yml:

  • Builds multi-arch (amd64 + arm64) images
  • Published to GHCR with cosign signing and SBOM attestation
  • Follows the existing agent-act pattern

4. Documentation

Updated docs/arc-dind.md with:

  • Runner topology selector usage
  • Build-tools sysroot walkthrough
  • Tool cache redirection guidance for ARC

Architecture

build-tools image provides:          setup-* actions provide (on demand):
─────────────────────────────        ──────────────────────────────────
bash, capsh, coreutils               Go 1.22 (setup-go)
gcc, g++, make, ld                   Node 20 (setup-node)
libssl-dev, libc6-dev                Java 21 (setup-java)
pkg-config, cmake, autoconf          .NET 8.0 (setup-dotnet)
git, curl, wget                      Rust (setup-rust)
system linker, dynamic loader        Python 3.12 (setup-python)

build-tools = static system base (same for all workflows, requires root → baked at image build time).
Tool cache = dynamic, per-workflow (setup-* actions choose versions, no root needed → installed at workflow runtime).

Privilege hierarchy

When Root? What happens
Image build (release pipeline) apt-get install gcc make libssl-dev ...
ARC pod startup (K8s) DinD daemon starts, shared volumes created
Workflow steps (runner container) setup-* actions, checkout, config generation
Agent container (AWF) Build, test, agent execution

Tests

32 new tests covering:

  • sysroot-service.test.ts: Unit tests for sysroot service builder, image resolution, enable detection
  • compose-generator.test.ts: Integration tests for sysroot-stage in compose output
  • config-file-mapping.test.ts: Config mapper tests for runner.topology and sysrootImage
  • config-file-validation.test.ts: Schema validation for runner section

All 3353 tests pass.

Add support for ARC/DinD deployments where the runner and Docker daemon
have separate filesystems and no root access is available in the workflow.

Key changes:

- New containers/build-tools/Dockerfile: Ubuntu 22.04 image with
  build-essential, gcc, g++, make, cmake, dev libraries, and system
  utilities. Provides the chroot base that the agent needs on ARC/DinD.

- New runner.topology config field (standard | arc-dind):
  Single declarative knob that activates sysroot staging, network
  isolation defaults, and tool cache validation for ARC/DinD.

- New runner.sysrootImage config field: Override the default
  build-tools image for custom sysroot setups.

- Compose generator emits sysroot-stage init service when topology
  is arc-dind: copies build-tools image FS into a named volume via
  cp -a, agent mounts it read-only at /host.

- Tool cache warning when RUNNER_TOOL_CACHE is under /opt on ARC/DinD
  (invisible to the DinD daemon filesystem).

- Release workflow: new build-build-tools job builds and publishes
  the build-tools image to GHCR with cosign signing and SBOM.

- Updated docs/arc-dind.md with runner.topology usage, build-tools
  walkthrough, and tool cache redirection guidance.

- 32 new tests covering sysroot service, config mapping, schema
  validation, and compose generation.

Closes #5693
Refs: gh-aw#42368, #5541, #5591

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 29, 2026 23:20
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit e724531

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 98.37% 98.42% 📈 +0.05%
Statements 98.30% 98.34% 📈 +0.04%
Functions 99.54% 99.54% ➡️ +0.00%
Branches 94.43% 94.38% 📉 -0.05%
📁 Per-file Coverage Changes (2 files)
File Lines (Before → After) Statements (Before → After)
src/compose-generator.ts 98.7% → 98.9% (+0.20%) 98.7% → 98.9% (+0.20%)
src/workdir-setup.ts 92.7% → 94.5% (+1.82%) 92.7% → 94.5% (+1.82%)
✨ New Files (1 files)
  • src/services/sysroot-service.ts: 100.0% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

@lpcox

lpcox commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator Author

The documentation build workflow (run 28409144039) actually completed successfully -- the build-preview job passed with conclusion: success. This bot comment appears to have been posted before the workflow finished. No action needed.

@lpcox lpcox changed the title [ARC-DinD] Add build-tools sysroot image and runner.topology config build(docker): add build-tools sysroot image and runner.topology config for arc-dind Jun 29, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 29, 2026 23:26
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 98.37% 98.42% 📈 +0.05%
Statements 98.30% 98.35% 📈 +0.05%
Functions 99.54% 99.54% ➡️ +0.00%
Branches 94.43% 94.45% 📈 +0.02%
📁 Per-file Coverage Changes (2 files)
File Lines (Before → After) Statements (Before → After)
src/compose-generator.ts 98.7% → 98.9% (+0.28%) 98.7% → 98.9% (+0.28%)
src/workdir-setup.ts 92.7% → 94.5% (+1.82%) 92.7% → 94.5% (+1.82%)
✨ New Files (1 files)
  • src/services/sysroot-service.ts: 100.0% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces first-class ARC/DinD support by adding a “sysroot staging” mechanism (a build-tools image copied into a named volume mounted at /host) and a new runner.topology configuration surface intended to activate ARC/DinD-specific behavior with minimal caller configuration.

Changes:

  • Add runner.topology (standard | arc-dind) plus runner.sysrootImage to the config schema and config-file mapping.
  • Generate a sysroot-stage init service and sysroot:/host:ro volume mount when ARC/DinD topology is enabled.
  • Add a new containers/build-tools image and publish it from the release workflow; update ARC/DinD documentation accordingly.
Show a summary per file
File Description
src/types/platform-options.ts Adds RunnerTopology type and new platform-level options (runnerTopology, sysrootImage).
src/types/docker.ts Extends Docker service typing with entrypoint.
src/services/sysroot-service.ts Introduces sysroot-stage init service builder + sysroot enable/image resolution helpers.
src/services/sysroot-service.test.ts Adds unit tests for sysroot enablement/image resolution/service generation.
src/config-mapper.ts Maps runner.topology and runner.sysrootImage from config file into CLI options.
src/config-file.ts Extends AwfFileConfig with a new runner section.
src/config-file-validation.test.ts Adds schema validation tests for the new runner section.
src/config-file-mapping.test.ts Adds mapping tests for runner.topology and runner.sysrootImage.
src/compose-generator.ts Adds sysroot-stage service generation, tool-cache warning, and sysroot named volume mounting.
src/compose-generator.test.ts Adds compose output tests for sysroot-stage service and sysroot volume behavior.
src/awf-config-schema.json Updates generated schema to include the new runner section.
schemas/token-usage.schema.json JSON formatting-only changes (enum/property formatting).
docs/awf-config.schema.json Updates documented schema to include the new runner section.
docs/arc-dind.md Documents runner.topology usage and sysroot/tool-cache behavior for ARC/DinD.
containers/build-tools/Dockerfile Adds the build-tools sysroot image (Ubuntu 22.04 + build essentials/libs/utilities).
.github/workflows/release.yml Adds a job to build/push/sign/attest the build-tools image and gates release on it.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 16/16 changed files
  • Comments generated: 5
  • Review effort level: Low

Comment thread src/compose-generator.ts
Comment on lines +252 to +256
if (sysrootActive) {
// The sysroot named volume provides /host content (system binaries, libs, etc.)
// via the sysroot-stage init container instead of per-directory bind mounts.
agentVolumes.push('sysroot:/host:ro');
}
Comment thread src/services/sysroot-service.ts Outdated
Comment thread src/config-mapper.ts
Comment on lines 134 to +137
platformType: config.platform?.type,

runnerTopology: config.runner?.topology,
sysrootImage: config.runner?.sysrootImage,
Comment thread docs/arc-dind.md Outdated
Comment on lines +17 to +24
When `runner.topology` is set to `"arc-dind"`, AWF applies overridable defaults:

| Behavior | Default | Override |
|----------|---------|----------|
| Network isolation (no NET_ADMIN) | `true` | `network.isolation` |
| DinD pre-stage dirs | `true` | `dind.preStageDirs` |
| Sysroot image for `/host` base | `build-tools:<tag>` | `runner.sysrootImage` |
| Tool cache warning if under `/opt` | Emitted | Set `RUNNER_TOOL_CACHE` to shared path |
Comment thread src/services/sysroot-service.ts Outdated
@lpcox

lpcox commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot address review feedback

Copilot AI commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@copilot address review feedback

Addressed the review feedback in commit b9b2bb3.

This wires runner.topology/runner.sysrootImage into runtime config, prevents base-system /host/* bind mounts from shadowing the sysroot volume in arc-dind mode, switches sysroot-stage to network_mode: 'none', hardens /lib64 copy error handling, updates the related tests, and corrects the ARC/DinD docs wording to match implemented behavior.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Security Guard failed. Please review the logs for details.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

Copy link
Copy Markdown
Contributor
  • build(docker): add build-tools sysroot image and runner.topology config for arc-dind ✅
  • GitHub MCP tool connectivity ✅
  • GitHub.com connectivity ✅
  • Agent file I/O ✅
  • Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) ✅

Overall: PASS

cc @lpcox

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

Copy link
Copy Markdown
Contributor

build(docker): add build-tools sysroot image and runner.topology config for arc-dind
✅ GitHub merged PR review
✅ GitHub PR list query
✅ GitHub.com title check
✅ Smoke file write/read
✅ npm ci && npm run build
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Status Detail
S1: Module Loading otel.js loads cleanly; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled
S2: Test Suite 59 tests pass, 0 fail across 2 suites (otel.test.js, otel-fanout.test.js)
S3: Env Var Forwarding src/services/api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME to the api-proxy container
S4: Token Tracker Integration onUsage callback exists at line 283/324 of token-tracker-http.js as the OTEL hook point
S5: OTEL Diagnostics No live containers (DinD not available in sandbox); otel.js gracefully degrades to file fallback (/var/log/api-proxy/otel.jsonl) when no OTLP endpoint is configured

All scenarios pass. OTEL tracing integration is fully functional.

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3 ❌ NO
Node.js v24.17.0 v22.23.0 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ Not all versions match — Python and Node.js differ between host and chroot environments.

Tested by Smoke Chroot

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test: Copilot PAT — PASS

Test Status
GitHub MCP connectivity
GitHub.com HTTP ✅ 200
File write/read

PR: build(docker): add build-tools sysroot image and runner.topology config for arc-dind
Author: @lpcox
Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Overall: PASS

🔑 PAT report filed by Smoke Copilot PAT

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct Mode) ✅ PASS

  • ✅ GitHub MCP connectivity verified (2 recent PRs fetched)
  • ✅ GitHub.com reachable (HTTP 200)
  • ✅ File I/O operational
  • ✅ Direct BYOK inference confirmed (agent → api-proxy sidecar → api.githubcopilot.com)

Mode: Direct BYOK via COPILOT_PROVIDER_API_KEY (real key held by api-proxy, dummy placeholder in agent)

All tests passed. See #5696.

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox

  • GitHub MCP Connectivity: ✅
  • GitHub.com Connectivity: ✅
  • File Write/Read Test: ✅
  • BYOK Inference Test: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

Copy link
Copy Markdown
Contributor

Gemini Engine Smoke Test Results

  1. GitHub MCP Testing: ❌ (Tools not found, only 1 PR in history: docs: add B7 failure mode (EACCES during chroot-home cleanup in rootless Docker) #5692)
  2. GitHub.com Connectivity: ❌ (SSL error 35/Connection blocked)
  3. File Writing Testing: ✅
  4. Bash Tool Testing: ✅

Overall Status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results — Services Connectivity

Check Result
Redis PING ❌ Timeout (no response on host.docker.internal:6379)
PostgreSQL pg_isready ❌ No response on host.docker.internal:5432
PostgreSQL SELECT 1 ❌ Not attempted (pg_isready failed)

Overall: FAIL

host.docker.internal resolves to 172.17.0.1 but both ports 6379 and 6432 are unreachable (connection timeout). GitHub Actions service containers are not accessible from this runner environment.

🔌 Service connectivity validated by Smoke Services

@github-actions github-actions Bot mentioned this pull request Jun 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for #5696 · 90.5 AIC · ⊞ 7.8K ·

@lpcox lpcox merged commit 32b519c into main Jun 30, 2026
89 of 90 checks passed
@lpcox lpcox deleted the arc-dind-build-tools-sysroot branch June 30, 2026 00:13
lpcox added a commit that referenced this pull request Jun 30, 2026
Reconcile duplicate runner topology definitions:
- Remove runnerTopology/sysrootImage from PlatformOptions (now in RunnerOptions)
- Add "standard" to RunnerOptions.runnerTopology enum for consistency
- Remove duplicate runnerTopology property in build-config.ts
- Fix test assertion for updated topology enum values

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github-actions Bot added a commit that referenced this pull request Jun 30, 2026
Add missing `runner` section to the Section 4 Data Model table and
add Section 5 CLI Mapping entries for `runner.topology` and
`runner.sysrootImage` in docs/awf-config-spec.md.

These fields were introduced in PRs #5696 and #5697 and correctly
propagated to src/awf-config-schema.json, docs/awf-config.schema.json,
src/types/runner-options.ts, src/types/wrapper-config.ts,
src/config-file.ts, and src/config-mapper.ts — but the spec doc was
not updated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lpcox added a commit that referenced this pull request Jun 30, 2026
* fix: propagate runner config fields to spec

Add missing `runner` section to the Section 4 Data Model table and
add Section 5 CLI Mapping entries for `runner.topology` and
`runner.sysrootImage` in docs/awf-config-spec.md.

These fields were introduced in PRs #5696 and #5697 and correctly
propagated to src/awf-config-schema.json, docs/awf-config.schema.json,
src/types/runner-options.ts, src/types/wrapper-config.ts,
src/config-file.ts, and src/config-mapper.ts — but the spec doc was
not updated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* docs: fix sysrootImage default registry description

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Landon Cox <landon.cox@microsoft.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ARC-DinD] Add versioned build-tools sysroot image for build-test workflows

3 participants