🩺 Runner Doctor UpdateRunner Doctor update: A16 (sysroot filter drops workspace mounts), A17 (build-tools unpin-able), B8 (pre-flight EACCES on persis [Content truncated due to length]

[github-actions]

Summary

Scan window: issues and PRs updated 2026-07-01 – 2026-07-07
Items reviewed: ~22 issues/PRs (searched: ARC, DinD, self-hosted, GHES, GHEC, DOCKER_HOST, chroot, musl, Alpine, capsh, IPv6, corporate proxy, resolv.conf, passwd, toolcache, one-shot-token, ghe.com, RHEL, CA bundle)
Genuinely new lessons: 6 (5 new failure mode rows + 1 citation/fix-text update to B7)

Citation numbers in the existing catalog top out at github/gh-aw-firewall#5963. All items below carry higher numbers and are not yet represented in the catalog.

---

Proposed knowledge-base changes

File: .github/workflows/shared/self-hosted-failure-modes.md

New row A16 (insert after A15)

| A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including #5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 |

New row A17 (insert after A16)

| A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:<digest>` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Open — not yet fixed** (PR github/gh-aw-firewall#5986 in review): adds `'build-tools'` to `IMAGE_DIGEST_KEYS` and threads `ParsedImageTag` through `SysrootServiceParams`. Until merged, security-sensitive deployments cannot digest-pin the sysroot-stage image. | `awf --image-tag build-tools=sha256:abc ...` — throws "invalid key" if fix not installed | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 |

Update row B7 (citation + fix-text)

In the Fix / flag column, append:

; additional hardening (#5766): changes chown && chmod to chown 2>/dev/null; chmod so chmod always runs as a fallback even when chown fails within the rootless UID namespace.

In the Citations column, append: , github/gh-aw-firewall#5766

New row B8 (insert after B7)

| B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 |

New row B9 (insert after B8)

| B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 |

New row C8 (insert after C7)

| C8 | `400 bad request: Authorization header is badly formatted` on **GHEC (`*.ghe.com`)** runners when `COPILOT_API_TARGET=api.business.githubcopilot.com`; Copilot Business calls receive `Bearer` instead of required `token` prefix. Reproduced on AWF v0.27.13 and v0.27.16. | `copilotTargetRequiresGitHubTokenPrefix()` checked `AWF_PLATFORM_TYPE` guard first. On GHEC, AWF auto-injects `AWF_PLATFORM_TYPE=ghec`, which short-circuited to `false` before querying the `GITHUB_TOKEN_PREFIX_COPILOT_TARGETS` catalog | **Fixed in AWF (PR github/gh-aw-firewall#5872)**: catalog endpoints (`api.enterprise.githubcopilot.com`, `api.business.githubcopilot.com`) are now checked first (always `token`); the platform-type guard now only affects the GHES heuristic for unknown targets. Upgrade to AWF version including #5872. | `awf --version`; inspect api-proxy logs for 400 on `api.business.githubcopilot.com`; confirm `AWF_PLATFORM_TYPE=ghec` is set | github/gh-aw-firewall#5871, github/gh-aw-firewall#5872 |

Error-string quick lookup — new rows

| `EACCES: permission denied, mkdir` on a `/tmp/gh-aw/...` path before containers start (pre-flight) | B8 |
| `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 |
| `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 |

Error-string quick lookup — update existing row

Replace:

| `400 bad request: Authorization header is badly formatted` | C3 |

With:

| `400 bad request: Authorization header is badly formatted` | C3 (general GHES header assembly, AWF ≤ v0.27.1); also C8 if on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` |

Known unresolved items — new entry

- A17 / github/gh-aw-firewall#5985 — ARC/DinD sysroot-stage `build-tools` image cannot be digest-pinned; `--image-tag build-tools=sha256:...` throws "invalid key" (fix pending in open PR github/gh-aw-firewall#5986)

---

Proposed doctor changes

File: .github/workflows/self-hosted-runner-doctor.md

This file imports shared/self-hosted-failure-modes.md, so catalog rows propagate automatically. If the file contains a playbook "§3. Match symptom → failure mode" section, add these recognizer hints:

- `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16
- `EACCES mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8
- `No CA certificates were loaded from the system` on RHEL/Amazon Linux → B9
- `400 Authorization header is badly formatted` on `*.ghe.com` with Copilot Business target → C8
- `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17

---

Proposed portable agent changes

File: .github/agents/self-hosted-runner-doctor.md

The portable agent embeds the full catalog verbatim. Apply all five catalog row changes above to the embedded copy.

Also update playbook §3 ("Match symptom → failure mode") with these additions:

- `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16 (sysroot filter was over-broad and dropped the workspace mount)
- `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent self-hosted runner → B8 (stale root-owned pre-flight dirs)
- `No CA certificates were loaded from the system` in chroot on RHEL/Fedora/Amazon Linux → B9 (missing /etc/pki/ mount)
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
- `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS)

In playbook §4 ("Check for known gaps and notable fixes"), add:

B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including #5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`.

B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including #5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`.

C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including #5872.

A16 / github/gh-aw-firewall#5739 — ARC/DinD sysroot filter over-broad (drops workspace mounts under `_work/`) is **fixed** in AWF version including #5739.

A17 / github/gh-aw-firewall#5985 — `build-tools` image cannot be digest-pinned; fix pending in open PR github/gh-aw-firewall#5986. Security-sensitive deployments must wait for the merge.

Also add to the embedded Known unresolved items:

- A17 / github/gh-aw-firewall#5985 — ARC/DinD sysroot-stage `build-tools` image cannot be digest-pinned; `--image-tag build-tools=sha256:...` throws "invalid key" (fix pending in PR github/gh-aw-firewall#5986)

---

Source issues and PRs

Citation	URL	Lesson
#5739	#5739	New row A16: sysroot filter drops _work/_temp workspace mounts
#5766	#5766	B7 citation update: chown && chmod → chown 2>/dev/null; chmod fallback
#5783	#5783	New row B9: RHEL/Amazon Linux CA bundle copy
#5733	#5733	Source report for B9
#5871	#5871	Source report for C8 (GHEC Copilot Business 400)
#5872	#5872	New row C8: Copilot Business token prefix short-circuit on GHEC
#5983	#5983	New row B8: pre-flight EACCES reclaim on persistent runners
#5985	#5985	New row A17: build-tools cannot be digest-pinned
#5986	#5986	Open fix for A17

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Runner Doctor Updater · 228 AIC · ⊞ 11.4K · ◷

• expires on Aug 6, 2026, 7:45 PM UTC