docs: Runner Doctor update — add failure modes A16, A17, B8, B9, C8; update B7, C3 lookup#5992
Conversation
…nd update B7, error lookup
There was a problem hiding this comment.
Pull request overview
Updates the Runner Doctor documentation knowledge base to reflect newly identified/triaged self-hosted runner failure modes and to refine lookup guidance, keeping the operational troubleshooting catalog current.
Changes:
- Added new failure modes A16, A17, B8, B9, and C8 to the shared failure-mode catalog, plus new quick-lookup entries and an A17 “known unresolved” note.
- Updated existing B7 remediation guidance to include the
chown 2>/dev/null; chmodhardening detail. - Extended the Runner Doctor symptom recognizers and “known fixes/gaps” sections to cover the new modes.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/shared/self-hosted-failure-modes.md | Adds five new failure-mode rows, updates B7 guidance, and expands error-string quick lookup + unresolved items. |
| .github/workflows/self-hosted-runner-doctor.md | Adds symptom recognizers and “known fixes/gaps” entries for the newly documented failure modes. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Low
| | B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 | | ||
| | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la <firewall-logs-dir>` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 | | ||
| | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf-<ts>-chroot-home/<path>` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717 | | ||
| | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf-<ts>-chroot-home/<path>` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir.; **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 | |
| A16 / github/gh-aw-firewall#5739 — ARC/DinD sysroot filter over-broad (drops workspace mounts under `_work/`) is **fixed** in AWF version including github/gh-aw-firewall#5739. Filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. | ||
|
|
||
| A17 / github/gh-aw-firewall#5985 — `build-tools` image cannot be digest-pinned; fix pending in open PR github/gh-aw-firewall#5986. Security-sensitive deployments must wait for the merge. | ||
|
|
||
| B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. |
|
@copilot address review feedback |
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Done in commit
|
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
✅ Smoke Claude passed |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
🚀 Security Guard has started processing this pull request |
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓 |
|
✅ Contribution Check completed successfully! Contribution guidelines check completed for PR #5992; no comment needed because the PR appears to follow the applicable CONTRIBUTING.md requirements. |
|
✅ Build Test Suite completed successfully! |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
Smoke Test Results
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Smoke Test: Claude Engine Validation
Overall Result: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
✅ Smoke Test: Copilot BYOK (Direct) Mode
Status: PASS Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy sidecar. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Services Connectivity
Overall: FAIL — Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🔬 Smoke Test: Copilot PAT Auth — PASS
Overall: PASS · Auth mode: PAT (COPILOT_GITHUB_TOKEN) Run: 28908736989 Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Smoke Test: Copilot BYOK (Direct) Mode Results:
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra Overall: FAIL cc Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
📡 OTel Tracing Smoke Test Results
Overall: 3 pass, 2 expected-pending (development in progress)
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Smoke test results:
Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"
- "registry.npmjs.org"See Network Configuration for more information.
|
🔬 Smoke Test Results
Overall: PASS (core connectivity verified) Author: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) Overall: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Chroot Version Comparison Results
Overall: Not all tests passed. Python and Node.js versions differ between host and chroot environments. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Weekly Runner Doctor knowledge-base sync adding 5 new failure mode rows and updating 2 existing entries based on issues/PRs #5739, #5766, #5783, #5871–#5872, #5983, #5985–#5986.
shared/self-hosted-failure-modes.mdNew rows:
filterAgentVolumesForSysroot) over-aggressively dropped workspace mounts under_work/_temp/, causingnode: command not foundeven when binary was correctly installed. Fixed in fix: sysroot filter should not drop workspace custom mounts #5739.--image-tag build-tools=sha256:<digest>throwsError: invalid key 'build-tools'becauseIMAGE_DIGEST_KEYSinsrc/image-tag.tsexcludes'build-tools'. Fix pending in open PR feat: addbuild-toolsto SHA digest pinning infrastructure #5986.EACCES: permission denied, mkdir '/tmp/gh-aw/...'on persistent runners from stale root-owned dirs left by prior sudo runs. Fixed in fix: pre-flight reclaim of stale root-owned directories before writeConfigs #5983 (preflight-reclaim.ts).No CA certificates were loaded from the systemin AWF chroot on RHEL/Fedora/Amazon Linux — chroot only mounted Debian/Ubuntu CA paths. Fixed in fix: detect and expose system CA bundle for RHEL/Amazon Linux in chroot #5783 (multi-distrocopy_system_ca_bundle()).400 Authorization header is badly formattedon GHEC (*.ghe.com) withCOPILOT_API_TARGET=api.business.githubcopilot.com:AWF_PLATFORM_TYPE=ghecshort-circuited the token-prefix catalog before checking Business/Enterprise endpoints. Fixed in fix: Copilot Business 'token' prefix not overridable by AWF_PLATFORM_TYPE #5872.Updated rows:
chown 2>/dev/null; chmodhardening from fix: ensure chmod runs even when chown fails in rootless permission repair #5766 to Fix/flag and Citations.*.ghe.com+Business target; three new lookup rows added for B8 (EACCES mkdir), B9 (CA certs), A17 (invalid key).build-toolsto SHA digest pinning infrastructure #5985 added (fix pending PR feat: addbuild-toolsto SHA digest pinning infrastructure #5986).self-hosted-runner-doctor.md