🩺 Runner Doctor UpdateRunner Doctor Update: A13 (empty /host staging), B5 (topology-attach deadlock), B6 (rootless artifact EACCES) + portable-agent A [Content truncated due to length]

[github-actions]

Summary

• Scan window: 2026-06-19 → 2026-06-26
• Items reviewed: [ARC-DinD] Chroot /host base userland not staged on split-fs runners — security-preserving fix + empty-/host test harness #5541, network-isolation (sudo:false) rollout regressions on standard runners: topology-attach ordering deadlock + rootless log EACCES #5542, network-isolation: topology-attach ordering deadlock starves the cli-proxy health gate (fixable in AWF) #5543, network-isolation: rootless firewall logs are unreadable by the runner → EACCES on artifact upload (fixable in AWF) #5545 (plus non-relevant refactoring and CI issues screened and skipped)
• New lessons: 3 genuine new failure modes (A13, B5, B6); 1 pre-existing sync gap (A12 missing from portable agent)
• Existing open runner-doctor proposals: none (search confirmed zero)

---

Proposed knowledge-base changes

File: .github/workflows/shared/self-hosted-failure-modes.md

1 — Add A13 (new row in Category A — ARC / DinD)

Insert after the A12 row:

| A13 | `chroot: failed to run command '/bin/sh': No such file or directory` or `[entrypoint][ERROR] capsh not found on host system` on a **glibc/Debian daemon** (not musl/Alpine) | ARC/DinD split-fs: system-mount source dirs (`/tmp/gh-aw/{usr,bin,lib,...}`) are empty because nothing populates them; `stageBaseSystem()` does not yet exist. The entrypoint "musl/Alpine" warning is **misleading** — it fires because no dynamic loader is found, not because the daemon is musl. `dind.preStageDirs` only mkdir's empty work dirs; it does not stage a base userland. | **Unresolved** — `stageBaseSystem()` capability not yet implemented; base userland must originate from the AWF-signed image to preserve security invariants (never from runner/daemon-writable paths for pre-`capsh` execution). Workaround: bake required binaries (`/bin/sh`, `bash`, `capsh`, loader, coreutils) directly into the DinD daemon image | Confirm daemon libc: `ldd --version` inside the DinD container. Then check whether the staging dir is populated: `ls /tmp/gh-aw/usr/bin/sh` (or `/tmp/gh-aw/bin/sh`) on the runner side. If the daemon is glibc but the file is missing, this is A13, not A4. | #5541 |

2 — Add B5 (new row in Category B — Self-hosted runners)

Append after the B4 row:

| B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 |

3 — Add B6 (new row in Category B — Self-hosted runners)

Append after the B5 row:

| B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la <firewall-logs-dir>` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 |

4 — Add error-string quick-lookup entries

Add the following rows to the error-string table (after the existing mkdirat row):

| `getaddrinfo EAI_AGAIN <topology-peer>` with `awf-cli-proxy could not connect to the external DIFC proxy` | B5 |
| `EACCES` in `upload-artifact` after `sudo: false` (`--network-isolation`) AWF run | B6 |
| `chroot: failed to run command '/bin/sh'` on glibc daemon (not musl — confirmed by `ldd --version`) | A13 |

5 — Add A13 to "Known unresolved items"

- A13 / #5541 — base-userland staging for ARC/DinD split-fs (`stageBaseSystem()` not yet implemented; security-preserving fix requires sourcing the base userland from the AWF-signed image)

---

Proposed doctor changes

File: .github/workflows/self-hosted-runner-doctor.md

This file imports shared/self-hosted-failure-modes.md and contains no duplicated catalog rows; it only needs playbook-level additions.

In §3 "Match symptom → failure mode", add to the hint list:

- `chroot: failed to run command '/bin/sh'` on a glibc daemon → A13 (empty staging, not A4 musl)
- `EAI_AGAIN <awmg-cli-proxy>` in network-isolation + topology-attach → B5
- `EACCES` in upload-artifact after sudo:false → B6

In §4 "Check for known unresolved problems", add:

A13 / #5541 — ARC/DinD split-fs base-userland staging is not yet implemented; AWF cannot currently run end-to-end on a split-fs runner with an empty /host.

---

Proposed portable agent changes

File: .github/agents/self-hosted-runner-doctor.md

This file embeds the full catalog and must be updated in parallel with the shared file. It is currently also missing A12 (the mkdirat / read-only filesystem fix from #5481/#5482 that was added to the shared file but never synced here).

1 — Add missing A12 (sync gap)

After the A11 row, insert:

| A12 | `mkdirat ... : read-only file system` during agent chroot startup on ARC/DinD | `chroot.binariesSourcePath` set to the same root as `--docker-host-path-prefix` (e.g. both `/tmp/gh-aw`); Docker mounts `/tmp/gh-aw/usr:/host/usr:ro` first, then the attempt to mkdir `/host/usr/local/bin` as a nested overlay mount point fails because the parent is read-only | **Fixed in firewall v0.27.10**: upgrade AWF; the overlay is now mounted at `/host/tmp/awf-runner-bin:ro` (writable `/host/tmp` parent) instead of `/host/usr/local/bin:ro` | Check `awf --version`; inspect agent container logs for `mkdirat`; verify `chroot.binariesSourcePath` equals `docker-host-path-prefix` root | #5481, #5482 |

2 — Add A13 (same text as shared file change above)

Insert after A12.

3 — Add B5 and B6 (same text as shared file changes above)

Append after B4.

4 — Add error-string quick-lookup entries

Add the mkdirat row that is in the shared file but missing from the portable agent, plus the three new rows:

| `mkdirat ... : read-only file system` during chroot agent startup | A12 |
| `getaddrinfo EAI_AGAIN <topology-peer>` with `awf-cli-proxy could not connect to the external DIFC proxy` | B5 |
| `EACCES` in `upload-artifact` after `sudo: false` (`--network-isolation`) AWF run | B6 |
| `chroot: failed to run command '/bin/sh'` on glibc daemon (not musl — confirmed by `ldd --version`) | A13 |

5 — Add A13 to "Known unresolved items"

- A13 / #5541 — base-userland staging for ARC/DinD split-fs (`stageBaseSystem()` not yet implemented; security-preserving fix requires sourcing the base userland from the AWF-signed image)

6 — Update §3 playbook hint (portable agent has same text as the workflow doctor)

Add the same three hint lines as the workflow doctor (A13, B5, B6).

7 — Update §4 "Check for known unresolved problems"

Add the A13 entry as in the workflow doctor.

---

Source issues and PRs

Citation	Title	State	Lesson
#5541	[ARC-DinD] Chroot /host base userland not staged on split-fs runners	Open	A13 — empty /host staging, unresolved
#5543	network-isolation: topology-attach ordering deadlock starves the cli-proxy health gate	Closed (completed)	B5 — EAI_AGAIN deadlock
#5545	network-isolation: rootless firewall logs are unreadable → EACCES on artifact upload	Closed (completed)	B6 — rootless EACCES
#5542	network-isolation (sudo:false) rollout regressions (parent tracking issue)	Closed (completed)	Covers B5 + B6

Generated by Runner Doctor Updater · 55.5 AIC · ⊞ 9.7K · ◷

• expires on Jul 26, 2026, 5:08 PM UTC