From 72fbe9ca484eadcebe01157074628fd45891451d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:32:15 +0000 Subject: [PATCH 1/4] Initial plan From 1faff0b29cf30db1e8be9aac515cd335639ead53 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:37:04 +0000 Subject: [PATCH 2/4] Add A19 (ARC/DinD safeoutputs mount) and B19 (rootless cleanup chmod noise) doctor entries Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com> --- .github/agents/self-hosted-runner-doctor.md | 10 ++++++++++ .github/workflows/self-hosted-runner-doctor.md | 6 ++++++ .github/workflows/shared/self-hosted-failure-modes.md | 4 ++++ 3 files changed, 20 insertions(+) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 95f0381c2..51ff296bd 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -88,6 +88,7 @@ Prefer the narrowest match. Examples: - AWF rejects `--mount` with "host path must be absolute" and the path visibly contains `${VAR_NAME}` unexpanded → B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; fixed in github/gh-aw-firewall#6655) - `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step → B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated) - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690) +- `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -101,6 +102,7 @@ Prefer the narrowest match. Examples: - `diagnosis=unknown` (proxy reachable, no connection error) or `reachable-but-api-error` from DIFC probe with `GITHUB_SERVER_URL=*.ghe.com` → C7 (DIFC proxy not enterprise-host-aware) - `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS) - `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` → A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658) +- `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) ### 4. Check for known gaps and notable fixes @@ -114,6 +116,8 @@ A17 / github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 — `build-tools` d A18 / github/gh-aw-firewall#6684, github/gh-aw#48658 — Under `runner.topology: arc-dind`, XDG-respecting tools (Flutter, etc.) fail with `EACCES` writing directly under `/home/runner` (for example `/home/runner/tool_state`) because `XDG_CONFIG_HOME` is set to the stale, root-owned `/home/runner` before `HOME` is updated to the writable `${RUNNER_TEMP}/gh-aw/home` path. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` overrides there are silently overwritten. **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). +A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 — On ARC/DinD, `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle`. The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`), so in DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem rather than the runner's staged path, and writes made inside the container land somewhere the runner-side ingestion step never sees. **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05):** the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. + B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`. @@ -136,6 +140,8 @@ B17 / github/gh-aw-firewall#6704, github/gh-aw-firewall#6705 — In `--network-i B18 / github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 — Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox because `.azure` was in `home.forbiddenSubdirs` (not `home.toolSubdirs`) in the canonical sandbox mount policy, and `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` were absent from the always-forwarded env var list. **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. Upgrade AWF to version including github/gh-aw-firewall#6690. +B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 — AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the last prominent diagnostic consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause. `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04):** `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. @@ -215,6 +221,7 @@ Establish these facts before matching a failure mode: | A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 | | A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — on a patched version the command succeeds and the sysroot-stage image ref includes the pinned digest (visible in generated compose YAML); on older AWF without the fix the command throws `Error: invalid key 'build-tools'` | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 | | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 | +| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated compose bind mount for `/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | ## Category B — Self-hosted runners @@ -238,6 +245,7 @@ Establish these facts before matching a failure mode: | B16 | AWF rejects a `--mount` entry with a non-absolute host path error; the mount spec was defined as `${TERRAFORM_CLI_PATH}/terraform:...` (or similar env-var pattern) but the literal string `${TERRAFORM_CLI_PATH}` reaches AWF's volume validator | The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` references in single quotes in the generated shell invocation. Bash single quotes prevent variable substitution, so the literal variable reference string (e.g. `${TERRAFORM_CLI_PATH}`) reaches AWF's volume validation instead of the resolved absolute path. | **Fixed in AWF (PR github/gh-aw-firewall#6655, merged 2026-07-27):** `expandEnvVarsInMount()` in `src/parsers/volume-parsers.ts` now expands `${VAR_NAME}` and `$VAR_NAME` patterns from `process.env` before path validation. If the variable is undefined, AWF emits a precise error (`Environment variable is not set: ${VAR_NAME}`) rather than a misleading path-absoluteness failure. Upgrade AWF to the version including github/gh-aw-firewall#6655. | Inspect the generated `awf` invocation in the compiled lock file — if mount specs appear in single quotes containing `${...}`, the bug is present. Running `awf ... --mount '${VAR}/path:/dest'` on AWF versions before github/gh-aw-firewall#6655 reproduces the rejection; on patched AWF the mount spec is accepted after variable expansion. | github/gh-aw-firewall#6649, github/gh-aw-firewall#6655 | | B17 | In `--network-isolation`/topology mode, when a workflow also brings up Tailscale as a later step, every Squid `CONNECT` to an allowlisted API host (e.g. `api.githubcopilot.com`) fails with `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`, DNS never resolved); agent reports `503 Service Unavailable` retries and fails. Same workflow works on host-access mode or in isolation mode without Tailscale. | When Tailscale starts, it can install a policy-routing rule covering `0.0.0.0/0` (exit node / accepted subnet route). Docker bridge traffic on the isolation network follows that route through the Tailscale tunnel. Host-specific DNS servers reachable only via the original network path (Azure DHCP DNS `168.63.129.16`, Tailscale Magic DNS `100.100.100.100`, link-local `169.254.x.x`) become unreachable once Tailscale captures the route, so Squid's DNS queries black-hole and every CONNECT fails at resolution. | **Fixed in AWF (PR github/gh-aw-firewall#6705, merged 2026-07-29):** new `isNonPortableDns(ip)` / `filterForNetworkIsolation(servers, logger)` in `src/dns-resolver.ts` strip non-portable DNS servers (Azure DHCP DNS, Tailscale Magic DNS, link-local) before generating `squid.conf` when `config.networkIsolation` is true; falls back to `8.8.8.8`/`8.8.4.4` if all detected servers are non-portable. Users needing specific DNS in isolation mode can still override via `--dns-servers`. Upgrade to AWF version including github/gh-aw-firewall#6705. | Squid access log shows `CONNECT ... 503 TCP_TUNNEL:HIER_NONE` with server field `-:-` for an allowlisted domain, specifically after a Tailscale-up step in isolation/topology mode; inspect `audit/awf-resolved-config.json` for `networkIsolation: true` + Tailscale in the same run; check whether the DNS servers in use are Azure DHCP (`168.63.129.16`) or Tailscale Magic DNS (`100.100.100.100`) | github/gh-aw-firewall#6704, github/gh-aw-firewall#6705 | | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | +| B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | ## Category C — GHES / GHEC / `ghe.com` @@ -290,6 +298,7 @@ Establish these facts before matching a failure mode: | `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 | | `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 | | `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` | A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658) | +| `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write | A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) | | `ENOENT ... /host/usr/local/bin/copilot` | A8 | | `mkdirat ... : read-only file system` during chroot agent startup | A12 | | `getaddrinfo EAI_AGAIN ` with `awf-cli-proxy could not connect to the external DIFC proxy`, but DinD `nslookup` succeeds once the peer is attached | B5 | @@ -315,6 +324,7 @@ Establish these facts before matching a failure mode: | AWF rejects `--mount` with "host path must be absolute" (or similar) and the path visibly contains `${VAR_NAME}` unexpanded | B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; fixed in github/gh-aw-firewall#6655) | | `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step | B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated) | | Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner | B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | +| `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | ## Known unresolved items diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 44fd5a63f..848de66aa 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -117,6 +117,7 @@ Prefer the narrowest match. Examples: - AWF rejects `--mount` with "host path must be absolute" and the path visibly contains `${VAR_NAME}` unexpanded → B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; fixed in github/gh-aw-firewall#6655) - `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step → B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated) - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) +- `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -129,6 +130,7 @@ Prefer the narrowest match. Examples: - `diagnosis=unknown` (proxy reachable, no connection error) or `reachable-but-api-error` from DIFC probe with `GITHUB_SERVER_URL=*.ghe.com` → C7 (DIFC proxy not enterprise-host-aware) - `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS) - `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` → A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658) +- `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) ### 4. Check for known gaps and notable fixes @@ -143,6 +145,8 @@ A17 / github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 — `build-tools` d A18 / github/gh-aw-firewall#6684, github/gh-aw#48658 — Under `runner.topology: arc-dind`, XDG-respecting tools (Flutter, etc.) fail with `EACCES` writing directly under `/home/runner` (for example `/home/runner/tool_state`) because `XDG_CONFIG_HOME` is set to the stale, root-owned `/home/runner` before `HOME` is updated to the writable `${RUNNER_TEMP}/gh-aw/home` path. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` overrides there are silently overwritten. **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). +A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 — On ARC/DinD, `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle`. The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`), so in DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem rather than the runner's staged path, and writes made inside the container land somewhere the runner-side ingestion step never sees. **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05):** the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. + B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`. @@ -165,6 +169,8 @@ B17 / github/gh-aw-firewall#6704, github/gh-aw-firewall#6705 — In `--network-i B18 / github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 — Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox because `.azure` was in `home.forbiddenSubdirs` (not `home.toolSubdirs`) in the canonical sandbox mount policy, and `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` were absent from the always-forwarded env var list. **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. +B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 — AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the last prominent diagnostic consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause. `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04):** `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 9af16bd96..f84f207c4 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -35,6 +35,7 @@ Establish these facts before matching a failure mode: | A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 | | A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — on a patched version the command succeeds and the sysroot-stage image ref includes the pinned digest (visible in generated compose YAML); on older AWF without the fix the command throws `Error: invalid key 'build-tools'` | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 | | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 | +| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated compose bind mount for `/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | ## Category B — Self-hosted runners @@ -58,6 +59,7 @@ Establish these facts before matching a failure mode: | B16 | AWF rejects a `--mount` entry with a non-absolute host path error; the mount spec was defined as `${TERRAFORM_CLI_PATH}/terraform:...` (or similar env-var pattern) but the literal string `${TERRAFORM_CLI_PATH}` reaches AWF's volume validator | The gh-aw compiler wraps `sandbox.agent.mounts` specs containing `${}` references in single quotes in the generated shell invocation. Bash single quotes prevent variable substitution, so the literal variable reference string (e.g. `${TERRAFORM_CLI_PATH}`) reaches AWF's volume validation instead of the resolved absolute path. | **Fixed in AWF (PR github/gh-aw-firewall#6655, merged 2026-07-27):** `expandEnvVarsInMount()` in `src/parsers/volume-parsers.ts` now expands `${VAR_NAME}` and `$VAR_NAME` patterns from `process.env` before path validation. If the variable is undefined, AWF emits a precise error (`Environment variable is not set: ${VAR_NAME}`) rather than a misleading path-absoluteness failure. Upgrade AWF to the version including github/gh-aw-firewall#6655. | Inspect the generated `awf` invocation in the compiled lock file — if mount specs appear in single quotes containing `${...}`, the bug is present. Running `awf ... --mount '${VAR}/path:/dest'` on AWF versions before github/gh-aw-firewall#6655 reproduces the rejection; on patched AWF the mount spec is accepted after variable expansion. | github/gh-aw-firewall#6649, github/gh-aw-firewall#6655 | | B17 | In `--network-isolation`/topology mode, when a workflow also brings up Tailscale as a later step, every Squid `CONNECT` to an allowlisted API host (e.g. `api.githubcopilot.com`) fails with `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`, DNS never resolved); agent reports `503 Service Unavailable` retries and fails. Same workflow works on host-access mode or in isolation mode without Tailscale. | When Tailscale starts, it can install a policy-routing rule covering `0.0.0.0/0` (exit node / accepted subnet route). Docker bridge traffic on the isolation network follows that route through the Tailscale tunnel. Host-specific DNS servers reachable only via the original network path (Azure DHCP DNS `168.63.129.16`, Tailscale Magic DNS `100.100.100.100`, link-local `169.254.x.x`) become unreachable once Tailscale captures the route, so Squid's DNS queries black-hole and every CONNECT fails at resolution. | **Fixed in AWF (PR github/gh-aw-firewall#6705, merged 2026-07-29):** new `isNonPortableDns(ip)` / `filterForNetworkIsolation(servers, logger)` in `src/dns-resolver.ts` strip non-portable DNS servers (Azure DHCP DNS, Tailscale Magic DNS, link-local) before generating `squid.conf` when `config.networkIsolation` is true; falls back to `8.8.8.8`/`8.8.4.4` if all detected servers are non-portable. Users needing specific DNS in isolation mode can still override via `--dns-servers`. Upgrade to AWF version including github/gh-aw-firewall#6705. | Squid access log shows `CONNECT ... 503 TCP_TUNNEL:HIER_NONE` with server field `-:-` for an allowlisted domain, specifically after a Tailscale-up step in isolation/topology mode; inspect `audit/awf-resolved-config.json` for `networkIsolation: true` + Tailscale in the same run; check whether the DNS servers in use are Azure DHCP (`168.63.129.16`) or Tailscale Magic DNS (`100.100.100.100`) | github/gh-aw-firewall#6704, github/gh-aw-firewall#6705 | | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | +| B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | ## Category C — GHES / GHEC / `ghe.com` @@ -110,6 +112,7 @@ Establish these facts before matching a failure mode: | `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 | | `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 | | `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` | A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658) | +| `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write | A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) | | `ENOENT ... /host/usr/local/bin/copilot` | A8 | | `mkdirat ... : read-only file system` during chroot agent startup | A12 | | `getaddrinfo EAI_AGAIN ` with `awf-cli-proxy could not connect to the external DIFC proxy`, but DinD `nslookup` succeeds once the peer is attached | B5 | @@ -135,6 +138,7 @@ Establish these facts before matching a failure mode: | AWF rejects `--mount` with "host path must be absolute" (or similar) and the path visibly contains `${VAR_NAME}` unexpanded | B16 (single-quote wrapping by compiler prevents shell expansion of `${}` in mount specs; fixed in github/gh-aw-firewall#6655) | | `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step | B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated) | | Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner | B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | +| `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | ## Known unresolved items From 5e2546c0a3445399a2dd5df8b5ed33fb729e0036 Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Fri, 7 Aug 2026 17:04:12 -0700 Subject: [PATCH 3/4] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/agents/self-hosted-runner-doctor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 51ff296bd..c2fb7b036 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -221,7 +221,7 @@ Establish these facts before matching a failure mode: | A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 | | A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — on a patched version the command succeeds and the sysroot-stage image ref includes the pinned digest (visible in generated compose YAML); on older AWF without the fix the command throws `Error: invalid key 'build-tools'` | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 | | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 | -| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated compose bind mount for `/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | +| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | ## Category B — Self-hosted runners From 5fd0b49322e7fc6f3f9e41ef5e9d566d2174244f Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Fri, 7 Aug 2026 17:04:26 -0700 Subject: [PATCH 4/4] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/workflows/shared/self-hosted-failure-modes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index f84f207c4..2ef6ef31e 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -35,7 +35,7 @@ Establish these facts before matching a failure mode: | A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 | | A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — on a patched version the command succeeds and the sysroot-stage image ref includes the pinned digest (visible in generated compose YAML); on older AWF without the fix the command throws `Error: invalid key 'build-tools'` | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 | | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 | -| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated compose bind mount for `/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | +| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | ## Category B — Self-hosted runners