diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 00c087122..b8ce50e3d 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -90,7 +90,7 @@ Prefer the narrowest match. Examples: - DNS resolution fails in `--network-isolation` mode specifically on GKE/ARC using NodeLocal DNSCache (resolver `169.254.20.10`), even though the resolver is reachable → B17 update (github/gh-aw-firewall#7188 — filtering is now reachability-probed, not category-blanket; upgrade AWF) - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690) - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) -- `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) +- `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (fixed in github/gh-aw-firewall#7338 via credential-free `cli-proxy-egress` relay attached to `awf-ext`; upgrade AWF) - `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) - Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports) - `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; AWF's chroot-side gap is fixed in github/gh-aw-firewall#7245; only the upstream installer/harness mismatch in github/gh-aw-firewall#7130 remains open) @@ -151,7 +151,7 @@ B18 / github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 — Azure CLI, Azur B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 — AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the last prominent diagnostic consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause. `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04):** `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. -B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted because attaching `cli-proxy` to `awf-ext` grants general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so upgrading to #7066 does not resolve this mode. +B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066, github/gh-aw-firewall#7335, github/gh-aw-firewall#7338 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Fixed in AWF (PR github/gh-aw-firewall#7338, merged 2026-08-14):** a credential-free `cli-proxy-egress` relay service is created and attached to `awf-ext` only when the DIFC proxy target is external; the credential-bearing `cli-proxy` stays on `awf-net` and reaches the relay there. No relay is created for attached sibling DIFC proxies already on `awf-net`. Loopback DIFC addresses are normalized to `host.docker.internal`. Upgrade AWF to include github/gh-aw-firewall#7338. B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit ` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds. @@ -266,7 +266,7 @@ Establish these facts before matching a failure mode: | B17 | In `--network-isolation`/topology mode, when a workflow also brings up Tailscale as a later step, every Squid `CONNECT` to an allowlisted API host (e.g. `api.githubcopilot.com`) fails with `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`, DNS never resolved); agent reports `503 Service Unavailable` retries and fails. Same workflow works on host-access mode or in isolation mode without Tailscale. | When Tailscale starts, it can install a policy-routing rule covering `0.0.0.0/0` (exit node / accepted subnet route). Docker bridge traffic on the isolation network follows that route through the Tailscale tunnel. Host-specific DNS servers reachable only via the original network path (Azure DHCP DNS `168.63.129.16`, Tailscale Magic DNS `100.100.100.100`, link-local `169.254.x.x`) become unreachable once Tailscale captures the route, so Squid's DNS queries black-hole and every CONNECT fails at resolution. | **Fixed in AWF (PR github/gh-aw-firewall#6705, merged 2026-07-29):** new `isNonPortableDns(ip)` / `filterForNetworkIsolation(servers, logger)` in `src/dns-resolver.ts` strip non-portable DNS servers (Azure DHCP DNS, Tailscale Magic DNS, link-local) before generating `squid.conf` when `config.networkIsolation` is true; falls back to `8.8.8.8`/`8.8.4.4` if all detected servers are non-portable. Users needing specific DNS in isolation mode can still override via `--dns-servers`. Upgrade to AWF version including github/gh-aw-firewall#6705. **Further refined in AWF (PR github/gh-aw-firewall#7188, merged 2026-08-10):** resolver filtering is now reachability-probed (TCP/53) rather than category-based, so reachable link-local resolvers like GKE/ARC NodeLocal DNSCache (`169.254.20.10`) are retained instead of being blanket-stripped. Upgrade AWF to include github/gh-aw-firewall#7188. | Squid access log shows `CONNECT ... 503 TCP_TUNNEL:HIER_NONE` with server field `-:-` for an allowlisted domain, specifically after a Tailscale-up step in isolation/topology mode; inspect `audit/awf-resolved-config.json` for `networkIsolation: true` + Tailscale in the same run; check whether the DNS servers in use are Azure DHCP (`168.63.129.16`) or Tailscale Magic DNS (`100.100.100.100`) | github/gh-aw-firewall#6704, github/gh-aw-firewall#6705, github/gh-aw-firewall#7185, github/gh-aw-firewall#7188 | | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | -| B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | +| B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Fixed in AWF (PR github/gh-aw-firewall#7338, merged 2026-08-14):** a credential-free `cli-proxy-egress` relay service is created and attached to `awf-ext` only when the DIFC proxy target is external; the credential-bearing `cli-proxy` stays on `awf-net` and reaches the relay there. No relay is created for attached sibling DIFC proxies already on `awf-net`. Loopback DIFC addresses are normalized to `host.docker.internal`. Upgrade AWF to include github/gh-aw-firewall#7338. | Inspect generated compose for a `cli-proxy-egress` service attached to `awf-ext` with no credential env vars and a read-only filesystem; confirm `cli-proxy` itself remains attached only to `awf-net`. | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066, github/gh-aw-firewall#7335, github/gh-aw-firewall#7338 | | B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | | B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example, Postgres on 5432) via `--enable-host-access` / `--allow-host-ports` | Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass. A raw PostgreSQL client cannot use Squid's HTTP `CONNECT` protocol, so allowing the port does not create a direct service route. gh-aw also does not derive service ports when it emits AWF flags. | **Known unresolved:** preserving or compiler-emitting `--enable-host-access` / `--allow-host-ports` is insufficient. A supported strict-topology route for raw service protocols is required in AWF, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. | In strict mode, `getent hosts host.docker.internal` is absent and `psql -h host.docker.internal -p 5432 ...` cannot connect; do not treat emitted host-access flags alone as a successful probe | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 | | B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. | **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. | Confirm gh-aw took the cache-hit path (`GITHUB_PATH` already set). On AWF including github/gh-aw-firewall#7245, check entrypoint logs for `ensure_usr_local_bin_shims` / `prepare_usr_local_bin_overlay`; on older AWF, absence of host `/usr/local/bin/copilot` reproduces the ENOENT | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 | @@ -351,7 +351,7 @@ Establish these facts before matching a failure mode: | DNS resolution fails in `--network-isolation` mode specifically on GKE/ARC using NodeLocal DNSCache (resolver `169.254.20.10`), even though the resolver is reachable | B17 update (github/gh-aw-firewall#7188 — filtering is now reachability-probed, not category-blanket; upgrade AWF) | | Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner | B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | | `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | -| `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | +| `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (fixed in github/gh-aw-firewall#7338 via credential-free `cli-proxy-egress` relay attached to `awf-ext`; upgrade AWF) | | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | | Strict-security workflow cannot reach a GitHub Actions `services:` raw-protocol port (e.g. Postgres 5432) | B22 (known unresolved: strict topology needs an AWF direct service route and gh-aw must derive service ports; emitted host-access flags alone are insufficient) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | diff --git a/.github/workflows/self-hosted-runner-doctor.lock.yml b/.github/workflows/self-hosted-runner-doctor.lock.yml index 86f397185..58473cdee 100644 --- a/.github/workflows/self-hosted-runner-doctor.lock.yml +++ b/.github/workflows/self-hosted-runner-doctor.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"1900b3f409d36f9b3c71e248351bee1bb50575fc27ce0cc652a148888bfabfbe","compiler_version":"v0.86.3","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"984ae878d7680f7a603c188012bfed9cb0edd7e38fa458eef2b533f735e24464","compiler_version":"v0.86.3","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"30aadb1626371455f145991c6385924babda2d04","version":"v0.86.3"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.1","digest":"sha256:5e3f6ee27eeae07195838b97ac4aa2f8aea42a7c55f1c0d3e17d8e88e294ad0d","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.1@sha256:5e3f6ee27eeae07195838b97ac4aa2f8aea42a7c55f1c0d3e17d8e88e294ad0d"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.1","digest":"sha256:288e7d2a12d5b430500d739f9c16e20bb1ed51b91f986f3f3eccde189f489f5c","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.1@sha256:288e7d2a12d5b430500d739f9c16e20bb1ed51b91f986f3f3eccde189f489f5c"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.1","digest":"sha256:9d428af47899bf18ef2d5618075777d76ef344c91e76c1f44ec1aaa0ee347e5f","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.1@sha256:9d428af47899bf18ef2d5618075777d76ef344c91e76c1f44ec1aaa0ee347e5f"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.9","digest":"sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.9@sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b","pinned_image":"ghcr.io/github/gh-aw-node@sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b"},{"image":"ghcr.io/github/github-mcp-server:v1.9.0","digest":"sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e","pinned_image":"ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e"}]} # This file was automatically generated by gh-aw (v0.86.3). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 02672bbcd..ffe8a98cb 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -119,7 +119,7 @@ Prefer the narrowest match. Examples: - DNS resolution fails in `--network-isolation` mode specifically on GKE/ARC using NodeLocal DNSCache (resolver `169.254.20.10`), even though the resolver is reachable → B17 update (github/gh-aw-firewall#7188 — filtering is now reachability-probed, not category-blanket; upgrade AWF) - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) -- `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) +- `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (fixed in github/gh-aw-firewall#7338 via credential-free `cli-proxy-egress` relay attached to `awf-ext`; upgrade AWF) - `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) - Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports) - `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; AWF's chroot-side gap is fixed in github/gh-aw-firewall#7245; only the upstream installer/harness mismatch in github/gh-aw-firewall#7130 remains open) @@ -180,8 +180,6 @@ B18 / github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 — Azure CLI, Azur B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 — AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the last prominent diagnostic consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause. `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04):** `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. -B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted because attaching `cli-proxy` to `awf-ext` grants general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so upgrading to #7066 does not resolve this mode. - B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit ` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds. B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 — Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example Postgres on 5432) through `--enable-host-access` / `--allow-host-ports`. Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass, while raw clients such as `psql` cannot use Squid's HTTP `CONNECT` protocol. **Known unresolved:** AWF needs a supported strict-topology route for raw service protocols, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index f976c312b..a98454bc5 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -61,7 +61,7 @@ Establish these facts before matching a failure mode: | B17 | In `--network-isolation`/topology mode, when a workflow also brings up Tailscale as a later step, every Squid `CONNECT` to an allowlisted API host (e.g. `api.githubcopilot.com`) fails with `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`, DNS never resolved); agent reports `503 Service Unavailable` retries and fails. Same workflow works on host-access mode or in isolation mode without Tailscale. | When Tailscale starts, it can install a policy-routing rule covering `0.0.0.0/0` (exit node / accepted subnet route). Docker bridge traffic on the isolation network follows that route through the Tailscale tunnel. Host-specific DNS servers reachable only via the original network path (Azure DHCP DNS `168.63.129.16`, Tailscale Magic DNS `100.100.100.100`, link-local `169.254.x.x`) become unreachable once Tailscale captures the route, so Squid's DNS queries black-hole and every CONNECT fails at resolution. | **Fixed in AWF (PR github/gh-aw-firewall#6705, merged 2026-07-29):** new `isNonPortableDns(ip)` / `filterForNetworkIsolation(servers, logger)` in `src/dns-resolver.ts` strip non-portable DNS servers (Azure DHCP DNS, Tailscale Magic DNS, link-local) before generating `squid.conf` when `config.networkIsolation` is true; falls back to `8.8.8.8`/`8.8.4.4` if all detected servers are non-portable. Users needing specific DNS in isolation mode can still override via `--dns-servers`. Upgrade to AWF version including github/gh-aw-firewall#6705. **Further refined in AWF (PR github/gh-aw-firewall#7188, merged 2026-08-10):** resolver filtering is now reachability-probed (TCP/53) rather than category-based, so reachable link-local resolvers like GKE/ARC NodeLocal DNSCache (`169.254.20.10`) are retained instead of being blanket-stripped. Upgrade AWF to include github/gh-aw-firewall#7188. | Squid access log shows `CONNECT ... 503 TCP_TUNNEL:HIER_NONE` with server field `-:-` for an allowlisted domain, specifically after a Tailscale-up step in isolation/topology mode; inspect `audit/awf-resolved-config.json` for `networkIsolation: true` + Tailscale in the same run; check whether the DNS servers in use are Azure DHCP (`168.63.129.16`) or Tailscale Magic DNS (`100.100.100.100`) | github/gh-aw-firewall#6704, github/gh-aw-firewall#6705, github/gh-aw-firewall#7185, github/gh-aw-firewall#7188 | | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | -| B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | +| B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Fixed in AWF (PR github/gh-aw-firewall#7338, merged 2026-08-14):** a credential-free `cli-proxy-egress` relay service is created and attached to `awf-ext` only when the DIFC proxy target is external; the credential-bearing `cli-proxy` stays on `awf-net` and reaches the relay there. No relay is created for attached sibling DIFC proxies already on `awf-net`. Loopback DIFC addresses are normalized to `host.docker.internal`. Upgrade AWF to include github/gh-aw-firewall#7338. | Inspect generated compose for a `cli-proxy-egress` service attached to `awf-ext` with no credential env vars and a read-only filesystem; confirm `cli-proxy` itself remains attached only to `awf-net`. | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066, github/gh-aw-firewall#7335, github/gh-aw-firewall#7338 | | B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | | B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example, Postgres on 5432) via `--enable-host-access` / `--allow-host-ports` | Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass. A raw PostgreSQL client cannot use Squid's HTTP `CONNECT` protocol, so allowing the port does not create a direct service route. gh-aw also does not derive service ports when it emits AWF flags. | **Known unresolved:** preserving or compiler-emitting `--enable-host-access` / `--allow-host-ports` is insufficient. A supported strict-topology route for raw service protocols is required in AWF, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. | In strict mode, `getent hosts host.docker.internal` is absent and `psql -h host.docker.internal -p 5432 ...` cannot connect; do not treat emitted host-access flags alone as a successful probe | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 | | B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. | **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. | Confirm gh-aw took the cache-hit path (`GITHUB_PATH` already set). On AWF including github/gh-aw-firewall#7245, check entrypoint logs for `ensure_usr_local_bin_shims` / `prepare_usr_local_bin_overlay`; on older AWF, absence of host `/usr/local/bin/copilot` reproduces the ENOENT | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 | @@ -147,7 +147,7 @@ Establish these facts before matching a failure mode: | DNS resolution fails in `--network-isolation` mode specifically on GKE/ARC using NodeLocal DNSCache (resolver `169.254.20.10`), even though the resolver is reachable | B17 update (github/gh-aw-firewall#7188 — filtering is now reachability-probed, not category-blanket; upgrade AWF) | | Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner | B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | | `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | -| `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | +| `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (fixed in github/gh-aw-firewall#7338 via credential-free `cli-proxy-egress` relay attached to `awf-ext`; upgrade AWF) | | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | | Strict-security workflow cannot reach a GitHub Actions `services:` raw-protocol port (e.g. Postgres 5432) | B22 (known unresolved: strict topology needs an AWF direct service route and gh-aw must derive service ports; emitted host-access flags alone are insufficient) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) |