github · lpcox · Aug 8, 2026 · Aug 8, 2026 · Aug 8, 2026 · Aug 8, 2026
diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md
@@ -89,6 +89,7 @@ Prefer the narrowest match. Examples:
 - `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step → B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated)
 - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690)
 - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939)
+- `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`)
 - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/<path>` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files)
 - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs)
 - `FATAL: http_port: IPv6 is not available` → B3
@@ -98,6 +99,7 @@ Prefer the narrowest match. Examples:
 - `none of the git remotes correspond to the GH_HOST environment variable` → C4
 - `400 bad request: Authorization header is badly formatted` → C3
 - `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog; also check for `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` sentinel suppressing GitHub-token auth path — fixed in github/gh-aw-firewall#6237)
+- `400 bad request: Authorization header is badly formatted` on derived `copilot-api.*.ghe.com` target specifically (not `api.business.githubcopilot.com`) → C9 (derived GHEC Copilot API target missing GitHub `token` prefix; fixed in github/gh-aw-firewall#6991)
 - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`)
 - `diagnosis=unknown` (proxy reachable, no connection error) or `reachable-but-api-error` from DIFC probe with `GITHUB_SERVER_URL=*.ghe.com` → C7 (DIFC proxy not enterprise-host-aware)
 - `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS)
@@ -142,6 +144,8 @@ B18 / github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 — Azure CLI, Azur
 
 B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 — AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the last prominent diagnostic consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause. `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04):** `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939.
 
+B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted because attaching `cli-proxy` to `awf-ext` grants general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so upgrading to #7066 does not resolve this mode.
+
 D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor.
 
 D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime.
@@ -156,6 +160,8 @@ C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residen
 
 C8 / github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens).
 
+C9 / github/gh-aw-firewall#6989, github/gh-aw-firewall#6991 — `400 bad request: Authorization header is badly formatted` on the derived GHEC data-residency Copilot target `copilot-api.<tenant>.ghe.com` is distinct from C8's `api.business.githubcopilot.com` case. `copilotTargetRequiresGitHubTokenPrefix()` did not recognize inferred `copilot-api.*.ghe.com` endpoints as GitHub-hosted GHEC data-residency targets requiring the `token` prefix, so GitHub OAuth/PAT credentials were sent as `Bearer`. **Fixed in AWF (PR github/gh-aw-firewall#6991, merged 2026-08-08):** `isGhecCopilotApiTarget()` sends GitHub OAuth/PAT credentials with `token` for derived `copilot-api.*.ghe.com` targets while preserving `Bearer` for BYOK credentials and the standard Copilot endpoint. Upgrade AWF to include github/gh-aw-firewall#6991.
+
 D7 / github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 — Claude Code (Bun/JSC) crashes with `SIGSEGV`/`SIGABRT` under `--container-runtime gvisor` because JSC JIT is incompatible with gVisor's W^X memory restrictions. **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` as a manual fallback.
 
 ## Output Requirements
@@ -246,6 +252,7 @@ Establish these facts before matching a failure mode:
 | B17 | In `--network-isolation`/topology mode, when a workflow also brings up Tailscale as a later step, every Squid `CONNECT` to an allowlisted API host (e.g. `api.githubcopilot.com`) fails with `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`, DNS never resolved); agent reports `503 Service Unavailable` retries and fails. Same workflow works on host-access mode or in isolation mode without Tailscale. | When Tailscale starts, it can install a policy-routing rule covering `0.0.0.0/0` (exit node / accepted subnet route). Docker bridge traffic on the isolation network follows that route through the Tailscale tunnel. Host-specific DNS servers reachable only via the original network path (Azure DHCP DNS `168.63.129.16`, Tailscale Magic DNS `100.100.100.100`, link-local `169.254.x.x`) become unreachable once Tailscale captures the route, so Squid's DNS queries black-hole and every CONNECT fails at resolution. | **Fixed in AWF (PR github/gh-aw-firewall#6705, merged 2026-07-29):** new `isNonPortableDns(ip)` / `filterForNetworkIsolation(servers, logger)` in `src/dns-resolver.ts` strip non-portable DNS servers (Azure DHCP DNS, Tailscale Magic DNS, link-local) before generating `squid.conf` when `config.networkIsolation` is true; falls back to `8.8.8.8`/`8.8.4.4` if all detected servers are non-portable. Users needing specific DNS in isolation mode can still override via `--dns-servers`. Upgrade to AWF version including github/gh-aw-firewall#6705. | Squid access log shows `CONNECT ... 503 TCP_TUNNEL:HIER_NONE` with server field `-:-` for an allowlisted domain, specifically after a Tailscale-up step in isolation/topology mode; inspect `audit/awf-resolved-config.json` for `networkIsolation: true` + Tailscale in the same run; check whether the DNS servers in use are Azure DHCP (`168.63.129.16`) or Tailscale Magic DNS (`100.100.100.100`) | github/gh-aw-firewall#6704, github/gh-aw-firewall#6705 |
 | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 |
 | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 |
+| B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 
@@ -259,6 +266,7 @@ Establish these facts before matching a failure mode:
 | C6 | Safe-outputs post-processing talks to github.com instead of GHES | gh-aw emitted `GH_HOST` to the wrong channel for later jobs | Fix the compiler / environment propagation in gh-aw | Inspect `$GITHUB_OUTPUT` and `$GITHUB_ENV` for `GH_HOST` | #1460, #1566 |
 | C7 | `awf-cli-proxy` DIFC-proxy liveness probe loops retrying; cli-proxy logs show `diagnosis=unknown` (AWF < v0.27.12) or `diagnosis=reachable-but-api-error (HTTP NNN)` with a `*.ghe.com` hint (AWF ≥ v0.27.12); AWF fails to start | DIFC proxy is reachable but the forwarded `gh api rate_limit` call returns an HTTP error because the DIFC proxy is not enterprise-host-aware on data-residency `*.ghe.com` tenants | **Partially mitigated**: upgrade to AWF ≥ v0.27.12 for a targeted `*.ghe.com` hint and HTTP status in cli-proxy logs; root cause (DIFC proxy enterprise-host awareness) is **unresolved** in companion projects (github/gh-aw-mcpg#8202, github/gh-aw#41911) | Check `GITHUB_SERVER_URL` for `*.ghe.com`; inspect cli-proxy logs for `diagnosis=unknown` or `reachable-but-api-error (HTTP NNN)`; confirm AWF ≥ v0.27.12 for the targeted hint | #5615, #5616 |
 | C8 | `400 bad request: Authorization header is badly formatted` on **GHEC (`*.ghe.com`)** runners when `COPILOT_API_TARGET=api.business.githubcopilot.com`; Copilot Business calls receive `Bearer` instead of required `token` prefix. Reproduced on AWF v0.27.13 and v0.27.16; or `400` persists even after upgrading past github/gh-aw-firewall#5872 when `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is set by `gh-aw` offline mode | Two distinct root causes: **(1) Pre-#5872:** `copilotTargetRequiresGitHubTokenPrefix()` checked `AWF_PLATFORM_TYPE` guard first. On GHEC, AWF auto-injects `AWF_PLATFORM_TYPE=ghec`, which short-circuited to `false` before querying the `GITHUB_TOKEN_PREFIX_COPILOT_TARGETS` catalog. **(2) Post-#5872 / #6237:** `gh-aw` offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel; AWF before #6237 treated it as a real BYOK key, which took precedence over and suppressed the GitHub-token auth path | **Fixed in AWF (PR github/gh-aw-firewall#5872)**: catalog endpoints (`api.enterprise.githubcopilot.com`, `api.business.githubcopilot.com`) are now checked first (always `token`); the platform-type guard now only affects the GHES heuristic for unknown targets. Upgrade to AWF version including github/gh-aw-firewall#5872. **Additional fix (PR github/gh-aw-firewall#6237):** treats `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens), restoring the GitHub-token auth path on Business/Enterprise targets. | `awf --version`; inspect api-proxy logs for 400 on `api.business.githubcopilot.com`; confirm `AWF_PLATFORM_TYPE=ghec` is set; check whether `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is present | github/gh-aw-firewall#5871, github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 |
+| C9 | `400 bad request: Authorization header is badly formatted` specifically on the **derived GHEC data-residency Copilot target** `copilot-api.<tenant>.ghe.com` (distinct from C8's `api.business.githubcopilot.com`); receives `Bearer` instead of required `token` prefix | `copilotTargetRequiresGitHubTokenPrefix()` did not recognize inferred `copilot-api.*.ghe.com` endpoints as GitHub-hosted GHEC data-residency targets requiring the `token` prefix | **Fixed in AWF (PR github/gh-aw-firewall#6991, merged 2026-08-08)**: added `isGhecCopilotApiTarget()` in `containers/api-proxy/providers/copilot-auth.js`; GitHub OAuth/PAT credentials now sent with `token` prefix to derived `copilot-api.*.ghe.com` targets while `Bearer` is preserved for BYOK credentials and the standard Copilot endpoint. Upgrade AWF to include github/gh-aw-firewall#6991. | Inspect api-proxy logs for 400 on `copilot-api.<tenant>.ghe.com`; confirm `GITHUB_SERVER_URL` is `*.ghe.com` and the target is the derived Copilot endpoint (not `api.business.githubcopilot.com`) | github/gh-aw-firewall#6989, github/gh-aw-firewall#6991 |
 
 ## Category D — Alternative runtimes and adjacent gaps