Update Runner Doctor with C9 and B20 failure modes - #7124
Merged
Conversation
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update Runner Doctor knowledge base for failure modes C9 and B20
Update Runner Doctor with C9 and B20 failure modes
Aug 8, 2026
lpcox
marked this pull request as ready for review
August 8, 2026 22:24
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Runner Doctor guidance for C9 Copilot authentication failures and B20 cli-proxy network failures.
Changes:
- Adds C9 and B20 catalog entries and lookups.
- Expands workflow diagnostic guidance.
- Updates generated workflow metadata.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Adds C9 and B20 catalog entries. |
.github/workflows/self-hosted-runner-doctor.md |
Adds diagnostic and remediation guidance. |
.github/workflows/self-hosted-runner-doctor.lock.yml |
Updates the generated body hash. |
Review details
Tip
Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Suppressed comments (2)
.github/workflows/shared/self-hosted-failure-modes.md:145
- This lookup incorrectly says #7066 fixed B20. The merged regression test in
src/compose-generator.test.ts:288-302explicitly expectscli-proxyto remain offawf-ext, so theENETUNREACHcondition remains unresolved for an external DIFC endpoint. Update this status rather than directing users to upgrade to #7066.
| `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (cli-proxy missing external bridge dual-home; fixed in github/gh-aw-firewall#7066) |
.github/workflows/self-hosted-runner-doctor.md:176
- This detailed remediation describes code that was reverted before #7066 merged. The final PR commit removed the
cli-proxy/awf-extattachment after review identified that it bypassed the isolation boundary, and the merged test requiresawf-extto be absent. Rewrite B20 as unresolved and document why direct dual-homing is not a safe fix.
B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Fixed in AWF (PR github/gh-aw-firewall#7066, merged 2026-08-08):** `src/compose-generator.ts` attaches `cli-proxy` to `EXTERNAL_BRIDGE_NAME` (`awf-ext`) as well as `awf-net` whenever `networkIsolation` is enabled and the service exists. Upgrade AWF to include github/gh-aw-firewall#7066.
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Balanced
| | B17 | In `--network-isolation`/topology mode, when a workflow also brings up Tailscale as a later step, every Squid `CONNECT` to an allowlisted API host (e.g. `api.githubcopilot.com`) fails with `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`, DNS never resolved); agent reports `503 Service Unavailable` retries and fails. Same workflow works on host-access mode or in isolation mode without Tailscale. | When Tailscale starts, it can install a policy-routing rule covering `0.0.0.0/0` (exit node / accepted subnet route). Docker bridge traffic on the isolation network follows that route through the Tailscale tunnel. Host-specific DNS servers reachable only via the original network path (Azure DHCP DNS `168.63.129.16`, Tailscale Magic DNS `100.100.100.100`, link-local `169.254.x.x`) become unreachable once Tailscale captures the route, so Squid's DNS queries black-hole and every CONNECT fails at resolution. | **Fixed in AWF (PR github/gh-aw-firewall#6705, merged 2026-07-29):** new `isNonPortableDns(ip)` / `filterForNetworkIsolation(servers, logger)` in `src/dns-resolver.ts` strip non-portable DNS servers (Azure DHCP DNS, Tailscale Magic DNS, link-local) before generating `squid.conf` when `config.networkIsolation` is true; falls back to `8.8.8.8`/`8.8.4.4` if all detected servers are non-portable. Users needing specific DNS in isolation mode can still override via `--dns-servers`. Upgrade to AWF version including github/gh-aw-firewall#6705. | Squid access log shows `CONNECT ... 503 TCP_TUNNEL:HIER_NONE` with server field `-:-` for an allowlisted domain, specifically after a Tailscale-up step in isolation/topology mode; inspect `audit/awf-resolved-config.json` for `networkIsolation: true` + Tailscale in the same run; check whether the DNS servers in use are Azure DHCP (`168.63.129.16`) or Tailscale Magic DNS (`100.100.100.100`) | github/gh-aw-firewall#6704, github/gh-aw-firewall#6705 | | ||
| | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | ||
| | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | | ||
| | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Fixed in AWF (PR github/gh-aw-firewall#7066, merged 2026-08-08)**: `src/compose-generator.ts` now attaches `cli-proxy` to `EXTERNAL_BRIDGE_NAME` (`awf-ext`) in addition to `awf-net` whenever `networkIsolation` is enabled and the `cli-proxy` service exists, matching the existing Squid/api-proxy dual-homing pattern. Upgrade AWF to include github/gh-aw-firewall#7066. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — should list both `awf-net` and `awf-ext` on patched AWF; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | |
| - `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step → B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated) | ||
| - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | ||
| - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | ||
| - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (cli-proxy missing external bridge dual-home; fixed in github/gh-aw-firewall#7066) |
| | C6 | Safe-outputs post-processing talks to github.com instead of GHES | gh-aw emitted `GH_HOST` to the wrong channel for later jobs | Fix the compiler / environment propagation in gh-aw | Inspect `$GITHUB_OUTPUT` and `$GITHUB_ENV` for `GH_HOST` | #1460, #1566 | | ||
| | C7 | `awf-cli-proxy` DIFC-proxy liveness probe loops retrying; cli-proxy logs show `diagnosis=unknown` (AWF < v0.27.12) or `diagnosis=reachable-but-api-error (HTTP NNN)` with a `*.ghe.com` hint (AWF ≥ v0.27.12); AWF fails to start | DIFC proxy is reachable but the forwarded `gh api rate_limit` call returns an HTTP error because the DIFC proxy is not enterprise-host-aware on data-residency `*.ghe.com` tenants | **Partially mitigated**: upgrade to AWF ≥ v0.27.12 for a targeted `*.ghe.com` hint and HTTP status in cli-proxy logs; root cause (DIFC proxy enterprise-host awareness) is **unresolved** in companion projects (github/gh-aw-mcpg#8202, github/gh-aw#41911) | Check `GITHUB_SERVER_URL` for `*.ghe.com`; inspect cli-proxy logs for `diagnosis=unknown` or `reachable-but-api-error (HTTP NNN)`; confirm AWF ≥ v0.27.12 for the targeted hint | #5615, #5616 | | ||
| | C8 | `400 bad request: Authorization header is badly formatted` on **GHEC (`*.ghe.com`)** runners when `COPILOT_API_TARGET=api.business.githubcopilot.com`; Copilot Business calls receive `Bearer` instead of required `token` prefix. Reproduced on AWF v0.27.13 and v0.27.16; or `400` persists even after upgrading past github/gh-aw-firewall#5872 when `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is set by `gh-aw` offline mode | Two distinct root causes: **(1) Pre-#5872:** `copilotTargetRequiresGitHubTokenPrefix()` checked `AWF_PLATFORM_TYPE` guard first. On GHEC, AWF auto-injects `AWF_PLATFORM_TYPE=ghec`, which short-circuited to `false` before querying the `GITHUB_TOKEN_PREFIX_COPILOT_TARGETS` catalog. **(2) Post-#5872 / #6237:** `gh-aw` offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel; AWF before #6237 treated it as a real BYOK key, which took precedence over and suppressed the GitHub-token auth path | **Fixed in AWF (PR github/gh-aw-firewall#5872)**: catalog endpoints (`api.enterprise.githubcopilot.com`, `api.business.githubcopilot.com`) are now checked first (always `token`); the platform-type guard now only affects the GHES heuristic for unknown targets. Upgrade to AWF version including github/gh-aw-firewall#5872. **Additional fix (PR github/gh-aw-firewall#6237):** treats `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens), restoring the GitHub-token auth path on Business/Enterprise targets. | `awf --version`; inspect api-proxy logs for 400 on `api.business.githubcopilot.com`; confirm `AWF_PLATFORM_TYPE=ghec` is set; check whether `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is present | github/gh-aw-firewall#5871, github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 | | ||
| | C9 | `400 bad request: Authorization header is badly formatted` specifically on the **derived GHEC data-residency Copilot target** `copilot-api.<tenant>.ghe.com` (distinct from C8's `api.business.githubcopilot.com`); receives `Bearer` instead of required `token` prefix | `copilotTargetRequiresGitHubTokenPrefix()` did not recognize inferred `copilot-api.*.ghe.com` endpoints as GitHub-hosted GHEC data-residency targets requiring the `token` prefix | **Fixed in AWF (PR github/gh-aw-firewall#6991, merged 2026-08-08)**: added `isGhecCopilotApiTarget()` in `containers/api-proxy/providers/copilot-auth.js`; GitHub OAuth/PAT credentials now sent with `token` prefix to derived `copilot-api.*.ghe.com` targets while `Bearer` is preserved for BYOK credentials and the standard Copilot endpoint. Upgrade AWF to include github/gh-aw-firewall#6991. | Inspect api-proxy logs for 400 on `copilot-api.<tenant>.ghe.com`; confirm `GITHUB_SERVER_URL` is `*.ghe.com` and the target is the derived Copilot endpoint (not `api.business.githubcopilot.com`) | github/gh-aw-firewall#6989, github/gh-aw-firewall#6991 | |
Collaborator
|
@copilot address review feedback |
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Contributor
Author
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runner Doctor was missing two recent AWF failure modes: derived GHEC Copilot auth targets requiring
tokenauth, andcli-proxyENETUNREACHstartup failures in network-isolation mode.Failure mode catalog
400 bad request: Authorization header is badly formattedon derivedcopilot-api.<tenant>.ghe.comtargets.awf-cli-proxyENETUNREACH ... :18443whencli-proxyis not dual-homed onto the external bridge.Runner Doctor workflow guidance
Generated workflow metadata