Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/agents/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ Prefer the narrowest match. Examples:
- Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374)
- `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407)
- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558)
- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811)
- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked)
- `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks → B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter)
- `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657)
Expand Down Expand Up @@ -143,6 +144,8 @@ D10 / github/gh-aw-firewall#6407 — On `--container-runtime sbx`, the Copilot C

D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139). **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` does a one-shot retry (`MAX_GVISOR_AGENT_RETRIES = 1`) when gVisor exits 134/139 within `GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`, but this does not prevent the underlying crash.

D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled. **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** api-proxy `checkUnknownModelRejection` now allows Copilot `auto` through pre-flight (`provider === 'copilot' && model.toLowerCase() === 'auto'`) and charges credits from the response's resolved model metadata. Non-Copilot providers still reject unresolved `auto`. Workaround (older AWF): pin a concrete priced model (for example `model: claude-sonnet-4.6`).

C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.

C8 / github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens).
Expand Down Expand Up @@ -264,6 +267,7 @@ Establish these facts before matching a failure mode:
| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-<pid>` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 |
| D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 |
| D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault, often before any model or tool call is issued. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since <restart-time> -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 |
| D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`; harness logs `awf-reflect: fetching (apiproxy/redacted)` then `request failed: fetch failed`, followed by `400 ... Model "auto" has no AI credits pricing and no default pricing is configured`; retries fail identically with zero tokens consumed. Same workflow succeeds under default (non-isolated) AWF runtime. | Isolated agent runtimes (gVisor, sbx) may not reach `(apiproxy/redacted)`, which the harness uses to pre-resolve `auto` to a concrete priced model. Without that resolution, api-proxy `maxAiCredits` pre-flight guard (`checkUnknownModelRejection` in `guards/ai-credits-guard.js`) had no pricing for literal `auto` and rejected the request with HTTP 400, even though Copilot resolves `auto` server-side and returns priced resolved-model metadata post-response. | **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** `checkUnknownModelRejection` now allows `provider === 'copilot' && model.toLowerCase() === 'auto'` to pass pre-flight; AI-credit accounting then uses the response's resolved model. Non-Copilot providers still reject unresolved `auto`. Upgrade AWF to include github/gh-aw-firewall#6811. Workaround (older AWF): pin a concrete priced model in workflow frontmatter (for example `model: claude-sonnet-4.6`) to avoid catalog-based `auto` resolution under isolated runtimes. | Confirm `sandbox.agent.runtime: gvisor` or `sbx` in resolved AWF config; check api-proxy logs for `400 ... Model "auto" has no AI credits pricing` alongside harness `awf-reflect: request failed: fetch failed`; verify whether `apiProxy.maxAiCredits` is enabled (guard only fires when enabled) | github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 |

## Error-string quick lookup

Expand Down Expand Up @@ -304,6 +308,7 @@ Establish these facts before matching a failure mode:
| Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) |
| `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) |
| `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) |
| `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) |
| `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) |
| `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks | B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) |
| `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) |
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ Prefer the narrowest match. Examples:
- Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374)
- `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407)
- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558)
- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811)
- `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked)
- `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks → B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter)
- `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657)
Expand Down Expand Up @@ -170,6 +171,8 @@ D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire h

D10 / github/gh-aw-firewall#6407 — On `--container-runtime sbx`, the Copilot CLI installed rootless to `~/.local/bin` was not found because sbx executes commands via `bash -lc` (login shell), and login initialization can reset PATH and discard `--env PATH=...`. **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx now wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`.

D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled. **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** api-proxy `checkUnknownModelRejection` now allows Copilot `auto` through pre-flight (`provider === 'copilot' && model.toLowerCase() === 'auto'`) and charges credits from the response's resolved model metadata. Non-Copilot providers still reject unresolved `auto`. Workaround (older AWF): pin a concrete priced model (for example `model: claude-sonnet-4.6`).

C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.

C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens).
Expand Down
Loading
Loading