diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index dc567dfa1..95f0381c2 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -81,6 +81,7 @@ Prefer the narrowest match. Examples: - Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374) - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) +- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) - `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) - `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks → B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) - `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) @@ -143,6 +144,8 @@ D10 / github/gh-aw-firewall#6407 — On `--container-runtime sbx`, the Copilot C D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139). **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` does a one-shot retry (`MAX_GVISOR_AGENT_RETRIES = 1`) when gVisor exits 134/139 within `GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`, but this does not prevent the underlying crash. +D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled. **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** api-proxy `checkUnknownModelRejection` now allows Copilot `auto` through pre-flight (`provider === 'copilot' && model.toLowerCase() === 'auto'`) and charges credits from the response's resolved model metadata. Non-Copilot providers still reject unresolved `auto`. Workaround (older AWF): pin a concrete priced model (for example `model: claude-sonnet-4.6`). + C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved. C8 / github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens). @@ -264,6 +267,7 @@ Establish these facts before matching a failure mode: | D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 | | D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 | | D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault, often before any model or tool call is issued. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 | +| D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`; harness logs `awf-reflect: fetching (apiproxy/redacted)` then `request failed: fetch failed`, followed by `400 ... Model "auto" has no AI credits pricing and no default pricing is configured`; retries fail identically with zero tokens consumed. Same workflow succeeds under default (non-isolated) AWF runtime. | Isolated agent runtimes (gVisor, sbx) may not reach `(apiproxy/redacted)`, which the harness uses to pre-resolve `auto` to a concrete priced model. Without that resolution, api-proxy `maxAiCredits` pre-flight guard (`checkUnknownModelRejection` in `guards/ai-credits-guard.js`) had no pricing for literal `auto` and rejected the request with HTTP 400, even though Copilot resolves `auto` server-side and returns priced resolved-model metadata post-response. | **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** `checkUnknownModelRejection` now allows `provider === 'copilot' && model.toLowerCase() === 'auto'` to pass pre-flight; AI-credit accounting then uses the response's resolved model. Non-Copilot providers still reject unresolved `auto`. Upgrade AWF to include github/gh-aw-firewall#6811. Workaround (older AWF): pin a concrete priced model in workflow frontmatter (for example `model: claude-sonnet-4.6`) to avoid catalog-based `auto` resolution under isolated runtimes. | Confirm `sandbox.agent.runtime: gvisor` or `sbx` in resolved AWF config; check api-proxy logs for `400 ... Model "auto" has no AI credits pricing` alongside harness `awf-reflect: request failed: fetch failed`; verify whether `apiProxy.maxAiCredits` is enabled (guard only fires when enabled) | github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 | ## Error-string quick lookup @@ -304,6 +308,7 @@ Establish these facts before matching a failure mode: | Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) | | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | +| `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) | | `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) | | `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks | B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) | | `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) | diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 3b9206cdf..44fd5a63f 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -110,6 +110,7 @@ Prefer the narrowest match. Examples: - Copilot CLI exits immediately (exit code 1, ~0.5 s, zero stdout/stderr) after AWF upgrade on Docker or gVisor but not sbx → B14 (`~/.copilot/config.json` incorrectly added to credential deny list; fixed in github/gh-aw-firewall#6374) - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) +- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) - `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) - `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks → B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) - `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) @@ -170,6 +171,8 @@ D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire h D10 / github/gh-aw-firewall#6407 — On `--container-runtime sbx`, the Copilot CLI installed rootless to `~/.local/bin` was not found because sbx executes commands via `bash -lc` (login shell), and login initialization can reset PATH and discard `--env PATH=...`. **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx now wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. +D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled. **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** api-proxy `checkUnknownModelRejection` now allows Copilot `auto` through pre-flight (`provider === 'copilot' && model.toLowerCase() === 'auto'`) and charges credits from the response's resolved model metadata. Non-Copilot providers still reject unresolved `auto`. Workaround (older AWF): pin a concrete priced model (for example `model: claude-sonnet-4.6`). + C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved. C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens). diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index c75fda5fe..9af16bd96 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -87,6 +87,7 @@ Establish these facts before matching a failure mode: | D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 | | D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 | | D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault, often before any model or tool call is issued. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 | +| D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`; harness logs `awf-reflect: fetching (apiproxy/redacted)` then `request failed: fetch failed`, followed by `400 ... Model "auto" has no AI credits pricing and no default pricing is configured`; retries fail identically with zero tokens consumed. Same workflow succeeds under default (non-isolated) AWF runtime. | Isolated agent runtimes (gVisor, sbx) may not reach `(apiproxy/redacted)`, which the harness uses to pre-resolve `auto` to a concrete priced model. Without that resolution, api-proxy `maxAiCredits` pre-flight guard (`checkUnknownModelRejection` in `guards/ai-credits-guard.js`) had no pricing for literal `auto` and rejected the request with HTTP 400, even though Copilot resolves `auto` server-side and returns priced resolved-model metadata post-response. | **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** `checkUnknownModelRejection` now allows `provider === 'copilot' && model.toLowerCase() === 'auto'` to pass pre-flight; AI-credit accounting then uses the response's resolved model. Non-Copilot providers still reject unresolved `auto`. Upgrade AWF to include github/gh-aw-firewall#6811. Workaround (older AWF): pin a concrete priced model in workflow frontmatter (for example `model: claude-sonnet-4.6`) to avoid catalog-based `auto` resolution under isolated runtimes. | Confirm `sandbox.agent.runtime: gvisor` or `sbx` in resolved AWF config; check api-proxy logs for `400 ... Model "auto" has no AI credits pricing` alongside harness `awf-reflect: request failed: fetch failed`; verify whether `apiProxy.maxAiCredits` is enabled (guard only fires when enabled) | github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 | ## Error-string quick lookup @@ -127,6 +128,7 @@ Establish these facts before matching a failure mode: | Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor | B14 (`.copilot/config.json` masked as credential; fixed in github/gh-aw-firewall#6374) | | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | +| `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) | | `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) | | `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks | B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) | | `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) | diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index d29f4b986..47eab1aec 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -88,6 +88,10 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558'); expect(content).toContain('| `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11'); expect(content).toContain('- D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash'); + expect(content).toContain('| D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`;'); + expect(content).toContain('`checkUnknownModelRejection` now allows `provider === \'copilot\' && model.toLowerCase() === \'auto\'` to pass pre-flight'); + expect(content).toContain('github/gh-aw-firewall#6810, github/gh-aw-firewall#6811'); + expect(content).toContain('| `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12'); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); @@ -96,16 +100,20 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; safe-output validation fails even though the agent completed → D8'); expect(source).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); expect(source).toContain('- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11'); + expect(source).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.'); expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.'); expect(source).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).'); + expect(source).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); expect(portableAgent).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(portableAgent).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); expect(portableAgent).toContain('- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5'); expect(portableAgent).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; safe-output validation fails even though the agent completed → D8'); expect(portableAgent).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); expect(portableAgent).toContain('- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11'); + expect(portableAgent).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); expect(portableAgent).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).'); + expect(portableAgent).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); }); });