fix: use tag-only image ref in rootless permission repair by lpcox · Pull Request #6025 · github/gh-aw-firewall

lpcox · 2026-07-08T18:39:44Z

Problem

fixArtifactPermissionsForRootless() builds compound tag@digest image references (e.g., agent:0.27.22@sha256:55f065...) for its docker run --pull never command. Despite --pull never, Docker still attempts registry manifest verification for compound references. When GHCR credentials are unavailable (cleaned in an earlier workflow step), this verification times out after ~30 seconds per directory.

Observed in: https://github.com/github/gh-aw-mcpg/actions/runs/28965706129/job/85948370446#step:29:317

[WARN] Rootless artifact permission repair failed for /tmp/gh-aw/sandbox/firewall/logs (exit 1)
[WARN] Rootless artifact permission repair failed for /tmp/gh-aw/sandbox/firewall/audit (exit 1)
[WARN] Rootless artifact permission repair failed for /tmp/awf-1783534966689-chroot-home (exit 1)
[WARN] Failed to remove chroot home directory after permission repair: Error: EACCES: permission denied

Each repair attempt takes exactly ~30s (TCP connect timeout), wasting ~90s total per run.

Fix

Strip the @sha256:... digest from the image ref in resolvePermFixerImageRef(). This is a local-only operation (--pull never) and the image is already tagged locally (e.g., agent:0.27.22) by the download step — no digest verification is needed.

Impact

Eliminates ~90s of wasted CI time (3 directories × 30s timeout)
Fixes cascading EACCES error when chroot-home cleanup fails after permission repair timeout
No functional change to the permission repair itself (same image, just referenced by tag only)

The permission fixer was building compound tag@digest image references (e.g. agent:0.27.22@sha256:...) for its docker run --pull never command. Docker still attempts registry verification for tag@digest refs even with --pull never, causing a 30-second timeout per directory when GHCR credentials are unavailable (cleaned before the agent step completes). Strip the digest from the image ref since this is a local-only operation and the image is already tagged locally by the download step. Fixes ~90s of wasted time and eliminates cascading EACCES errors when chroot-home cleanup fails after the permission repair times out. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

lpcox · 2026-07-08T18:41:14Z

@copilot merge main

github-actions · 2026-07-08T18:42:33Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	98.92%	98.92%	➡️ +0.00%
Statements	98.87%	98.87%	➡️ +0.00%
Functions	99.44%	99.44%	➡️ +0.00%
Branches	95.39%	95.35%	📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR updates the rootless artifact permission repair flow to avoid using compound Docker image references (tag@digest) when running a local-only docker run --pull never, preventing Docker from attempting registry verification (and timing out when GHCR creds are unavailable).

Changes:

Stop building tag@digest image refs for the permission-fixer container and instead use a tag-only ref (<registry>/<image>:<tag>).
Remove the now-unneeded buildRuntimeImageRef import from the permission repair module.

Show a summary per file

File	Description
`src/artifact-permissions.ts`	Switches permission-fixer image reference generation to tag-only to avoid registry verification timeouts under `--pull never`.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 1/1 changed files
Comments generated: 1
Review effort level: Low

+    // Use tag-only ref (no digest) because this runs with --pull never.
+    // Including the digest causes Docker to attempt registry verification
+    // even with --pull never, which times out if credentials are unavailable.
+    return `${registry}/${imageName}:${parsedImageTag.tag}`;


Copilot · 2026-07-08T18:45:35Z

@copilot merge main

Merged from main—the branch was already up to date, so no merge commit was needed. Current HEAD: a55ddeb.

github-actions · 2026-07-08T18:52:27Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-07-08T18:52:27Z

✅ Smoke Gemini completed. All facets verified. 💎

Smoke test completed. Results reported to PR #6025. Status: FAIL.

github-actions · 2026-07-08T18:52:29Z

✅ Smoke Claude passed

github-actions · 2026-07-08T18:52:30Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-07-08T18:52:31Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-07-08T18:52:31Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-07-08T18:52:32Z

✅ Build Test Suite completed successfully!

github-actions · 2026-07-08T18:52:33Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

github-actions · 2026-07-08T18:52:33Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-07-08T18:52:34Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-07-08T18:52:35Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-07-08T18:52:35Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-07-08T18:52:40Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

github-actions · 2026-07-08T18:52:56Z

✅ Contribution Check completed successfully!

PR #6025 follows the applicable CONTRIBUTING.md guidelines; no contribution-guidelines comment needed.

github-actions · 2026-07-08T18:57:02Z

Smoke Test Results for Gemini

GitHub MCP Testing: ❌ (Tools not found)
GitHub.com Connectivity: ❌ (DNS failure)
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall Status: FAIL

Merged PRs (found in log):

fix: add Gemini usageMetadata support to token parsers (fix: add Gemini usageMetadata support to token parsers #6008)
(Log grafted, unable to fetch 2nd PR)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:57:08Z

Smoke Test: Claude Engine Validation

Overall result: ✅ PASS

Check	Status
API	✅ PASS
gh CLI	✅ PASS
File	✅ PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #6025 · 34.9 AIC · ⊞ 3.3K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:57:37Z

Smoke Test Results

Test	Status
GitHub MCP Connectivity	✅ Connected
GitHub.com HTTP	✅ HTTP 200
File Write/Read	⚠️ Pre-step template vars not expanded

Overall: PASS (core connectivity verified)

@lpcox — smoke tests completed.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:57:52Z

Smoke Test: GitHub Actions Services Connectivity

Redis PING: ❌ Network is unreachable
PostgreSQL pg_isready: ❌ No response
PostgreSQL SELECT 1: ❌ Network is unreachable

Overall: FAIL — host.docker.internal (172.17.0.1) is not reachable from this runner environment.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:57:59Z

Smoke Test: Copilot PAT Auth — PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅ 200
File write/read	✅

Overall: PASS | Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:58:03Z

✅ Smoke Test: Copilot BYOK (Direct) Mode — PASS

Test	Result
BYOK Inference	✅
Direct Mode	✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:58:04Z

@lpcox smoke test results:

GitHub MCP Testing: ✅
GitHub.com Connectivity: ✅
File write/read test: ✅
BYOK inference test: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:58:30Z

Smoke test results:\n- #6008 fix: add Gemini usageMetadata support to token parsers: ✅\n- #6005 Clarify MCP localhost rewrite, transport semantics, and reachability under network isolation: ✅\n- #6025 fix: use tag-only image ref in rootless permission repair: ✅\n- Overall: PASS

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

awmgmcpg
registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:58:57Z

Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌ NO
Node.js	v24.18.0	v22.23.1	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall: ❌ FAILED — Python and Node.js versions differ between host and chroot environments.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T18:59:59Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	ok	✅ PASS
Go	env	✅	ok	✅ PASS
Go	uuid	✅	ok	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #6025 · 40.9 AIC · ⊞ 6.9K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-08T19:01:20Z

🔭 OTEL Tracing Smoke Test Results

Scenario	Status	Notes
1. Module Loading	✅ Pass	`otel.js` loads successfully; `isEnabled: true`; exports 14 symbols
2. Test Suite	✅ Pass	59/59 tests pass across 2 suites (`otel.test.js`, `otel-fanout.test.js`)
3. Env Var Forwarding	✅ Pass	OTEL env vars forwarded via `src/services/api-proxy-env-config.ts` (correct module)
4. Token Tracker Integration	✅ Pass	`onUsage` callback present in `token-tracker-http.js`
5. OTEL Diagnostics	i️ Expected	No span file — api-proxy not active during pre-step validation; graceful degradation confirmed

All scenarios pass. OTEL tracing integration is functional.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

Add new failure mode B10 (rootless permission repair timeout on tag@digest ref) to all three mirrored catalog files: - .github/workflows/shared/self-hosted-failure-modes.md - .github/workflows/self-hosted-runner-doctor.md - .github/agents/self-hosted-runner-doctor.md B10 captures the symptom from PR #6025: fixArtifactPermissionsForRootless() built compound tag@digest refs that caused Docker to attempt GHCR manifest verification even under --pull never, timing out ~30 s per directory. Fixed by resolvePermFixerImageRef() returning tag-only refs. Closes #6069

…st timeout) (#6078) * Initial plan * docs: add B10 failure mode to runner doctor catalog Add new failure mode B10 (rootless permission repair timeout on tag@digest ref) to all three mirrored catalog files: - .github/workflows/shared/self-hosted-failure-modes.md - .github/workflows/self-hosted-runner-doctor.md - .github/agents/self-hosted-runner-doctor.md B10 captures the symptom from PR #6025: fixArtifactPermissionsForRootless() built compound tag@digest refs that caused Docker to attempt GHCR manifest verification even under --pull never, timing out ~30 s per directory. Fixed by resolvePermFixerImageRef() returning tag-only refs. Closes #6069 * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Landon Cox <landon.cox@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 8, 2026 18:39

Copilot started reviewing on behalf of lpcox July 8, 2026 18:41 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Copilot started work on behalf of lpcox July 8, 2026 18:43 View session

Copilot finished work on behalf of lpcox July 8, 2026 18:45

lpcox added the ready-for-aw label Jul 8, 2026

lpcox temporarily deployed to aoai-model July 8, 2026 18:52 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Jul 8, 2026

[aw] No-Op Runs #5883

Open

github-actions Bot added the smoke-claude label Jul 8, 2026

github-actions Bot added the smoke-copilot label Jul 8, 2026

lpcox temporarily deployed to aoai-model July 8, 2026 18:57 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-pat label Jul 8, 2026

github-actions Bot added smoke-copilot-byok smoke-copilot-byok-aoai-entra labels Jul 8, 2026

lpcox temporarily deployed to aoai-model July 8, 2026 18:58 — with GitHub Actions Inactive

github-actions Bot added the smoke-codex label Jul 8, 2026

github-actions Bot added the build-test label Jul 8, 2026

github-actions Bot mentioned this pull request Jul 8, 2026

[aw] Smoke Copilot BYOK AOAI (api-key) failed #6029

Closed

lpcox merged commit 054a359 into main Jul 9, 2026
89 of 91 checks passed

lpcox deleted the fix/artifact-permissions-digest-timeout branch July 9, 2026 02:39

Copilot AI mentioned this pull request Jul 10, 2026

docs: add B10 to runner doctor catalog (rootless perm-repair tag@digest timeout) #6078

Merged

Uh oh!

Conversation

lpcox commented Jul 8, 2026

Problem

Fix

Impact

Uh oh!

lpcox commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Copilot AI commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Smoke Test Results for Gemini

Uh oh!

github-actions Bot commented Jul 8, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Jul 8, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented Jul 8, 2026

Smoke Test: GitHub Actions Services Connectivity

Uh oh!

github-actions Bot commented Jul 8, 2026

Smoke Test: Copilot PAT Auth — PASS

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading