Skip to content

docs: add B10 to runner doctor catalog (rootless perm-repair tag@digest timeout)#6078

Merged
lpcox merged 4 commits into
mainfrom
copilot/runner-doctor-update-b10
Jul 10, 2026
Merged

docs: add B10 to runner doctor catalog (rootless perm-repair tag@digest timeout)#6078
lpcox merged 4 commits into
mainfrom
copilot/runner-doctor-update-b10

Conversation

Copilot AI commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

PR #6025 introduced a new distinct failure mode not yet captured in the runner doctor knowledge base: fixArtifactPermissionsForRootless() was building compound tag@digest refs (e.g. agent:0.27.22@sha256:55f065...) that caused Docker to attempt GHCR manifest verification even under --pull never, timing out ~30 s per directory (~90 s total) when GHCR credentials were unavailable.

Changes

Adds B10 to all three mirrored catalog files:

  • shared/self-hosted-failure-modes.md — new Category B table row (after B9) + error-string quick lookup entry (after A15)
  • self-hosted-runner-doctor.md — § 3 symptom recognizer hint (~30 s timeout distinguishes B10 from A15's instant error); § 4 fix paragraph citing PR fix: use tag-only image ref in rootless permission repair #6025
  • agents/self-hosted-runner-doctor.md — identical four-point update to the embedded catalog copy (B10 table row, quick lookup entry, § 3 hint, § 4 fix)

B10 distinguishing signal: each [WARN] Rootless artifact permission repair failed attempt takes exactly ~30 s (TCP connect timeout to GHCR), not an instant failure — docker inspect agent:<tag> succeeds while docker pull --dry-run agent:<tag>@sha256:... times out.

Add new failure mode B10 (rootless permission repair timeout on
tag@digest ref) to all three mirrored catalog files:

- .github/workflows/shared/self-hosted-failure-modes.md
- .github/workflows/self-hosted-runner-doctor.md
- .github/agents/self-hosted-runner-doctor.md

B10 captures the symptom from PR #6025: fixArtifactPermissionsForRootless()
built compound tag@digest refs that caused Docker to attempt GHCR
manifest verification even under --pull never, timing out ~30 s per
directory. Fixed by resolvePermFixerImageRef() returning tag-only refs.

Closes #6069
Copilot AI changed the title [WIP] Update runner doctor catalog for new failure mode B10 docs: add B10 to runner doctor catalog (rootless perm-repair tag@digest timeout) Jul 10, 2026
Copilot AI requested a review from lpcox July 10, 2026 02:33
Copilot finished work on behalf of lpcox July 10, 2026 02:33
@lpcox lpcox marked this pull request as ready for review July 10, 2026 02:41
Copilot AI review requested due to automatic review settings July 10, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the self-hosted runner “runner doctor” knowledge base to document a newly identified failure mode (B10) where rootless artifact permission repair incurs ~30s GHCR timeouts due to compound tag@digest image references, even under --pull never.

Changes:

  • Adds B10 to the Category B failure-mode catalog (signal/root cause/fix/probes) and the error-string quick lookup.
  • Updates the runner-doctor symptom recognizer to distinguish B10 from similar modes via the characteristic ~30s timeout timing.
  • Adds a B10 “notable fix” entry referencing the remediation from #6025.
Show a summary per file
File Description
.github/workflows/shared/self-hosted-failure-modes.md Adds B10 to the shared failure-mode catalog and quick-lookup table.
.github/workflows/self-hosted-runner-doctor.md Adds B10 to the symptom recognizer and known-fixes section.
.github/agents/self-hosted-runner-doctor.md Mirrors the B10 updates into the embedded agent copy (catalog + recognizer + known-fixes + quick lookup).

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Low

Comment thread .github/workflows/shared/self-hosted-failure-modes.md Outdated
Comment thread .github/agents/self-hosted-runner-doctor.md Outdated
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

lpcox and others added 2 commits July 9, 2026 19:46
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

PR #6078 follows the applicable CONTRIBUTING.md guidelines; no comment is needed.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Result
API Status ✅ PASS
GH Check ✅ PASS
File Status ✅ PASS

Overall Result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #6078 · 55.1 AIC · ⊞ 3.3K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: PAT Auth Validation

Test Result
GitHub MCP connectivity
GitHub.com HTTP ⚠️ pre-step data unavailable
File write/read ⚠️ pre-step data unavailable

Overall: PASS (MCP verified; pre-step template outputs not expanded)
Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test Results

Test Status
GitHub MCP Connectivity ✅ PASS
GitHub.com HTTP ⚠️ N/A (template vars unexpanded)
File Write/Read ⚠️ N/A (template vars unexpanded)

Overall: PARTIAL — Workflow template variables were not substituted (pre-step outputs missing).

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results (Gemini)

  • GitHub MCP Testing: ❌ (Tools missing)
  • GitHub.com Connectivity: ❌ (Status 000, Exit 7)
  • File Writing Testing: ✅
  • Bash Tool Testing: ✅

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode

Test Result
GitHub MCP (list PRs)
GitHub.com connectivity (HTTP 200)
File write/read
BYOK inference (agent → api-proxy → api.githubcopilot.com)

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

cc @lpcox

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Services Connectivity

  • Redis PING: ❌ Network unreachable
  • PostgreSQL pg_isready: ❌ No response
  • PostgreSQL SELECT 1: ❌ Network unreachable

Overall: FAILhost.docker.internal (172.17.0.1) unreachable. Service containers may not be running or the host bridge network is unavailable.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Status Notes
Module Loading otel.js loads successfully; exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, and internal helpers
Test Suite 59 tests passed across 2 suites — 0 failures
Env Var Forwarding api-proxy-service.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME
Token Tracker Integration onUsage callback present in token-tracker-http.js as the OTEL hook point
OTEL Diagnostics No OTLP endpoint configured; spans fall back to local NDJSON (/var/log/api-proxy/otel.jsonl) per design

All 5 scenarios passed. OTEL tracing integration is working correctly.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Status: FAIL\n- fix: log stderr from rootless permission repair and make chroot-home removal non-fatal ✅\n- chore: upgrade gh-aw to v0.82.7 pre-release and recompile workflows ✅\n- File write ✅\n- Discussion ✅\n- Build ❌ npm ci failed: Exit handler never called!\n- Playwright ❌ unavailable in this environment\nOverall: FAIL

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • awmgmcpg
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@lpcox lpcox merged commit c38c273 into main Jul 10, 2026
74 of 76 checks passed
@lpcox lpcox deleted the copilot/runner-doctor-update-b10 branch July 10, 2026 02:51
@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3 ❌ NO
Node.js v24.18.0 v22.23.1 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

ALL_TESTS_PASSED: false — Python and Node.js versions differ between host and chroot.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color passed ✅ PASS
Go env passed ✅ PASS
Go uuid passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #6078 · 35.9 AIC · ⊞ 6.9K ·
Add label ready-for-aw to run again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🩺 Runner Doctor Update🩺 Runner Doctor update: B10 (rootless permission repair timeout on tag@digest ref)

3 participants