Skip to content

docs: add B7 failure mode (EACCES during chroot-home cleanup in rootless Docker)#5692

Merged
lpcox merged 1 commit into
mainfrom
runner-doctor-b7-eacces-chroot-home
Jun 29, 2026
Merged

docs: add B7 failure mode (EACCES during chroot-home cleanup in rootless Docker)#5692
lpcox merged 1 commit into
mainfrom
runner-doctor-b7-eacces-chroot-home

Conversation

@lpcox

@lpcox lpcox commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the B7 failure mode to the runner doctor knowledge base, as proposed in #5689.

B7: AWF exits with unhandled EACCES during cleanup when removeWorkDirectories() tries to delete agent-written files in /tmp/awf-<ts>-chroot-home/ that are owned by remapped UIDs (rootless Docker UID namespace remapping). Fixed in AWF v0.27.13 via a repair-container pattern.

Changes

Updates three files:

  • .github/workflows/shared/self-hosted-failure-modes.md — B7 row + error-string lookup entry
  • .github/workflows/self-hosted-runner-doctor.md — symptom → failure mode hint
  • .github/agents/self-hosted-runner-doctor.md — B7 row + error-string lookup + symptom hint

Closes #5689

…ess Docker)

Add B7 to the runner doctor knowledge base across all three files:
- .github/workflows/shared/self-hosted-failure-modes.md
- .github/workflows/self-hosted-runner-doctor.md
- .github/agents/self-hosted-runner-doctor.md

B7 covers the case where AWF's removeWorkDirectories() fails with
EACCES on agent-written files in the chroot-home temp directory due to
UID namespace remapping in rootless Docker mode. Fixed in AWF v0.27.13.

Closes #5689

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 29, 2026 21:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new self-hosted runner “runner doctor” failure mode entry (B7) to document and help diagnose AWF cleanup failures on rootless Docker where UID namespace remapping can cause EACCES during deletion of chroot-home temp files.

Changes:

  • Added failure mode B7 to the shared failure-mode catalog, including an error-string quick lookup entry.
  • Updated the workflow runner-doctor playbook to map the new EACCES/unlink symptom to B7.
  • Updated the portable agent copy of the runner-doctor doc to keep it in sync with the shared catalog and playbook.
Show a summary per file
File Description
.github/workflows/shared/self-hosted-failure-modes.md Adds B7 row and corresponding error-string lookup mapping for chroot-home cleanup EACCES/unlink.
.github/workflows/self-hosted-runner-doctor.md Adds a symptom → failure mode hint mapping the cleanup EACCES/unlink pattern to B7.
.github/agents/self-hosted-runner-doctor.md Mirrors the B7 row + lookup entry + symptom hint in the portable agent version to keep catalogs consistent.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 0
  • Review effort level: Low

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

Contribution check complete for PR #5692: the docs-only change follows the applicable CONTRIBUTING.md guidelines, with clear PR description, issue reference, appropriate file placement, and no missing tests required for new code functionality.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Result
API status ✅ PASS
GH check ✅ PASS
File status ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #5692 · 52.2 AIC · ⊞ 3.3K ·

@github-actions

Copy link
Copy Markdown
Contributor

Copilot BYOK (Direct) Mode: PASS

All smoke tests passed:

  • ✅ GitHub MCP connectivity
  • ✅ GitHub.com HTTP (200)
  • ✅ File write/read
  • ✅ BYOK inference path active

Running direct BYOK mode via COPILOT_PROVIDER_API_KEY → api-proxy sidecar → api.githubcopilot.com

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Smoke Test: PASS

Test Result
GitHub MCP connectivity
GitHub.com HTTP ✅ 200
File write/read

PR: docs: add B7 failure mode (EACCES during chroot-home cleanup in rootless Docker)
Author: @lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test: Copilot PAT — PASS

Test Result
GitHub MCP connectivity
GitHub.com HTTP
File write/read

Auth mode: PAT (COPILOT_GITHUB_TOKEN)
Author: @lpcox

Overall: PASS

🔑 PAT report filed by Smoke Copilot PAT

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test

  • fix: export missing symbols required by tests
  • Remove unused \buildProviderTargetEnv` export from API proxy env config` ✅
  • GitHub reads
  • Playwright title
  • File write
  • Discussion lookup
  • Build
  • Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox

  • MCP testing: ✅
  • GitHub.com connectivity: ✅
  • File I/O test: ✅
  • BYOK inference: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Status: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

Copy link
Copy Markdown
Contributor

🧪 Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3 ❌ NO
Node.js v24.17.0 v22.23.0 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ Tests did not fully pass — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox Smoke Test (Direct BYOK → Azure Foundry) Results:

  • fix: export missing symbols required by tests – ✅
  • Remove unused buildProviderTargetEnv export from API proxy env config – ✅
  • GitHub.com connectivity – ✅
  • File I/O – ✅
  • BYOK inference – ✅

Running in direct BYOK mode via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)

Overall: PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

Copy link
Copy Markdown
Contributor

🔬 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Result Notes
1. Module Loading otel.js loads cleanly; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled + test internals
2. Test Suite 59 tests pass across otel.test.js + otel-fanout.test.js (run from containers/api-proxy/)
3. Env Var Forwarding src/services/api-proxy-env-config.ts:116–123 forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME
4. Token Tracker Integration onUsage callback present in token-tracker-http.js:324 — invoked after usage extraction, wired as OTEL hook point
5. OTEL Diagnostics No OTLP endpoint configured → graceful degradation to file exporter (/var/log/api-proxy/otel.jsonl); no span export errors

Overall: all scenarios pass (graceful degradation confirmed for unconfigured OTEL environment).

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: GitHub Actions Services Connectivity

  • Redis PING: ❌ (connection timed out — port 6379 blocked)
  • PostgreSQL pg_isready: ❌ (no response — port 5432 blocked)
  • PostgreSQL SELECT 1: ❌ (connection timed out — port 5432 blocked)

Result: FAIL

host.docker.internal resolves to 172.17.0.1 but both ports are in AWF DANGEROUS_PORTS. AWF_ALLOW_HOST_PORTS=80,443,8080 — service ports not included. No AWF_HOST_SERVICE_PORTS env var was set.

🔌 Service connectivity validated by Smoke Services

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx all passed ✅ PASS
Node.js execa all passed ✅ PASS
Node.js p-limit all passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for #5692 · 45.5 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Gemini Engine Validation

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

@lpcox lpcox merged commit 8aeeea3 into main Jun 29, 2026
81 checks passed
@lpcox lpcox deleted the runner-doctor-b7-eacces-chroot-home branch June 29, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🩺 Runner Doctor UpdateRunner Doctor Update: B7 — EACCES during chroot-home cleanup in rootless Docker

2 participants