fix: make test-coverage-reporter resilient to test failures by lpcox · Pull Request #5879 · github/gh-aw-firewall

lpcox · 2026-07-03T15:49:03Z

Summary

The test-coverage-reporter workflow was failing entirely when any test failed (exit code 1 from npm run test:coverage), which caused all subsequent steps to be skipped and prevented the agent from running.

Root Cause

In the failed run, a test in compose-generator.test.ts:511 was failing (fixed separately in #5857). Because the coverage step used set -o pipefail and the command exited non-zero, the step failed and all downstream data-extraction steps were skipped. The agent never received any coverage data to report on.

Fix

Since this is a coverage reporter (not a quality gate), it should continue with whatever coverage data is available:

Changed the coverage step to capture the exit code rather than fail on it
Emits a ::warning:: annotation when tests exit non-zero
Always exits 0 so subsequent data-extraction steps can proceed
All downstream steps already have graceful fallbacks for missing coverage-summary.json

This means the reporter will still produce useful output even when individual tests are broken — it'll report on the coverage of tests that did pass.

Fixes #5855

The reporter workflow was failing entirely when any test failed (exit code 1 from npm run test:coverage), skipping all subsequent steps and preventing the agent from running. Since this is a coverage reporter (not a gate), it should continue with whatever coverage data is available and report on it. Changed the coverage step to capture the exit code, emit a warning annotation when tests fail, and continue to the data-extraction steps. All downstream steps already have fallbacks for missing coverage data. Fixes #5855 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-07-03T15:51:01Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	98.61%	98.64%	📈 +0.03%
Statements	98.53%	98.57%	📈 +0.04%
Functions	99.43%	99.43%	➡️ +0.00%
Branches	94.37%	94.37%	➡️ +0.00%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/workdir-setup.ts`	93.0% → 94.8% (+1.74%)	93.0% → 94.8% (+1.74%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR aims to make the Test Coverage Reporter workflow resilient to failing tests so it can still extract whatever coverage artifacts exist and run the downstream reporting/agent steps, instead of aborting the job when npm run test:coverage exits non-zero.

Changes:

Updates the “Run coverage” step to continue on non-zero test exit codes and emit a ::warning:: annotation.
Regenerates the corresponding locked workflow YAML to match the markdown workflow source.
Includes lockfile metadata/body hash updates for related locked workflows (likely via lock regeneration).

Show a summary per file

File	Description
.github/workflows/test-coverage-reporter.md	Adjusts the coverage-running step to avoid failing the workflow when tests fail, enabling downstream reporting.
.github/workflows/test-coverage-reporter.lock.yml	Regenerated locked workflow reflecting the updated coverage step behavior.
.github/workflows/self-hosted-runner-doctor.lock.yml	Locked workflow metadata/body hash update from regeneration.
.github/workflows/self-hosted-runner-doctor-updater.lock.yml	Locked workflow metadata/body hash update from regeneration.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 1
Review effort level: Low

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-07-03T15:52:07Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-07-03T15:54:48Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-07-03T15:54:49Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-07-03T15:54:49Z

✅ Smoke Claude passed

github-actions · 2026-07-03T15:54:49Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

github-actions · 2026-07-03T15:54:50Z

✅ Smoke Gemini completed. All facets verified. 💎

Smoke test completed with partial failures. Results reported to PR #5879.

github-actions · 2026-07-03T15:54:50Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-07-03T15:54:50Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-07-03T15:54:51Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-07-03T15:54:51Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-07-03T15:54:52Z

Chroot tests failed Smoke Chroot failed - See logs for details.

github-actions · 2026-07-03T15:54:53Z

✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

github-actions · 2026-07-03T15:54:54Z

✅ Build Test Suite completed successfully!

github-actions · 2026-07-03T15:54:59Z

✅ Contribution Check completed successfully!

github-actions · 2026-07-03T15:55:08Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-07-03T15:59:26Z

Smoke Test: Copilot BYOK Direct Mode ✅

Results:

✅ GitHub MCP Connectivity
✅ GitHub.com HTTP (200)
✅ File Write/Read
✅ BYOK Inference Path

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) via api-proxy sidecar → api.githubcopilot.com

Status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T15:59:45Z

🔬 Smoke Test Results

Test	Result
GitHub.com Connectivity	✅ HTTP 200
File Write/Read	⚠️ Pre-step template vars unresolved
MCP Connectivity	✅ Verified

Overall: PASS (core connectivity confirmed)

Author: @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T15:59:46Z

Smoke Test: Claude Engine Validation

Check	Result
API Status	✅ PASS
GH Check	✅ PASS
File Status	✅ PASS

Overall Result: PASS ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #5879 · 55.6 AIC · ⊞ 3.3K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T15:59:47Z

Thanks for the clear PR description and for updating both the workflow source and generated lock file. One contribution-guideline item looks missing: CONTRIBUTING.md’s Contributing Code, Testing, and Pull request requirements sections ask contributors to include tests for new functionality. Since this changes the coverage reporter’s failure-handling behavior, could you add a focused test or documented validation for the reporter behavior so future changes don’t regress it?

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Contribution Check for #5879 · 43.7 AIC · ⊞ 20.8K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:00:05Z

Smoke Test: Copilot PAT Auth

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅ 200
File write/read	⚠️ template vars unresolved

Overall: PASS | Auth mode: PAT (COPILOT_GITHUB_TOKEN)

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:00:15Z

@lpcox
✅ GitHub MCP test
✅ GitHub.com connectivity
✅ File I/O test
✅ BYOK inference test

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:00:24Z

📡 OTel Tracing Smoke Test Results

Scenario	Result	Detail
1. Module Loading	✅ Pass	`otel.js` loaded; `isEnabled()=true`; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
2. Test Suite	✅ Pass	59 tests passed, 0 failed across 2 suites (`otel.test.js`, `otel-fanout.test.js`)
3. Env Var Forwarding	⚠️ Pending	`src/services/api-proxy-service.ts` does not yet forward `OTEL_EXPORTER_OTLP_ENDPOINT`/`GITHUB_AW_OTEL_TRACE_ID` — expected during development
4. Token Tracker Integration	✅ Pass	`onUsage` callback exists in `token-tracker-http.js` as OTEL hook point
5. OTEL Diagnostics	⚠️ Pending	No span file found — api-proxy not active during this run (no proxied requests)

Summary: Core OTEL implementation is solid (module loads, all 59 unit tests pass, token-tracker hook in place). Env var forwarding and live span export are expected-pending until api-proxy-service.ts wires up the OTEL env vars to the container.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:00:34Z

✅ MCP connectivity check
✅ GitHub.com connectivity
✅ File write/read test
✅ BYOK inference path

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:00:52Z

Smoke Test: Services Connectivity

Redis PING: ❌ (Network is unreachable)
PostgreSQL pg_isready: ❌ (no response)
PostgreSQL SELECT 1: ❌ (Network is unreachable)

Overall: FAIL — host.docker.internal (172.17.0.1) unreachable. Service containers not accessible from this runner.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:01:05Z

fix: make test-coverage-reporter resilient to test failures
fix: Copilot Business 'token' prefix not overridable by AWF_PLATFORM_TYPE
refactor: extract shared credential-isolation scaffold for API proxy providers
GitHub title check: ✅
File write/read: ✅
Discussion comment: ✅
Build: ✅
Overall status: PASS

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

awmgmcpg
registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:01:52Z

Smoke Test Results (Gemini)

GitHub MCP: ❌ (Tool 'github:list_pull_requests' not found)
GitHub.com Connectivity: ❌ (Network unreachable, status 000)
File Writing: ✅
Bash Tool: ✅

PRs Found (Git Log):

chore: fix 3 moderate CVEs via dep updates (2026-07-03) (chore: fix 3 moderate CVEs via dep updates (2026-07-03) #5856)

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

github-actions · 2026-07-03T16:04:41Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	passed	✅ PASS
Node.js	execa	✅	passed	✅ PASS
Node.js	p-limit	✅	passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #5879 · 130.7 AIC · ⊞ 6.9K · ◷
_{Add label ready-for-aw to run again}

Copilot AI review requested due to automatic review settings July 3, 2026 15:49

Copilot started reviewing on behalf of lpcox July 3, 2026 15:49 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Comment thread .github/workflows/test-coverage-reporter.md

Potential fix for pull request finding

ea83c35

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

lpcox added the ready-for-aw label Jul 3, 2026

lpcox temporarily deployed to aoai-model July 3, 2026 15:54 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok label Jul 3, 2026

github-actions Bot added smoke-copilot smoke-claude labels Jul 3, 2026

github-actions Bot mentioned this pull request Jul 3, 2026

[aw] Smoke Chroot failed #5880

Open

github-actions Bot added the smoke-copilot-pat label Jul 3, 2026

github-actions Bot added the smoke-copilot-byok-aoai-apikey label Jul 3, 2026

lpcox temporarily deployed to aoai-model July 3, 2026 16:00 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok-aoai-entra label Jul 3, 2026

lpcox temporarily deployed to aoai-model July 3, 2026 16:00 — with GitHub Actions Inactive

github-actions Bot added the smoke-codex label Jul 3, 2026

github-actions Bot added the build-test label Jul 3, 2026

lpcox merged commit 5612a0d into main Jul 3, 2026
88 of 89 checks passed

lpcox deleted the copilot/fix-artifact-permission-repair branch July 3, 2026 17:30

Uh oh!

Conversation

lpcox commented Jul 3, 2026

Summary

Root Cause

Fix

Uh oh!

github-actions Bot commented Jul 3, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Smoke Test: Copilot BYOK Direct Mode ✅

Uh oh!

github-actions Bot commented Jul 3, 2026

🔬 Smoke Test Results

Uh oh!

github-actions Bot commented Jul 3, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Smoke Test: Copilot PAT Auth

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

📡 OTel Tracing Smoke Test Results

Uh oh!

github-actions Bot commented Jul 3, 2026

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading