fix(test): update BYOK workflow test to match recompiled lock files by lpcox · Pull Request #5950 · github/gh-aw-firewall

lpcox · 2026-07-06T13:14:24Z

Summary

The BYOK workflow test (scripts/ci/smoke-copilot-byok-workflow.test.ts) was failing after the lock files were recompiled in #5946 (gh-aw v0.82.2 upgrade).

Problem

The recompilation changed the agent job's model selection from:

COPILOT_MODEL: ${{ env.COPILOT_MODEL }}

to a vars-based expression:

COPILOT_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'claude-sonnet-4.6' }}

The test was asserting the old ${{ env.COPILOT_MODEL }} pattern existed in the lock files.

Fix

Updated the test to verify what still matters: the workflow-level COPILOT_MODEL env pin exists (e.g. claude-haiku-4.5 or o4-mini-aw), and the old deprecated ${{ vars.GH_AW_MODEL_AGENT_COPILOT || env.COPILOT_MODEL }} pattern is not used.

Testing

All 3,491 tests pass locally.

The lock files were recompiled in #5946 (gh-aw v0.82.2 upgrade), which changed the agent job model selection from ${{ env.COPILOT_MODEL }} to a vars-based expression. Update the test to verify the workflow-level env pin still exists (claude-haiku-4.5 / o4-mini-aw) instead of checking for the old ${{ env.COPILOT_MODEL }} reference in the agent job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-07-06T13:16:49Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	98.81%	98.81%	➡️ +0.00%
Statements	98.73%	98.73%	➡️ +0.00%
Functions	99.58%	99.58%	➡️ +0.00%
Branches	95.14%	95.10%	📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Updates the BYOK smoke-workflow lockfile test so it matches the newer gh-aw v0.82.2-compiled lockfile structure (workflow-level model pin + vars-based agent-job selection), preventing false failures after workflow recompilation.

Changes:

Adjusts the lockfile assertion to validate a pinned workflow-level COPILOT_MODEL value (rather than ${{ env.COPILOT_MODEL }}).
Keeps a guard that the deprecated ${{ vars.GH_AW_MODEL_AGENT_COPILOT || env.COPILOT_MODEL }} pattern is not used.

Show a summary per file

File	Description
scripts/ci/smoke-copilot-byok-workflow.test.ts	Updates assertions to align with the post-upgrade compiled lockfile model-selection patterns.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 1/1 changed files
Comments generated: 1
Review effort level: Low

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-07-06T13:17:56Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-07-06T13:19:04Z

✅ Smoke Claude passed

github-actions · 2026-07-06T13:19:05Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-07-06T13:19:05Z

✅ Smoke Gemini completed. All facets verified. 💎

Smoke test completed with FAIL status. GitHub MCP tools were missing.

github-actions · 2026-07-06T13:19:05Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-07-06T13:19:06Z

Chroot tests failed Smoke Chroot failed - See logs for details.

github-actions · 2026-07-06T13:19:06Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-07-06T13:19:06Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-07-06T13:19:07Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-07-06T13:19:07Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-07-06T13:19:10Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-07-06T13:19:11Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

github-actions · 2026-07-06T13:19:13Z

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

github-actions · 2026-07-06T13:19:14Z

✅ Contribution Check completed successfully!

PR #5950 follows the applicable CONTRIBUTING.md guidelines; no review comment needed.

github-actions · 2026-07-06T13:19:17Z

✅ Build Test Suite completed successfully!

github-actions · 2026-07-06T13:19:38Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	98.81%	98.81%	➡️ +0.00%
Statements	98.73%	98.73%	➡️ +0.00%
Functions	99.58%	99.58%	➡️ +0.00%
Branches	95.14%	95.10%	📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-07-06T13:22:18Z

Smoke Test: Claude Engine Validation

Check	Result
API Status	✅ PASS
GH Check	✅ PASS
File Status	✅ PASS

Overall Result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #5950 · 55.7 AIC · ⊞ 5.8K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:22:43Z

Smoke Test Results

Redis PING: ❌ Network unreachable
PostgreSQL pg_isready: ❌ No response
PostgreSQL SELECT 1: ❌ Network unreachable

Overall: FAIL — host.docker.internal (172.17.0.1) is unreachable from this runner.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:22:56Z

Smoke test results — Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅ 200
File write/read	✅

Overall: PASS — @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:23:05Z

Smoke Test Results

Test	Result
MCP Connectivity	✅
GitHub.com HTTP	✅ 200
File Write/Read	✅

Overall: PASS 🎉

Author: @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:23:15Z

Smoke Test: Copilot BYOK (Direct) — PASS

Test	Result
GitHub MCP (list PRs)	✅
GitHub.com connectivity (HTTP 200)	✅
File write/read	⚠️ template vars unresolved in agent
BYOK inference (api-proxy → api.githubcopilot.com)	✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com.
cc @lpcox — Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:23:19Z

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Details
1. Module Loading	✅ Pass	`otel.js` loads OK; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled` + internal helpers
2. Test Suite	✅ Pass	59 tests passed, 0 failed across `otel.test.js` + `otel-fanout.test.js`
3. Env Var Forwarding	✅ Pass	`api-proxy-env-config.ts` forwards `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID`, `OTEL_SERVICE_NAME`; `observability-environment.ts` passes all `OTEL_*` vars
4. Token Tracker Integration	✅ Pass	`onUsage` callback exists in `token-tracker-http.js` as the OTEL hook point
5. OTEL Diagnostics	✅ Expected	No spans exported — no OTEL endpoint configured (graceful degradation working)

All scenarios pass. OTEL tracing integration is functioning correctly.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:25:10Z

Smoke Test: Gemini Engine Validation

GitHub MCP Testing: ❌ (Tools missing, MCP issues detected)
GitHub.com Connectivity: ✅ (HTTP 200 via proxy)
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall Status: FAIL

Last 2 Merged PRs (via API):

chore: upgrade gh-aw to v0.82.2 pre-release and recompile workflows
Upgrade gh-aw to v0.82.2 pre-release and recompile workflows

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:25:12Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note: Java required a writable Maven local repository (-Dmaven.repo.local=/tmp/gh-aw/agent/m2-repo) because /home/runner/.m2 was owned by root in this environment.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #5950 · 64.9 AIC · ⊞ 6.9K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-07-06T13:34:04Z

Smoke Test

chore: upgrade gh-aw to v0.82.2 pre-release and recompile workflows
Upgrade gh-aw to v0.82.2 pre-release and recompile workflows
GitHub PR read: ✅
safeinputs-gh query: ❌ (tool unavailable)
Playwright GitHub title: ✅
File write/read: ✅
Discussion query/comment: ❌ (tool unavailable)
Build AWF: ❌
Overall: FAIL

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

awmgmcpg
registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

Copilot AI review requested due to automatic review settings July 6, 2026 13:14

Copilot started reviewing on behalf of lpcox July 6, 2026 13:15 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread scripts/ci/smoke-copilot-byok-workflow.test.ts Outdated

Potential fix for pull request finding

dd751d5

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

lpcox added the ready-for-aw label Jul 6, 2026

lpcox had a problem deploying to aoai-model July 6, 2026 13:19 — with GitHub Actions Failure

github-actions Bot added the smoke-claude label Jul 6, 2026

lpcox temporarily deployed to aoai-model July 6, 2026 13:22 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-pat label Jul 6, 2026

lpcox temporarily deployed to aoai-model July 6, 2026 13:23 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot label Jul 6, 2026

github-actions Bot mentioned this pull request Jul 6, 2026

[aw] No-Op Runs #5883

Open

github-actions Bot added the smoke-copilot-byok label Jul 6, 2026

This was referenced Jul 6, 2026

[aw] Smoke Copilot BYOK AOAI (Entra) failed #5951

Closed

[aw] Smoke Copilot BYOK AOAI (api-key) failed #5952

Closed

[aw] Smoke Chroot failed #5953

Closed

github-actions Bot added the build-test label Jul 6, 2026

lpcox merged commit 47e758f into main Jul 6, 2026
83 of 88 checks passed

lpcox deleted the fix/byok-workflow-test-expectations branch July 6, 2026 14:04

github-actions Bot mentioned this pull request Jul 6, 2026

fix: handle EPERM when chmod'ing pre-existing mcp-logs directory #5956

Merged

Uh oh!

Conversation

lpcox commented Jul 6, 2026

Summary

Problem

Fix

Testing

Uh oh!

github-actions Bot commented Jul 6, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented Jul 6, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Jul 6, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented Jul 6, 2026

Smoke Test: Copilot BYOK (Direct) — PASS

Uh oh!

github-actions Bot commented Jul 6, 2026

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading