fix(api-proxy): stop alias fallback picking arbitrary models by davidslater · Pull Request #6996 · github/gh-aw-firewall

davidslater · 2026-08-06T22:28:58Z

Created by GitHub Ace · View Session

Problem

Alias resolution could resolve to an internal provider staging model that nobody configured. In gh-aw-threat-detection run 31034552109, the detection alias resolved to crest-alpha-0418-block-cy4.5.

The chain is gh-aw's built-in table, not user config:

detection -> [small] -> [mini] -> [haiku, gpt-5-mini, gpt-5-nano, gemini-flash-lite]
haiku     -> [copilot/*haiku*, anthropic/*haiku*]

On an OpenAI-only proxy, haiku and gemini-flash-lite are scoped to other providers and legitimately match nothing. Rather than contributing no candidates, each triggered middle-power fallback, which could not infer a family prefix, fell back to the entire ~220-model live catalog, tiered it (only gpt-5/gpt-4/gpt-3.5 get real tiers, so ~200 staging entries tie at tier 1), and took the alphabetical median. compareByVersion then ranked that synthesized pick ahead of the two legitimate candidates the same fan-out had already found.

Measured against a representative catalog, 52 of the 59 built-in aliases resolved to a staging model — including every meta-alias (agent, auto, large, small, summarization) and the engine aliases. detection is simply the one that ran.

Changes

Synthesized candidates no longer out-rank genuine matches. _resolveAliasPatterns now tracks candidates from a nested alias's fallback separately, and only uses them when no sibling pattern matched anything. The returned fallback.activated flag reflects whether the winner was synthesized.

Provider-mismatched nested aliases are skipped. The fallback gate checked patterns.some(p => p.includes('/')) — "some pattern names a provider" — instead of "some pattern names the current provider". Scoped deliberately to nested references: a directly requested model keeps today's graceful-degradation behaviour, which is intentional and covered by an existing test (should fall back when provider patterns do not match current provider).

Soft price filter on the fallback pool. selectMiddlePowerFallback accepts an optional isModelPriceable predicate, injected from model-config.js and active only when the AI-credits guard is (a credit cap set, no configured default pricing). Applied only to this synthesized-selection path — explicitly requested and pattern-matched models are untouched. If filtering would empty the pool it uses the unfiltered pool, and a throwing predicate is treated as priceable, so it can never turn a success into a failure.

Verification

Against the real gh-aw alias table and a synthetic 208-model catalog:

	staging-model resolutions
before	52 / 59
after (fan-out fix)	43 / 59
after (+ price filter)	6 / 59

detection now resolves to gpt-5-mini, as intended.

Full api-proxy suite: 1574 passed, no regressions
9 new regression tests covering the fan-out, the nested/top-level distinction, and the price filter's soft-failure modes
npm run build clean; verified no circular dependency from the new model-config.js → ai-credits-guard import

Deliberately not included

Ordered-list semantics. gh-aw documents patterns as "tried in sequence until one resolves", which AWF does not implement. This looked like the highest-coverage fix, but it regresses sonnet-6x from claude-sonnet-5 to claude-sonnet-4.5 — so the built-in table is not consistently written assuming priority order. That is a cross-repo contract ambiguity needing gh-aw's input.

Whether middle_power should be on by default. It currently fabricates a model whenever an alias misses, and nobody opted in.

Whether "median capability" suits a mini/small alias. Price-filtering can yield gpt-4-turbo ($10/$30 per 1M) where the alias asked for gpt-5-nano ($0.05/$0.40) — trading a hard failure for a silent ~100× cost increase. A cheapest_priced strategy may be more appropriate.

Known residual

Six aliases can still reach a staging model: any, agent, copilot, claude, codex, gemini. All bottom out in any -> [copilot/*, anthropic/*, openai/*, ...], whose openai/* wildcard genuinely matches everything. That is compareByVersion ranking rather than fallback, so it is out of scope here and warrants a separate issue.

Caveat

The 220-model catalog used for measurement is synthetic, matching the shape reported from the run; the affected set is a property of the alias table rather than the catalog, but exact picks vary. I could not read the run's artifacts directly, so the assumption that the failure surfaced via checkUnknownModelRejection is unverified — the run's models.json artifact would confirm.

A nested alias scoped to other providers (e.g. "haiku" on an OpenAI-only proxy) has a legitimately empty candidate set, but instead of contributing nothing it triggered middle-power fallback, which synthesized a pick from the entire live-discovered catalog. That guess then out-ranked the genuine siblings the same fan-out had already found. - Track fallback-derived candidates separately from genuine pattern matches and only use them when no sibling pattern matched anything. - Skip fallback for nested aliases that do not name the current provider. Scoped to nested references only; a directly requested model retains the existing graceful-degradation behaviour. - Soft price-filter the middle-power pool via an injected predicate, gated on the AI-credits guard being active. Falls back to the unfiltered pool when filtering would empty it. Refs #6993 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>

Copilot

Pull request overview

Fixes alias fallback so synthesized models do not outrank genuine provider matches.

Changes:

Separates synthesized and genuine alias candidates.
Skips provider-mismatched nested fallbacks.
Soft-filters fallback candidates by pricing availability.
Adds regression tests.

Show a summary per file

File	Description
`containers/api-proxy/model-resolver.js`	Prioritizes genuine alias matches.
`containers/api-proxy/model-fallback.js`	Adds optional pricing filtering.
`containers/api-proxy/model-config.js`	Connects pricing checks to fallback configuration.
`containers/api-proxy/model-resolver.test.js`	Adds fan-out and pricing regressions.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 2
Review effort level: Balanced

Addresses PR review feedback. The fallback price filter used checkUnknownModelRejection, which is not a predicate: for every unpriceable candidate it reached resolveLowerPriorityPricing, emitting unknown_model_ai_credits_pricing and recording the model in warnedUnknownModels. Verified this emitted one warning per probed candidate (50/50 in a reproduction) and then suppressed the warning when such a model was genuinely requested later. Adds a quiet option to the pricing resolution path and exports a pure isModelPriceable(), used for speculative candidate filtering. The real request path keeps warning exactly as before. Also splits the mis-named fan-out test: the original asserted the all-mismatched null case under a name describing synthesized activation, leaving the synthesized-only branch uncovered. Now covers all three cases — all-mismatched returns null, synthesized-only activates fallback, and a genuine sibling match beats a synthesized one. Refs #6993 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>

github-actions · 2026-08-06T23:09:59Z

✅ Copilot review passed with no inline comments.

@davidslater Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-08-06T23:12:56Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	98.90%	98.92%	📈 +0.02%
Statements	98.79%	98.81%	📈 +0.02%
Functions	99.10%	99.10%	➡️ +0.00%
Branches	94.99%	95.01%	📈 +0.02%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-08-06T23:48:20Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-08-06T23:48:20Z

Build Test Failed Build Test Suite - See logs for details

github-actions · 2026-08-06T23:48:20Z

✅ Smoke Claude passed

github-actions · 2026-08-06T23:48:21Z

🛡️ Smoke Copilot Network Isolation reports failed while checking network isolation. Investigate the egress model.

github-actions · 2026-08-06T23:48:22Z

❌ Security Guard failed. Please review the logs for details.

github-actions · 2026-08-06T23:48:23Z

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

github-actions · 2026-08-06T23:48:23Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-08-06T23:48:23Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

github-actions · 2026-08-06T23:48:24Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-08-06T23:48:24Z

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-08-06T23:48:26Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-08-06T23:48:28Z

✅ Smoke Gemini completed. All facets verified. 💎

github-actions · 2026-08-06T23:48:31Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-08-06T23:48:33Z

✅ Contribution Check completed successfully!

PR #6996 follows the contribution guidelines: the change is in the correct container/source paths, includes substantial regression tests, and the PR description is clear with a related issue reference. No contribution-guideline issues found.

github-actions · 2026-08-06T23:49:17Z

Chroot tests failed Smoke Chroot failed - See logs for details.

github-actions · 2026-08-07T01:21:26Z

✅ Build Test Suite completed successfully!

github-actions · 2026-08-07T01:21:42Z

✅ Smoke Gemini completed. All facets verified. 💎

github-actions · 2026-08-07T01:23:16Z

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (github.com) reachable: allowed=200
✅ Blocked domain (example.com) denied: CONNECT tunnel failed, response 403

Overall: PASS @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:23:17Z

Smoke Test: Copilot Engine (@lpcox)

✅ GitHub MCP: PR chore: upgrade gh-aw to v0.85.1 pre-release and recompile workflows #6985 "chore: upgrade gh-aw to v0.85.1 pre-release and recompile workflows", PR fix: propagate config fields to all layers #6970 "fix: propagate config fields to all layers"
✅ GitHub.com connectivity (HTTP 200)
✅ File write/read test

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:23:17Z

Smoke Test: Claude Engine Validation

Check	Status
API	✅ PASS
GH CLI	✅ PASS
File	✅ PASS

Overall result: PASS

Generated by Smoke Claude for #6996 · haiku45 · 55.8 AIC · ⊞ 3.6K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:23:23Z

Smoke Test: Copilot BYOK (Direct) Mode

Recent merged PRs (MCP check):

PR chore: upgrade gh-aw to v0.85.1 pre-release and recompile workflows #6985: chore: upgrade gh-aw to v0.85.1 pre-release and recompile workflows
PR fix: propagate config fields to all layers #6970: fix: propagate config fields to all layers
✅ GitHub MCP connectivity
✅ GitHub.com connectivity (HTTP 200)
✅ File write/read test
✅ BYOK inference test

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

Overall: PASS

cc @lpcox

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:23:39Z

Smoke Test Results: Services Connectivity

Redis: ❌ (name resolution failure)
PostgreSQL pg_isready: ❌ (no response)
PostgreSQL SELECT 1: ❌ (name resolution failure)

Overall: FAIL — host.docker.internal could not be resolved (Temporary failure in name resolution) from the AWF sandbox.

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:24:15Z

Smoke test results: MCP ✅, Connectivity ❌, Writing ✅, Bash ✅. Overall: FAIL.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:24:51Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A (ran, "Hello, World!")	✅ PASS
.NET	json-parse	✅	N/A (ran, produced expected JSON output)	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	❌	0/0	❌ FAIL
Java	caffeine	❌	0/0	❌ FAIL
Node.js	clsx	✅	pass (all tests)	✅ PASS
Node.js	execa	✅	pass (all tests)	✅ PASS
Node.js	p-limit	✅	pass (all tests)	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 7/8 ecosystems passed — FAIL

Error Details

Java (gson, caffeine) — mvn compile failed with:

[ERROR] Plugin org.apache.maven.plugins:maven-resources-plugin:3.4.0 or one of its dependencies could not be resolved:
[ERROR] The following artifacts could not be resolved: org.apache.maven.plugins:maven-resources-plugin:pom:3.4.0 (absent): Could not transfer artifact org.apache.maven.plugins:maven-resources-plugin:pom:3.4.0 from/to central (https://repo.maven.apache.org/maven2): Network is unreachable

The Maven proxy in ~/.m2/settings.xml points to the squid-proxy hostname (per the AWF sandbox network), but this hostname does not resolve/is not reachable in the current test execution context, so Maven cannot reach repo.maven.apache.org for required plugin dependencies. This indicates the runner was not executed inside the actual AWF-wrapped sandbox network where squid-proxy is a live alias — it is an environment/connectivity limitation rather than a code defect in the projects themselves.

Generated by Build Test Suite for #6996 · auto · 47.3 AIC · ⊞ 11.6K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:25:17Z

📡 OTel Tracing Smoke Test Results

✅ Scenario 1 — Module Loading: otel.js loaded successfully. isEnabled: true. Exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, _provider, _ProxyAwareOtlpExporter, _FileSpanExporter, _FanOutSpanExporter, _parseEndpoints, _parseOtlpHeaders, _buildResourceSpans, _createOtlpWorkloadIdentity.
✅ Scenario 2 — Test Suite: otel.test.js, otel-fanout.test.js, otel-workload-identity.test.js — 3 suites / 68 tests, all passed.
✅ Scenario 3 — Env Var Forwarding: src/services/agent-environment/env-passthrough.ts forwards GITHUB_AW_OTEL_TRACE_ID and GITHUB_AW_OTEL_PARENT_SPAN_ID; src/services/api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, plus the trace/parent-span context vars.
✅ Scenario 4 — Token Tracker Integration: token-tracker-http.js has an onUsage callback hook (invoked with normalized usage + model) as the OTEL integration point.
✅ Scenario 5 — OTEL Diagnostics: /tmp/gh-aw/otel.jsonl contains an exported span (gh-aw.agent.setup) with gen_ai.system and workflow/run resource attributes, confirming OTLP export executed during this run.

All scenarios pass — no regression detected in OTel tracing integration.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:25:49Z

Chroot Version Comparison

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.13	✅ YES
Node.js	v24.18.0	v22.23.2	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall: FAILED (ALL_TESTS_PASSED=false) — Node.js version differs between host and chroot environment. smoke-chroot label not applied.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:26:34Z

Smoke Test: Docker Sbx

✅ GitHub MCP connectivity (PR chore: upgrade gh-aw to v0.85.1 pre-release and recompile workflows #6985, fix: propagate config fields to all layers #6970 verified)
✅ GitHub.com connectivity (HTTP 200)
✅ File write/read test

Overall: PASS

cc @lpcox

📰 BREAKING: Report filed by Smoke Docker Sbx
_{Add label ready-for-aw to run again}

github-actions · 2026-08-07T01:26:41Z

Smoke test results:

Last 2 merged PRs: ✅
GH CLI query: ✅
GitHub page title: ✅
File write/read: ✅
npm ci && npm run build: ❌

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

Copilot AI balanced review requested due to automatic review settings August 6, 2026 22:28

Copilot started reviewing on behalf of davidslater August 6, 2026 22:37 View session

Copilot AI reviewed Aug 6, 2026

View reviewed changes

Comment thread containers/api-proxy/model-config.js Outdated

Comment thread containers/api-proxy/model-resolver.test.js Outdated

lpcox added the ready-for-aw label Aug 6, 2026

lpcox had a problem deploying to aoai-model August 6, 2026 23:48 — with GitHub Actions Failure

lpcox temporarily deployed to aoai-model August 6, 2026 23:51 — with GitHub Actions Inactive

This comment has been minimized.

Sign in to view

github-actions Bot added the smoke-copilot-byok label Aug 6, 2026

This comment has been minimized.

Sign in to view

github-actions Bot added the smoke-copilot label Aug 6, 2026

lpcox had a problem deploying to aoai-model August 7, 2026 01:21 — with GitHub Actions Failure

github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Aug 7, 2026

lpcox temporarily deployed to aoai-model August 7, 2026 01:23 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Aug 7, 2026

[aw] Contribution Check hit engine rate limit (HTTP 429) #7008

Closed

github-actions Bot mentioned this pull request Aug 7, 2026

[aw] Smoke Codex is missing required tool #7009

Closed

lpcox merged commit 6d2c00a into main Aug 7, 2026
138 of 141 checks passed

lpcox deleted the ace/01KZBZ1DMBD7VFBC5YYJHKY7Y6 branch August 7, 2026 01:41

Uh oh!

Conversation

davidslater commented Aug 6, 2026

Problem

Changes

Verification

Deliberately not included

Known residual

Caveat

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading