Skip to content

fix: harden token tracker against unsupported response encodings and WS fragmentation#6003

Merged
lpcox merged 3 commits into
mainfrom
fix/token-tracker-response-parsing
Jul 8, 2026
Merged

fix: harden token tracker against unsupported response encodings and WS fragmentation#6003
lpcox merged 3 commits into
mainfrom
fix/token-tracker-response-parsing

Conversation

@lpcox

@lpcox lpcox commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

Intermittent token-usage.jsonl failures in Smoke Codex CI — multiple unrelated PRs failed the verify_token_usage step simultaneously, then runs immediately after succeeded. Root cause: transient upstream API behavior (likely content-encoding: zstd) that the token tracker couldn't decompress, causing silent data loss.

Evidence of transient failure (not code-related):

Run Branch Result
28907953012 fix-claude-sonnet-5-credits ❌ verify_token_usage
28908000014 refactor-split-server-js ❌ verify_token_usage
28908020313 refactor-split-proxy-request-module ✅ (1 min later)

Root Cause

Two gaps in the token tracker's response parsing:

  1. Unsupported compression encoding — If the upstream API responds with content-encoding: zstd, the tracker's isCompressedResponse() returned false (only knew gzip/deflate/br). Raw compressed bytes were then fed to the SSE parser as UTF-8 text → garbage → no usage extracted → token-usage.jsonl never written.

  2. WebSocket frame fragmentationparseWebSocketFrames() only extracted unfragmented text frames (FIN=1, opcode=1). Fragmented messages (FIN=0 initial frame + continuation frames) were consumed but payloads silently discarded.

Fixes

  1. Strip unsupported encodings from Accept-Encoding before forwarding to upstream APIs. This prevents them from responding with encodings the tracker cannot decompress. (proxy-utils.js, request-headers.js)

  2. Detect zstd in isCompressedResponse so the tracker recognizes it's compressed and bails early with a clear warning instead of parsing garbage. (token-parsers.js, token-tracker-http.js)

  3. Add info-level logging when usage extraction fails on 2xx responses, making the issue visible in CI without AWF_DEBUG_TOKENS. (token-tracker-http.js)

  4. Handle WebSocket frame fragmentation — reassemble fragmented text messages across continuation frames. (token-tracker-ws.js)

Testing

  • All 1429 api-proxy tests pass
  • All 3536 project tests pass
  • Added 15 new tests covering sanitizeAcceptEncoding and WS fragmentation reassembly

…WS fragmentation

Fixes intermittent token-usage.jsonl failures when upstream APIs (e.g.
OpenAI) respond with content-encoding values the tracker cannot decompress
(e.g. zstd), or when WebSocket messages are fragmented across frames.

Changes:
- Strip unsupported encodings (zstd, compress, etc.) from Accept-Encoding
  before forwarding to upstream APIs, preventing them from responding with
  encodings the tracker cannot parse
- Detect zstd in isCompressedResponse so compressed=true is set correctly,
  and bail early with a clear warning instead of parsing garbage bytes
- Add info-level logging when usage extraction fails on 2xx responses,
  making the root cause visible in CI without AWF_DEBUG_TOKENS
- Handle WebSocket frame fragmentation in parseWebSocketFrames — previously
  fragmented messages (FIN=0 + continuation frames) were silently dropped

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 8, 2026 01:35
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 98.92% 98.92% ➡️ +0.00%
Statements 98.87% 98.87% ➡️ +0.00%
Functions 99.44% 99.44% ➡️ +0.00%
Branches 95.39% 95.35% 📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the api-proxy token tracking pipeline to avoid intermittent token-usage.jsonl data loss caused by upstream responses using unsupported compression (notably content-encoding: zstd) and by fragmented WebSocket text messages.

Changes:

  • Sanitize forwarded Accept-Encoding to prevent upstream APIs from selecting encodings the tracker can’t inspect.
  • Detect unsupported zstd responses and skip tracking with explicit logging instead of parsing compressed bytes as UTF-8.
  • Reassemble fragmented WebSocket text messages across continuation frames, with new targeted unit tests.
Show a summary per file
File Description
containers/api-proxy/token-tracker-ws.js Adds fragmented WS text message reassembly support via a fragments accumulator.
containers/api-proxy/token-tracker-ws.fragmentation.test.js Adds coverage for fragmented WS message reassembly across frames and data events.
containers/api-proxy/token-tracker-http.js Logs missing-usage extraction on 2xx and warns/short-circuits on unsupported encodings.
containers/api-proxy/token-parsers.js Adds zstd detection plus isUnsupportedEncoding() export for fail-fast behavior.
containers/api-proxy/request-headers.js Applies sanitizeAcceptEncoding() to forwarded requests.
containers/api-proxy/proxy-utils.js Introduces sanitizeAcceptEncoding() implementation and supported-encoding allowlist.
containers/api-proxy/proxy-utils.accept-encoding.test.js Adds unit tests for Accept-Encoding sanitization behavior.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 7/7 changed files
  • Comments generated: 3
  • Review effort level: Low

Comment on lines +156 to +158
// Accumulates payloads from fragmented WebSocket text messages across frames
const fragments = [];

Comment on lines +288 to +297
function sanitizeAcceptEncoding(value) {
if (!value) return 'gzip, deflate, br';
const parts = value.split(',').map(p => p.trim()).filter(Boolean);
const supported = parts.filter(p => {
// Extract encoding name (strip quality value like ";q=0.5")
const encoding = p.split(';')[0].trim().toLowerCase();
return SUPPORTED_ENCODINGS.has(encoding);
});
return supported.length > 0 ? supported.join(', ') : 'gzip, deflate, br';
}
Comment on lines +18 to +20
it('handles zstd-only Accept-Encoding by returning defaults', () => {
expect(sanitizeAcceptEncoding('zstd')).toBe('gzip, deflate, br');
});
@lpcox

lpcox commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot address review feedback

Copilot AI commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

@copilot address review feedback

Done in the latest commit:

  1. token-tracker-ws.jsfragments.length = 0 is now cleared alongside buffer on MAX_WS_BUFFER overflow, keeping fragmentation state consistent with the dropped data.
  2. proxy-utils.js — When the client's Accept-Encoding contains only unsupported encodings (e.g. zstd), sanitizeAcceptEncoding now returns identity instead of gzip, deflate, br, avoiding advertising compression the client didn't request.
  3. proxy-utils.accept-encoding.test.js — Updated the zstd-only test case to expect identity.

Copilot finished work on behalf of lpcox July 8, 2026 01:45
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Security Guard failed. Please review the logs for details.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

Smoke test finished. Results: MCP ❌, Connectivity ❌, File Write ✅, Bash ✅. Overall FAIL.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

PR #6003 follows the applicable CONTRIBUTING.md guidelines: clear description, tests included for new functionality, relevant files organized under containers/api-proxy, and no missing documentation requirement identified from the pre-fetched diff.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 98.92% 98.92% ➡️ +0.00%
Statements 98.87% 98.87% ➡️ +0.00%
Functions 99.44% 99.44% ➡️ +0.00%
Branches 95.39% 95.35% 📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions github-actions Bot mentioned this pull request Jul 8, 2026
@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Smoke test completed; PR label added, but PR comment could not be posted because the safe-output bridge reported no workflow event context.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Security Guard failed. Please review the logs for details.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Build Test Failed Build Test Suite - See logs for details

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 98.92% 98.92% ➡️ +0.00%
Statements 98.87% 98.87% ➡️ +0.00%
Functions 99.44% 99.44% ➡️ +0.00%
Branches 95.39% 95.35% 📉 -0.04%

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Result
API status ✅ PASS
gh check ✅ PASS
File status ✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #6003 · 55.3 AIC · ⊞ 3.3K ·
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode - PASS

• GitHub MCP connectivity ✅
• GitHub.com reachability ✅ (HTTP 200)
• File I/O test ✅
• BYOK inference ✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test Results

Test Status
GitHub MCP connectivity N/A - pre-step data unavailable
GitHub.com HTTP N/A - template vars unresolved
File write/read N/A - template vars unresolved

Note: Pre-step outputs were not substituted (smoke-data step did not produce outputs).

Overall: INCONCLUSIVE - Cannot validate test results without pre-step data.

/cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Services Connectivity

  • Redis PING: ❌ (Network is unreachable)
  • PostgreSQL pg_isready: ❌ (no response)
  • PostgreSQL SELECT 1: ❌ (Network is unreachable)

Overall: FAILhost.docker.internal (172.17.0.1) is unreachable. Services may not be running or network routing is not configured.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot PAT Auth ✅ PASS

Test Result
GitHub MCP connectivity ✅ (PR data pre-fetched)
github.com HTTP ✅ 200
File write/read ✅ Content verified

Auth mode: PAT (COPILOT_GITHUB_TOKEN)
Author: @lpcox

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Gemini Engine Validation

  • Review last 2 merged PRs: ❌ (Cannot reach GitHub API)
  • GitHub.com Connectivity: ❌ (Connection failed)
  • File Writing Testing: ✅
  • Bash Tool Testing: ✅

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

@lpcox

  • GitHub MCP Testing: ✅
  • GitHub.com Connectivity: ✅
  • File Write/Read Test: ✅
  • BYOK Inference Test: ✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3 ❌ NO
Node.js v24.18.0 v22.23.1 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ Not all tests passed. Python and Node.js versions differ between host and chroot environments.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Result Notes
1. Module Loading ✅ Pass otel.js loads successfully; exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, plus internal helpers
2. Test Suite ✅ Pass 59/59 OTEL tests pass across 2 suites (otel.test.js, otel-fanout.test.js). Non-OTEL failures (13) are pre-existing timeouts in OIDC provider tests, unrelated.
3. Env Var Forwarding ✅ Pass src/services/api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME to api-proxy container
4. Token Tracker Integration ✅ Pass onUsage callback present in token-tracker-http.js — OTEL hook point confirmed
5. OTEL Diagnostics ✅ Pass Graceful degradation confirmed: spans fall back to /var/log/api-proxy/otel.jsonl when no endpoint configured

Overall: All scenarios pass ✅

Warning

Firewall blocked 4 domains

The following domains were blocked by the firewall during workflow execution:

  • 127.0.0.1
  • api.example.com
  • api.openai.com
  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "127.0.0.1"
    - "api.example.com"
    - "api.openai.com"
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@lpcox lpcox merged commit 42d0df3 into main Jul 8, 2026
84 of 88 checks passed
@lpcox lpcox deleted the fix/token-tracker-response-parsing branch July 8, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants