fix: harden token tracker against unsupported response encodings and WS fragmentation#6003
Conversation
…WS fragmentation Fixes intermittent token-usage.jsonl failures when upstream APIs (e.g. OpenAI) respond with content-encoding values the tracker cannot decompress (e.g. zstd), or when WebSocket messages are fragmented across frames. Changes: - Strip unsupported encodings (zstd, compress, etc.) from Accept-Encoding before forwarding to upstream APIs, preventing them from responding with encodings the tracker cannot parse - Detect zstd in isCompressedResponse so compressed=true is set correctly, and bail early with a clear warning instead of parsing garbage bytes - Add info-level logging when usage extraction fails on 2xx responses, making the root cause visible in CI without AWF_DEBUG_TOKENS - Handle WebSocket frame fragmentation in parseWebSocketFrames — previously fragmented messages (FIN=0 + continuation frames) were silently dropped Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 98.92% | 98.92% | ➡️ +0.00% |
| Statements | 98.87% | 98.87% | ➡️ +0.00% |
| Functions | 99.44% | 99.44% | ➡️ +0.00% |
| Branches | 95.39% | 95.35% | 📉 -0.04% |
Coverage comparison generated by scripts/ci/compare-coverage.ts
There was a problem hiding this comment.
Pull request overview
This PR hardens the api-proxy token tracking pipeline to avoid intermittent token-usage.jsonl data loss caused by upstream responses using unsupported compression (notably content-encoding: zstd) and by fragmented WebSocket text messages.
Changes:
- Sanitize forwarded
Accept-Encodingto prevent upstream APIs from selecting encodings the tracker can’t inspect. - Detect unsupported
zstdresponses and skip tracking with explicit logging instead of parsing compressed bytes as UTF-8. - Reassemble fragmented WebSocket text messages across continuation frames, with new targeted unit tests.
Show a summary per file
| File | Description |
|---|---|
| containers/api-proxy/token-tracker-ws.js | Adds fragmented WS text message reassembly support via a fragments accumulator. |
| containers/api-proxy/token-tracker-ws.fragmentation.test.js | Adds coverage for fragmented WS message reassembly across frames and data events. |
| containers/api-proxy/token-tracker-http.js | Logs missing-usage extraction on 2xx and warns/short-circuits on unsupported encodings. |
| containers/api-proxy/token-parsers.js | Adds zstd detection plus isUnsupportedEncoding() export for fail-fast behavior. |
| containers/api-proxy/request-headers.js | Applies sanitizeAcceptEncoding() to forwarded requests. |
| containers/api-proxy/proxy-utils.js | Introduces sanitizeAcceptEncoding() implementation and supported-encoding allowlist. |
| containers/api-proxy/proxy-utils.accept-encoding.test.js | Adds unit tests for Accept-Encoding sanitization behavior. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 7/7 changed files
- Comments generated: 3
- Review effort level: Low
| // Accumulates payloads from fragmented WebSocket text messages across frames | ||
| const fragments = []; | ||
|
|
| function sanitizeAcceptEncoding(value) { | ||
| if (!value) return 'gzip, deflate, br'; | ||
| const parts = value.split(',').map(p => p.trim()).filter(Boolean); | ||
| const supported = parts.filter(p => { | ||
| // Extract encoding name (strip quality value like ";q=0.5") | ||
| const encoding = p.split(';')[0].trim().toLowerCase(); | ||
| return SUPPORTED_ENCODINGS.has(encoding); | ||
| }); | ||
| return supported.length > 0 ? supported.join(', ') : 'gzip, deflate, br'; | ||
| } |
| it('handles zstd-only Accept-Encoding by returning defaults', () => { | ||
| expect(sanitizeAcceptEncoding('zstd')).toBe('gzip, deflate, br'); | ||
| }); |
|
@copilot address review feedback |
… unsupported-only Accept-Encoding
Done in the latest commit:
|
|
✅ Copilot review passed with no inline comments. @lpcox Add the |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓 |
|
✅ Build Test Suite completed successfully! |
|
❌ Security Guard failed. Please review the logs for details. |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
✅ Smoke Gemini completed. All facets verified. 💎 Smoke test finished. Results: MCP ❌, Connectivity ❌, File Write ✅, Bash ✅. Overall FAIL. |
|
✅ Contribution Check completed successfully! PR #6003 follows the applicable CONTRIBUTING.md guidelines: clear description, tests included for new functionality, relevant files organized under containers/api-proxy, and no missing documentation requirement identified from the pre-fetched diff. |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
✅ Smoke Claude passed |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 98.92% | 98.92% | ➡️ +0.00% |
| Statements | 98.87% | 98.87% | ➡️ +0.00% |
| Functions | 99.44% | 99.44% | ➡️ +0.00% |
| Branches | 95.39% | 95.35% | 📉 -0.04% |
Coverage comparison generated by scripts/ci/compare-coverage.ts
This comment has been minimized.
This comment has been minimized.
|
✅ Smoke Claude passed |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed... |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 Smoke test completed; PR label added, but PR comment could not be posted because the safe-output bridge reported no workflow event context. |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
❌ Security Guard failed. Please review the logs for details. |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
Build Test Failed Build Test Suite - See logs for details |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 98.92% | 98.92% | ➡️ +0.00% |
| Statements | 98.87% | 98.87% | ➡️ +0.00% |
| Functions | 99.44% | 99.44% | ➡️ +0.00% |
| Branches | 95.39% | 95.35% | 📉 -0.04% |
Coverage comparison generated by scripts/ci/compare-coverage.ts
Smoke Test: Claude Engine Validation
Overall result: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
✅ Smoke Test: Copilot BYOK (Direct) Mode - PASS • GitHub MCP connectivity ✅ Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test Results
Note: Pre-step outputs were not substituted (smoke-data step did not produce outputs). Overall: INCONCLUSIVE - Cannot validate test results without pre-step data. /cc @lpcox Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Services Connectivity
Overall: FAIL — Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Copilot PAT Auth ✅ PASS
Auth mode: PAT (COPILOT_GITHUB_TOKEN) Overall: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Gemini Engine Validation
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) Overall: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Chroot Version Comparison Results
Overall: ❌ Not all tests passed. Python and Node.js versions differ between host and chroot environments. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: API Proxy OpenTelemetry Tracing
Overall: All scenarios pass ✅ Warning Firewall blocked 4 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "127.0.0.1"
- "api.example.com"
- "api.openai.com"
- "awmgmcpg"See Network Configuration for more information.
|
Problem
Intermittent
token-usage.jsonlfailures in Smoke Codex CI — multiple unrelated PRs failed theverify_token_usagestep simultaneously, then runs immediately after succeeded. Root cause: transient upstream API behavior (likelycontent-encoding: zstd) that the token tracker couldn't decompress, causing silent data loss.Evidence of transient failure (not code-related):
Root Cause
Two gaps in the token tracker's response parsing:
Unsupported compression encoding — If the upstream API responds with
content-encoding: zstd, the tracker'sisCompressedResponse()returnedfalse(only knew gzip/deflate/br). Raw compressed bytes were then fed to the SSE parser as UTF-8 text → garbage → no usage extracted →token-usage.jsonlnever written.WebSocket frame fragmentation —
parseWebSocketFrames()only extracted unfragmented text frames (FIN=1, opcode=1). Fragmented messages (FIN=0 initial frame + continuation frames) were consumed but payloads silently discarded.Fixes
Strip unsupported encodings from
Accept-Encodingbefore forwarding to upstream APIs. This prevents them from responding with encodings the tracker cannot decompress. (proxy-utils.js,request-headers.js)Detect zstd in
isCompressedResponseso the tracker recognizes it's compressed and bails early with a clear warning instead of parsing garbage. (token-parsers.js,token-tracker-http.js)Add info-level logging when usage extraction fails on 2xx responses, making the issue visible in CI without
AWF_DEBUG_TOKENS. (token-tracker-http.js)Handle WebSocket frame fragmentation — reassemble fragmented text messages across continuation frames. (
token-tracker-ws.js)Testing
sanitizeAcceptEncodingand WS fragmentation reassembly