Skip to content

fix: buffer non-streaming upstream responses for currency e2e demo#952

Draft
Spherrrical wants to merge 1 commit into
mainfrom
musa/fix-currency-e2e-e292
Draft

fix: buffer non-streaming upstream responses for currency e2e demo#952
Spherrrical wants to merge 1 commit into
mainfrom
musa/fix-currency-e2e-e292

Conversation

@Spherrrical
Copy link
Copy Markdown
Collaborator

Problem

The e2e-demo-currency CI job fails on advanced/currency_exchange hurl tests even though the demo flow mostly works. Logs show:

  • llm_gateway: upstream response parse error: JSON parsing error: EOF while parsing a string on non-streaming OpenAI responses (body truncated mid-JSON)
  • prompt_gateway: response body empty, chunk_start: 0, chunk_size: 0 during streaming
  • Hurl: transfer closed with outstanding read data remaining on the chat completions request

Root cause

Both WASM filters parsed non-streaming upstream bodies on the first response chunk. When Envoy delivered the JSON across multiple chunks, partial bodies were forwarded/parsed, causing JSON EOF errors and broken client streams.

Fix

  • llm_gateway: Accumulate non-streaming response chunks (suppress intermediate chunks) and parse only when the stream ends. Flush any remaining SSE processor/buffer bytes on a final empty EOS chunk for streaming responses.
  • prompt_gateway: Same buffering for non-streaming tool-call responses that enrich metadata before returning to the client.
  • Test runner: Add advanced/currency_exchange to the hurl retry list (3 attempts), matching preference_based_routing, to tolerate transient external API flakiness.

Testing

  • cargo +1.93.0 clippy -p llm_gateway -p prompt_gateway -- -D warnings
  • cargo +1.93.0 test -p llm_gateway -p prompt_gateway --lib
  • cargo +1.93.0 build --release --target=wasm32-wasip1 -p llm_gateway -p prompt_gateway

Full e2e requires CI (e2e-demo-currency) with API keys and the plano Docker image rebuild.

Open in Web Open in Cursor 

Co-authored-by: Musa <musa@spherrrical.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants