Skip to content

Fix streaming pipeline: reasoning parser support, context limits, and…#173

Merged
FranardoHuang merged 1 commit intomainfrom
updated_model
Mar 12, 2026
Merged

Fix streaming pipeline: reasoning parser support, context limits, and…#173
FranardoHuang merged 1 commit intomainfrom
updated_model

Conversation

@bruceyxli
Copy link
Collaborator

… query reformulation

  • base_handler: robust reasoning_content reading via model_extra fallback ('reasoning'/'reasoning_content'); fix flush block to recompute channels from accumulated text; update previous_channels after flush
  • model_call: reduce max_tokens to 2000 to stay within 10000 context limit; add thinking_budget: 512 to sampling params
  • parser: strip tag from analysis channel in streaming/incomplete case for consistency; add all partial suffixes to incomplete_patterns; handle vLLM stripping opening token
  • reformulation: singleton AsyncOpenAI client to fix connection pool leak; remove garbage strings from prompt; add timeout, reasoning fallback, and user_message fallback
  • start_vllm_servers: update model path, GPU memory utilizations

… query reformulation

- base_handler: robust reasoning_content reading via model_extra fallback ('reasoning'/'reasoning_content'); fix flush block to recompute channels from accumulated text; update previous_channels after flush
- model_call: reduce max_tokens to 2000 to stay within 10000 context limit; add thinking_budget: 512 to sampling params
- parser: strip <think> tag from analysis channel in streaming/incomplete case for consistency; add all partial </think> suffixes to incomplete_patterns; handle vLLM stripping opening <think> token
- reformulation: singleton AsyncOpenAI client to fix connection pool leak; remove garbage strings from prompt; add timeout, reasoning fallback, and user_message fallback
- start_vllm_servers: update model path, GPU memory utilizations

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@FranardoHuang FranardoHuang merged commit 6a759d5 into main Mar 12, 2026
1 check failed
@bruceyxli bruceyxli deleted the updated_model branch March 12, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants