LCORE-3582: fix compacted-mode 500s in the agent pipeline - #2451
LCORE-3582: fix compacted-mode 500s in the agent pipeline#2451max-svistunov wants to merge 1 commit into
Conversation
Once a conversation compacted, every subsequent request on it failed with HTTP 500 on both /v1/query and /v1/streaming_query, permanently bricking the conversation. Root cause: in compacted mode (LCORE-1572) CompactionResult.params.input is an explicit item list (summaries + recent verbatim turns + new query) with the conversation parameter omitted, but the pydantic-ai agent pipeline did prompt = cast(str, responses_params.input) and handed the list to agent.run(), which dies client-side before any request reaches Llama Stack. The A2A executor had a quieter variant of the same gap: it passes the raw user text as the prompt, so compacted A2A turns silently lost all conversation context. The fix makes the explicit input reach the wire through the existing params-to-model-settings seam: - _model_settings_from_responses_params (pydantic_ai_lightspeed/ llamastack/_model.py): when omit_conversation is set and input is an item list, the dumped list is added to extra_body. The OpenAI SDK merges extra_body into the request body with precedence (_merge_mappings: "the second mapping takes precedence"), so the explicit list replaces the prompt-derived input — the request body matches what the non-agent /v1/responses path sends. This also fixes the A2A context loss with no a2a.py changes, since build_agent already routes params through this seam. - OgxResponsesModel gains _prepare_compacted_input, applied in both request() and request_stream() after the conversation-continuation trim: once a ModelResponse exists in the message history (a client-side tool-loop continuation), the input override is dropped so pydantic-ai's mapped messages — which carry the tool results — win. - New agent_prompt_text() helper (utils/conversation_compaction.py) replaces the cast(str, ...) at all four call sites (non-streaming and streaming, plain and multimodal): returns input unchanged when it is a string, else the text of the trailing message item of the explicit list. The prompt still drives capabilities and multimodal input construction; the wire input comes from the override. - The blocked-moderation path in retrieve_agent_response now skips append_turn_items_to_conversation when omit_conversation is set, mirroring the streaming path — appending the full explicit list would duplicate summaries and history into the conversation. - map_agent_inference_error now logs the original exception at error level with the traceback before mapping. Previously the mapped HTTPException discarded it and callers raised without logging, so these failures produced a generic 500 with nothing in the logs — which is what made this bug expensive to diagnose. Verified live against llama-stack 0.6.0: on a compacted conversation, turn after turn returns 200 on both endpoints (previously 500), the streaming path emits compaction/token/end events, and the model correctly answers questions about pre-compaction turns, proving the summary context reaches the model. Known limitation: image attachments are not folded into the overridden explicit input, so a compacted turn with images sends the text-only list (compaction+images previously hard -failed; noted in LCORE-3582). Unit tests cover the extra_body override (present in compacted mode, absent otherwise), the tool-loop guard, agent_prompt_text, the prompt threading through both retrieve paths, the moderation-append guard, and the error logging.
|
Warning Review limit reached
Next review available in: 32 minutes Limit details: You’ve used all 1 included review currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (9)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Description
Fix LCORE-3582: once a conversation compacted (LCORE-1572), every subsequent request on it failed with HTTP 500 on both
/v1/queryand/v1/streaming_query, permanently bricking the conversation. In compacted modeCompactionResult.params.inputis an explicit item list (summaries + recent turns + new query) with theconversationparameter omitted, but the pydantic-ai agent pipeline didprompt = cast(str, responses_params.input)and handed the list toagent.run(), which dies client-side before any request reaches Llama Stack. The A2A executor had a quieter variant: it passes raw user text as the prompt, so compacted A2A turns silently lost all conversation context.The fix routes the explicit input through the existing params→model-settings seam:
_model_settings_from_responses_params(src/pydantic_ai_lightspeed/llamastack/_model.py): in compacted mode the dumped input list is added toextra_body, which the OpenAI SDK merges into the request body with precedence — the wire request matches what the non-agent/v1/responsespath sends. This also fixes the A2A context loss with noa2a.pychanges.OgxResponsesModel._prepare_compacted_input(applied inrequest()andrequest_stream()): drops the override once aModelResponseexists in the message history, so client-side tool-loop iterations keep pydantic-ai's mapped messages (which carry tool results).agent_prompt_text()(src/utils/conversation_compaction.py) replacescast(str, ...)at all four call sites: returns string input unchanged, else the text of the trailing message item.retrieve_agent_responsenow skipsappend_turn_items_to_conversationin compacted mode (mirrors the streaming path), preventing summary/history duplication into the conversation.map_agent_inference_errornow logs the original exception with traceback at error level — previously these failures produced a generic 500 with nothing in the logs.Known limitation (noted on the Jira): image attachments are not folded into the overridden explicit input, so a compacted turn with images sends the text-only explicit list (before this fix, compaction+images hard-failed with 500).
Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Start the local stack with compaction configured to trigger aggressively:
plus a sqlite
conversation_cache, and setOTEL_ANONYMIZATION_SECRETin the environment.Send a first
/v1/query(new conversation), then follow-up queries with the returnedconversation_id.Expected: every turn returns 200 (turn 2+ previously returned the generic 500).
Actual (verified live against llama-stack 0.6.0):
Connect to
/v1/streaming_queryon the compacted conversation.Expected:
compactionevent, token stream,endevent (previously anerrorevent with status 500).Actual:
Run the tests specific to this change:
Result: 163 passed.
Full suites: