feat(agent): pluggable LLM domain classifier for multi-language tool retrieval - #5751
Open
holden093 wants to merge 7 commits into
Open
feat(agent): pluggable LLM domain classifier for multi-language tool retrieval#5751holden093 wants to merge 7 commits into
holden093 wants to merge 7 commits into
Conversation
…retrieval Add an async _llm_classify_domains() function that calls the utility model to classify user queries into domain categories. Gated by the environment variable ODYSSEUS_DOMAIN_CLASSIFIER=llm. When the env var is set, the English regex classifier is skipped entirely and every query is routed through the LLM for language-agnostic classification. Without the env var the existing regex path runs unchanged (the LLM is available as a fallback when no regex domains match). The LLM call uses the configured utility model. A capable instruction- following model (e.g. deepseek-v4-flash) is required for reliable results; smaller local models may not follow the constrained output format. The feature is opt-in so operators can enable it only when a suitable utility model is available. Closes: odysseus-dev#3713, odysseus-dev#3766 (non-English domain classifier)
Always pass _recent_context_for_retrieval() to _llm_classify_domains() so multi-language follow-ups like "Fai tutto tu", "Sì", "Fallo" inherit domain from prior user turns instead of losing tool context. Removes the continuation guard in LLM mode — the LLM now runs every turn with full context, making regex language limitations irrelevant. Also updates the classifier prompt to label multi-turn context as "Recent conversation" and adds a "Classify the latest request" instruction for clarity. Closes: odysseus-dev#4355 domain-context-loss on non-English follow-ups
…ompt Split the prompt into "Conversation history" and "Latest request" sections when latest_msg is provided. Prevents short keywords like "internet" from being drowned by domain-heavy conversation context when the user asks a cross-domain question (e.g. "search the web for this" while in a contacts-heavy conversation).
github-actions
Bot
added
the
ready for review
Commit 5019845 (the original domain-classifier feature commit) committed an unresolved conflict (<<<<<<< HEAD / ======= with no closing marker) into src/agent_loop.py, breaking 'python -m compileall'. Both sides are valid code; resolution is simply dropping the two stray marker lines, matching the author's own fix on local-dev. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GDcZuLNAo4FVc8WtMpZWVp
dev's src/agent_loop.py annotates with dict[str, Any] at lines 1503/1877 but only imports AsyncGenerator, List, Dict, Optional, Set (no Any, no from __future__ import annotations), so importing the module raises NameError: name 'Any' is not defined — breaking pytest collection for every test that transitively imports agent_loop. Add Any to the typing import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GDcZuLNAo4FVc8WtMpZWVp
This was referenced Jul 25, 2026
15 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_classify_agent_request()uses English-only regex patterns to detect domains (contacts,web,email, …). Non-English queries match no domains, are flaggedlow_signal, and the agent receives only a handful of always-available tools — it cannot reachresolve_contact,web_search,send_email, or any domain-specific tool. This is reported in #3713, #3766, and #3668.This PR adds an opt-in LLM-based domain classifier that replaces the regex patterns with a small utility-model call. The model understands every language without maintaining keyword lists.
How it works
Set the environment variable:
When enabled, every user query is classified by the utility model into zero or more domain categories. The regex classifier is bypassed entirely. When unset, the existing regex path runs unchanged (the LLM is still available as a fallback when no regex domains match).
Model requirements
A capable instruction-following model is required for reliable results. Tested successfully with
deepseek-v4-flashacross Italian, English, German, French, and Spanish. Smaller / weaker local models may not follow the constrained output format and will produce junk classifications — the feature is opt-in so operators can enable it only when a suitable utility model is available.Target branch
dev, notmain.Linked Issues
Closes #3713
Closes #3766
Related: #3668
Type of Change
Checklist
devHow to Test
ODYSSEUS_DOMAIN_CLASSIFIER=llmin the environment (.envor compose)resolve_contact_llm_classify_domainsreturnscontacts, tools seed correctly, agent callsresolve_contactRegression: with the env var unset, English regex classification works as before.
Changes
One file, +98 lines in
src/agent_loop.py:_llm_classify_domains()— async function that calls the utility model for domain classificationstream_agent_loop— gated byODYSSEUS_DOMAIN_CLASSIFIER == "llm"Visual / UI changes