Skip to content

Commit

Permalink
update classifier prompt
Browse files Browse the repository at this point in the history
  • Loading branch information
dhirenmathur committed Dec 20, 2024
1 parent 3c05de4 commit ade7591
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 89 deletions.
48 changes: 40 additions & 8 deletions app/modules/conversations/conversation/conversation_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,15 +104,46 @@ async def initialize(self, user_id: str):

# Enhanced classifier prompt with agent descriptions
self.classifier_prompt = """
Given the user query, determine which agent should handle it based on their specialties:
Given the user query and the current agent ID, select the most appropriate agent by comparing the query’s requirements with each agent’s specialties.
Query: {query}
Current Agent ID: {agent_id}
Available agents and their specialties:
{agent_descriptions}
Return ONLY the agent id and confidence score in format: agent_id|confidence
Example: debugging_agent|0.85
Follow the instructions below to determine the best matching agent and provide a confidence score:
Analysis Instructions (DO NOT include these instructions in the final answer):
1. **Semantic Analysis:**
- Identify the key topics, technical terms, and the user’s intent from the query.
- Compare these elements to each agent’s detailed specialty description.
- Focus on specific skills, tools, frameworks, and domain expertise mentioned.
2. **Contextual Weighting:**
- If the query strongly aligns with the current agent’s known capabilities, add +0.15 confidence for direct core expertise and +0.1 for related domain knowledge.
- If the query introduces new topics outside the current agent’s domain, do not apply the current agent bias. Instead, evaluate all agents equally based on their described expertise.
3. **Multi-Agent Evaluation:**
- Consider all agents’ described specialties thoroughly, not just the current agent.
- For overlapping capabilities, favor the agent with more specialized expertise or more relevant tools/methodologies.
- If no agent clearly surpasses a 0.5 confidence threshold, select the agent with the highest confidence score, even if it is below 0.5.
4. **Confidence Scoring Guidelines:**
- 0.9-1.0: Ideal match with the agent’s core, primary expertise.
- 0.7-0.9: Strong match with the agent’s known capabilities.
- 0.5-0.7: Partial or related match, not a direct specialty.
- Below 0.5: Weak match; consider if another agent is more suitable, but still choose the best available option.
Final Output Requirements:
- Return ONLY the chosen agent_id and the confidence score in the format:
`agent_id|confidence`
Examples:
- Direct expertise match: `debugging_agent|0.95`
- Related capability (current agent): `current_agent_id|0.75`
- Need different expertise: `ml_training_agent|0.85`
- Overlapping domains, choose more specialized: `choose_higher_expertise_agent|0.80`
"""

# Format agent descriptions for the prompt
Expand All @@ -136,8 +167,9 @@ async def classifier_node(self, state: State) -> Command:

# Classification using LLM with enhanced prompt
prompt = self.classifier_prompt.format(
query=state["query"],
agent_descriptions=self.agent_descriptions
query=state["query"],
agent_id=state["agent_id"],
agent_descriptions=self.agent_descriptions,
)
response = await self.llm.ainvoke(prompt)

Expand Down
80 changes: 0 additions & 80 deletions app/modules/intelligence/agents/agent_classifier.py

This file was deleted.

2 changes: 1 addition & 1 deletion app/modules/intelligence/agents/agents_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ async def list_available_agents(
AgentInfo(
id="code_changes_agent",
name="Code Changes Agent",
description="An agent specialized in generating detailed analysis of code changes in your current branch compared to default branch. Works best with Py, JS, TS",
description="An agent specialized in generating blast radius of the code changes in your current branch compared to default branch. Use this for functional review of your code changes. Works best with Py, JS, TS",
status="SYSTEM",
),
AgentInfo(
Expand Down

0 comments on commit ade7591

Please sign in to comment.