[Future] Audit and Enforce Tool Definition Stability

[frankbria]

Summary

Audit tool definition stability during task execution to prevent cache invalidation and model confusion from mid-task tool changes.

Background: State of the Art

From Philipp Schmid's 5 Practical Tips for Context Engineering:

"Manage Tools Statically: Avoid changing tool order or availability mid-task, if not explicitly needed. This might break context caching and can/will confuse models if used tools in the history are no longer defined."

Two problems arise from dynamic tool changes:

1. Cache invalidation: Tool definitions are part of the prompt prefix. Changing them invalidates caching.
2. Model confusion: If the model called tool_X earlier in the conversation but tool_X is no longer in the tool list, the model may hallucinate or reference unavailable capabilities.

The 12-Factor Agents framework also emphasizes: tools should be predictable and consistent within a task's lifecycle.

Current State in CodeFRAME

Each worker type (Backend, Frontend, Test, Review) presumably has role-specific tooling. Questions:

1. Tool list consistency: Does a worker's tool list change during task execution?
2. Cross-task tool changes: When a worker gets a new task, do available tools shift?
3. Dynamic tool injection: Can the Lead Agent add/remove tools from workers mid-task?
4. Tool ordering: Is the order of tool definitions consistent across calls?

Even if tools don't change, reordering tool definitions between calls can break caching.

Investigation Tasks

1. Audit tool definition patterns

   • Log tool lists sent with each LLM call
   • Diff consecutive calls to identify any changes
   • Check both tool availability AND ordering

2. Identify dynamic tool scenarios

   • Does context promotion/demotion affect tools?
   • Are tools added based on task content (e.g., "this task involves DB, add SQL tools")?
   • Do quality gates modify available tools?

3. Implement tool stability guarantees

   • Freeze tool list at task start
   • Ensure consistent ordering (alphabetical, or fixed priority order)
   • If tools must change, do so only at explicit task boundaries

4. Handle legitimate dynamic cases

   • If some tools ARE task-dependent, document the pattern
   • Consider "tool profiles" per task type, selected at start, frozen during execution
   • Log tool changes for debugging

Success Criteria

• Documented tool stability audit results
• Identified any mid-task tool changes
• Implemented tool list freezing (if changes found)
• Verified consistent tool ordering across all LLM calls
• Measured cache hit improvement (if applicable)

Implementation Notes

Tool ordering convention: Pick a deterministic order (alphabetical by name, or priority-based) and enforce it in the context builder.

Tool profiles: If workers need different tools for different task types:

TOOL_PROFILES = {
    "backend_api": ["file_read", "file_write", "test_run", "lint"],
    "backend_db": ["file_read", "file_write", "sql_query", "migration_run"],
    "frontend_component": ["file_read", "file_write", "npm_run", "browser_preview"],
}
# Select profile at task start, freeze for duration

History consistency: If a tool was used earlier in conversation history, it should remain in the tool list even if "not needed" for current step, until task completes.

References

• Context Engineering Tips - Philipp Schmid
• 12-Factor Agents - Tool management principles
• Anthropic tool use documentation - caching implications

[traycerai]

Plan

Observations

CodeFRAME currently has good tool stability for SDK-based agents (Backend, Hybrid workers) where tools are frozen at initialization. However, Frontend and Test workers use direct Anthropic API calls without any tool definitions, creating inconsistency. Tool ordering is deterministic (Python list order) but not explicitly enforced. The system lacks logging/auditing of tool definitions sent to LLM, making cache invalidation issues difficult to diagnose.

Approach

The plan focuses on auditing existing tool stability rather than fixing non-existent problems. Since tools are already frozen at initialization for SDK agents, the main tasks are: (1) add comprehensive logging to verify tool consistency, (2) document the current stable patterns, (3) address the Frontend/Test worker inconsistency, and (4) implement monitoring to detect any future regressions. This aligns with the "audit first, fix if needed" philosophy from the ticket.

Implementation Steps

1. Add Tool Definition Logging Infrastructure

File: file:codeframe/providers/sdk_client.py

Enhance SDKClientWrapper.__init__() (lines 34-68) to log tool definitions:

• Add logging statement after line 67 to record: allowed_tools list, order, and hash
• Log format: "SDK client initialized with tools: {allowed_tools} (hash: {hash(tuple(allowed_tools))})"
• This creates an audit trail for tool definitions at agent creation

Enhance send_message() method (lines 72-109) to log tool usage:

• Add debug logging before line 96 (query() call) to record tools being sent
• Log: "Sending LLM request with tools: {self._options.allowed_tools}"
• This verifies tools remain unchanged across calls

File: file:codeframe/agents/backend_worker_agent.py

Add tool verification in __init__() (lines 57-117):

• After line 99 (SDK client creation), add assertion: assert self.sdk_client._options.allowed_tools == ["Read", "Write", "Bash", "Glob", "Grep"]
• Log warning if tools differ from expected
• This catches any unintended tool list changes

2. Implement Tool Consistency Verification

File: file:codeframe/lib/tool_audit.py (new file)

Create a tool auditing utility class:

class ToolAuditor:
    """Audit tool definition stability across LLM calls."""
    
    def __init__(self):
        self.tool_snapshots = {}  # {agent_id: {call_count, tool_hash, tool_list}}
    
    def record_tool_usage(agent_id: str, tools: List[str]) -> None:
        """Record tool list for an agent's LLM call."""
        # Calculate hash of tool list (order-sensitive)
        # Compare with previous snapshot
        # Log warning if tools changed or reordered
    
    def verify_consistency(agent_id: str) -> bool:
        """Check if agent's tools remained consistent."""
        # Return True if all calls used same tools in same order
    
    def get_report() -> Dict[str, Any]:
        """Generate audit report of tool stability."""
        # Return summary: agents audited, consistency violations, cache breaks

Integrate into SDKClientWrapper.send_message():

• Call ToolAuditor.record_tool_usage() before each LLM call
• Log warnings for any tool changes detected

3. Document Tool Stability Patterns

File: file:docs/tool_stability_audit.md (new file)

Create comprehensive documentation:

Section 1: Current State

• Table showing tool definitions per agent type (Backend, Frontend, Test, Review, Hybrid)
• Note that Frontend/Test workers don't use tools (direct API calls)
• Document tool ordering: alphabetical within categories (Read/Write/Bash/Glob/Grep)

Section 2: Stability Guarantees

• Tools are frozen at agent initialization (SDK agents)
• Same ClaudeAgentOptions object reused for all calls
• No dynamic tool injection or removal during task execution
• Tool order is deterministic (Python list order)

Section 3: Cache Implications

• Tool definitions are part of prompt prefix (Anthropic caching)
• Changing tools invalidates cache
• Current implementation avoids this by freezing tools

Section 4: Monitoring

• How to use ToolAuditor to verify consistency
• Log patterns to watch for tool changes
• Metrics to track cache hit rates

4. Address Frontend/Test Worker Inconsistency

File: file:codeframe/agents/frontend_worker_agent.py

Modify __init__() (lines 34-68) to optionally use SDK:

• Add use_sdk: bool = False parameter
• If use_sdk=True, create SDKClientWrapper with tools ["Read", "Write", "Bash", "Glob", "Grep"]
• Store in self.sdk_client alongside existing self.client
• Update _generate_react_component() (lines 266-314) to use SDK client when available

File: file:codeframe/agents/test_worker_agent.py

Similar changes to __init__() (lines 40-76):

• Add use_sdk: bool = False parameter
• Create SDK client with same tool set when enabled
• Update _generate_pytest_tests() (lines 296-357) to use SDK client

File: file:codeframe/agents/agent_pool_manager.py

Update _create_traditional_agent() (lines 422-468):

• Pass use_sdk=self.use_sdk to Frontend/Test worker constructors
• This allows gradual migration to SDK for all agent types

5. Implement Tool Change Detection Tests

File: file:tests/lib/test_tool_audit.py (new file)

Create test suite for tool auditing:

Test: test_tool_consistency_across_calls

• Create SDK agent, make 10 LLM calls
• Verify tools remain identical (same list, same order)
• Assert no cache invalidation warnings

Test: test_tool_ordering_deterministic

• Create multiple agents of same type
• Verify all have identical tool lists in identical order
• Assert tool hash matches expected value

Test: test_tool_change_detection

• Simulate tool list modification (for testing only)
• Verify ToolAuditor detects and logs warning
• Assert consistency check fails

Test: test_cross_agent_type_tools

• Create Backend, Frontend, Test, Review agents
• Document tool differences per type
• Verify each type has consistent tools

6. Add Cache Hit Rate Monitoring

File: file:codeframe/lib/metrics_tracker.py

Extend existing metrics tracking (if not already present):

Add method record_cache_metrics():

• Track cache hits vs misses per agent
• Correlate with tool definition changes
• Log warnings if cache hit rate drops after tool changes

Integrate into SDKClientWrapper.send_message():

• Extract cache hit info from Anthropic response (if available)
• Call metrics_tracker.record_cache_metrics()
• Alert if hit rate drops below threshold (e.g., 70%)

7. Create Tool Profile Registry

File: file:codeframe/agents/tool_profiles.py (new file)

Centralize tool definitions:

TOOL_PROFILES = {
    "backend": ["Read", "Write", "Bash", "Glob", "Grep"],
    "frontend": ["Read", "Write", "Bash", "Glob", "Grep"],
    "test": ["Read", "Write", "Bash", "Glob", "Grep"],
    "review": ["Read", "Glob", "Grep", "Bash"],
    "default": ["Read", "Write", "Bash", "Glob", "Grep"],
}

def get_tools_for_agent_type(agent_type: str) -> List[str]:
    """Get frozen tool list for agent type."""
    return TOOL_PROFILES.get(agent_type, TOOL_PROFILES["default"]).copy()

Refactor existing code:

• Update file:codeframe/agents/agent_pool_manager.py lines 389-395 to use get_tools_for_agent_type()
• Update file:codeframe/agents/backend_worker_agent.py line 99 to use tool profile
• This ensures single source of truth for tool definitions

8. Validation and Verification

Manual Testing:

1. Run Lead Agent with multi-agent execution
2. Monitor logs for tool definition messages
3. Verify no tool changes during task execution
4. Check ToolAuditor report for consistency violations

Automated Testing:

1. Run new test suite: pytest tests/lib/test_tool_audit.py -v
2. Run existing agent tests to ensure no regressions
3. Verify cache hit rates remain high (>70%)

Documentation Review:

1. Review file:docs/tool_stability_audit.md for completeness
2. Ensure all agent types documented
3. Add examples of tool audit logs

Success Criteria

• ✅ Tool definitions logged at agent initialization and before each LLM call
• ✅ ToolAuditor detects any tool changes or reordering
• ✅ Documentation clearly explains tool stability guarantees
• ✅ Frontend/Test workers optionally support SDK with tools
• ✅ Tool profiles centralized in single registry
• ✅ Tests verify tool consistency across 10+ calls per agent
• ✅ Cache hit rate monitoring alerts on drops
• ✅ No tool changes detected during task execution in production logs

Architecture Diagram

sequenceDiagram
    participant APM as AgentPoolManager
    participant Agent as WorkerAgent
    participant SDK as SDKClientWrapper
    participant Auditor as ToolAuditor
    participant LLM as Claude API
    
    APM->>Agent: create_agent(type="backend")
    Agent->>SDK: __init__(allowed_tools=[...])
    SDK->>SDK: Store tools in self._options
    SDK->>Auditor: record_tool_usage(agent_id, tools)
    Note over SDK: Tools frozen at initialization
    
    loop Task Execution
        Agent->>SDK: send_message(conversation)
        SDK->>Auditor: record_tool_usage(agent_id, tools)
        Auditor->>Auditor: verify_consistency()
        alt Tools Changed
            Auditor-->>SDK: ⚠️ WARNING: Tool change detected
        end
        SDK->>LLM: query(prompt, options=self._options)
        Note over LLM: Same tools every call<br/>Cache remains valid
        LLM-->>SDK: response
        SDK-->>Agent: result
    end
    
    Auditor->>Auditor: get_report()
    Note over Auditor: All calls used same tools<br/>No cache invalidation

Loading

Tool Profile Comparison

Agent Type	Tools	Order Sensitive	SDK Used	Notes
Backend	Read, Write, Bash, Glob, Grep	✅ Yes	✅ Yes	Frozen at init
Frontend	(none)	N/A	❌ No	Direct API calls
Test	(none)	N/A	❌ No	Direct API calls
Review	Read, Glob, Grep, Bash	✅ Yes	✅ Yes (Hybrid)	Different order
Hybrid	Type-specific	✅ Yes	✅ Yes	Uses tool profiles

Import In IDE

🤖 Prompt for AI Agents

## Observations

CodeFRAME currently has **good tool stability** for SDK-based agents (Backend, Hybrid workers) where tools are frozen at initialization. However, **Frontend and Test workers use direct Anthropic API calls without any tool definitions**, creating inconsistency. Tool ordering is deterministic (Python list order) but not explicitly enforced. The system lacks logging/auditing of tool definitions sent to LLM, making cache invalidation issues difficult to diagnose.

## Approach

The plan focuses on **auditing existing tool stability** rather than fixing non-existent problems. Since tools are already frozen at initialization for SDK agents, the main tasks are: (1) add comprehensive logging to verify tool consistency, (2) document the current stable patterns, (3) address the Frontend/Test worker inconsistency, and (4) implement monitoring to detect any future regressions. This aligns with the "audit first, fix if needed" philosophy from the ticket.

## Implementation Steps

### 1. Add Tool Definition Logging Infrastructure

**File: `file:codeframe/providers/sdk_client.py`**

Enhance `SDKClientWrapper.__init__()` (lines 34-68) to log tool definitions:
- Add logging statement after line 67 to record: `allowed_tools` list, order, and hash
- Log format: `"SDK client initialized with tools: {allowed_tools} (hash: {hash(tuple(allowed_tools))})"` 
- This creates an audit trail for tool definitions at agent creation

Enhance `send_message()` method (lines 72-109) to log tool usage:
- Add debug logging before line 96 (`query()` call) to record tools being sent
- Log: `"Sending LLM request with tools: {self._options.allowed_tools}"`
- This verifies tools remain unchanged across calls

**File: `file:codeframe/agents/backend_worker_agent.py`**

Add tool verification in `__init__()` (lines 57-117):
- After line 99 (SDK client creation), add assertion: `assert self.sdk_client._options.allowed_tools == ["Read", "Write", "Bash", "Glob", "Grep"]`
- Log warning if tools differ from expected
- This catches any unintended tool list changes

### 2. Implement Tool Consistency Verification

**File: `file:codeframe/lib/tool_audit.py`** (new file)

Create a tool auditing utility class:

```python
class ToolAuditor:
    """Audit tool definition stability across LLM calls."""
    
    def __init__(self):
        self.tool_snapshots = {}  # {agent_id: {call_count, tool_hash, tool_list}}
    
    def record_tool_usage(agent_id: str, tools: List[str]) -> None:
        """Record tool list for an agent's LLM call."""
        # Calculate hash of tool list (order-sensitive)
        # Compare with previous snapshot
        # Log warning if tools changed or reordered
    
    def verify_consistency(agent_id: str) -> bool:
        """Check if agent's tools remained consistent."""
        # Return True if all calls used same tools in same order
    
    def get_report() -> Dict[str, Any]:
        """Generate audit report of tool stability."""
        # Return summary: agents audited, consistency violations, cache breaks
```

Integrate into `SDKClientWrapper.send_message()`:
- Call `ToolAuditor.record_tool_usage()` before each LLM call
- Log warnings for any tool changes detected

### 3. Document Tool Stability Patterns

**File: `file:docs/tool_stability_audit.md`** (new file)

Create comprehensive documentation:

**Section 1: Current State**
- Table showing tool definitions per agent type (Backend, Frontend, Test, Review, Hybrid)
- Note that Frontend/Test workers don't use tools (direct API calls)
- Document tool ordering: alphabetical within categories (Read/Write/Bash/Glob/Grep)

**Section 2: Stability Guarantees**
- Tools are frozen at agent initialization (SDK agents)
- Same `ClaudeAgentOptions` object reused for all calls
- No dynamic tool injection or removal during task execution
- Tool order is deterministic (Python list order)

**Section 3: Cache Implications**
- Tool definitions are part of prompt prefix (Anthropic caching)
- Changing tools invalidates cache
- Current implementation avoids this by freezing tools

**Section 4: Monitoring**
- How to use `ToolAuditor` to verify consistency
- Log patterns to watch for tool changes
- Metrics to track cache hit rates

### 4. Address Frontend/Test Worker Inconsistency

**File: `file:codeframe/agents/frontend_worker_agent.py`**

Modify `__init__()` (lines 34-68) to optionally use SDK:
- Add `use_sdk: bool = False` parameter
- If `use_sdk=True`, create `SDKClientWrapper` with tools `["Read", "Write", "Bash", "Glob", "Grep"]`
- Store in `self.sdk_client` alongside existing `self.client`
- Update `_generate_react_component()` (lines 266-314) to use SDK client when available

**File: `file:codeframe/agents/test_worker_agent.py`**

Similar changes to `__init__()` (lines 40-76):
- Add `use_sdk: bool = False` parameter  
- Create SDK client with same tool set when enabled
- Update `_generate_pytest_tests()` (lines 296-357) to use SDK client

**File: `file:codeframe/agents/agent_pool_manager.py`**

Update `_create_traditional_agent()` (lines 422-468):
- Pass `use_sdk=self.use_sdk` to Frontend/Test worker constructors
- This allows gradual migration to SDK for all agent types

### 5. Implement Tool Change Detection Tests

**File: `file:tests/lib/test_tool_audit.py`** (new file)

Create test suite for tool auditing:

**Test: `test_tool_consistency_across_calls`**
- Create SDK agent, make 10 LLM calls
- Verify tools remain identical (same list, same order)
- Assert no cache invalidation warnings

**Test: `test_tool_ordering_deterministic`**
- Create multiple agents of same type
- Verify all have identical tool lists in identical order
- Assert tool hash matches expected value

**Test: `test_tool_change_detection`**
- Simulate tool list modification (for testing only)
- Verify `ToolAuditor` detects and logs warning
- Assert consistency check fails

**Test: `test_cross_agent_type_tools`**
- Create Backend, Frontend, Test, Review agents
- Document tool differences per type
- Verify each type has consistent tools

### 6. Add Cache Hit Rate Monitoring

**File: `file:codeframe/lib/metrics_tracker.py`**

Extend existing metrics tracking (if not already present):

Add method `record_cache_metrics()`:
- Track cache hits vs misses per agent
- Correlate with tool definition changes
- Log warnings if cache hit rate drops after tool changes

Integrate into `SDKClientWrapper.send_message()`:
- Extract cache hit info from Anthropic response (if available)
- Call `metrics_tracker.record_cache_metrics()`
- Alert if hit rate drops below threshold (e.g., 70%)

### 7. Create Tool Profile Registry

**File: `file:codeframe/agents/tool_profiles.py`** (new file)

Centralize tool definitions:

```python
TOOL_PROFILES = {
    "backend": ["Read", "Write", "Bash", "Glob", "Grep"],
    "frontend": ["Read", "Write", "Bash", "Glob", "Grep"],
    "test": ["Read", "Write", "Bash", "Glob", "Grep"],
    "review": ["Read", "Glob", "Grep", "Bash"],
    "default": ["Read", "Write", "Bash", "Glob", "Grep"],
}

def get_tools_for_agent_type(agent_type: str) -> List[str]:
    """Get frozen tool list for agent type."""
    return TOOL_PROFILES.get(agent_type, TOOL_PROFILES["default"]).copy()
```

**Refactor existing code:**
- Update `file:codeframe/agents/agent_pool_manager.py` lines 389-395 to use `get_tools_for_agent_type()`
- Update `file:codeframe/agents/backend_worker_agent.py` line 99 to use tool profile
- This ensures single source of truth for tool definitions

### 8. Validation and Verification

**Manual Testing:**
1. Run Lead Agent with multi-agent execution
2. Monitor logs for tool definition messages
3. Verify no tool changes during task execution
4. Check `ToolAuditor` report for consistency violations

**Automated Testing:**
1. Run new test suite: `pytest tests/lib/test_tool_audit.py -v`
2. Run existing agent tests to ensure no regressions
3. Verify cache hit rates remain high (>70%)

**Documentation Review:**
1. Review `file:docs/tool_stability_audit.md` for completeness
2. Ensure all agent types documented
3. Add examples of tool audit logs

## Success Criteria

- ✅ Tool definitions logged at agent initialization and before each LLM call
- ✅ `ToolAuditor` detects any tool changes or reordering
- ✅ Documentation clearly explains tool stability guarantees
- ✅ Frontend/Test workers optionally support SDK with tools
- ✅ Tool profiles centralized in single registry
- ✅ Tests verify tool consistency across 10+ calls per agent
- ✅ Cache hit rate monitoring alerts on drops
- ✅ No tool changes detected during task execution in production logs

## Architecture Diagram

```mermaid
sequenceDiagram
    participant APM as AgentPoolManager
    participant Agent as WorkerAgent
    participant SDK as SDKClientWrapper
    participant Auditor as ToolAuditor
    participant LLM as Claude API
    
    APM->>Agent: create_agent(type="backend")
    Agent->>SDK: __init__(allowed_tools=[...])
    SDK->>SDK: Store tools in self._options
    SDK->>Auditor: record_tool_usage(agent_id, tools)
    Note over SDK: Tools frozen at initialization
    
    loop Task Execution
        Agent->>SDK: send_message(conversation)
        SDK->>Auditor: record_tool_usage(agent_id, tools)
        Auditor->>Auditor: verify_consistency()
        alt Tools Changed
            Auditor-->>SDK: ⚠️ WARNING: Tool change detected
        end
        SDK->>LLM: query(prompt, options=self._options)
        Note over LLM: Same tools every call<br/>Cache remains valid
        LLM-->>SDK: response
        SDK-->>Agent: result
    end
    
    Auditor->>Auditor: get_report()
    Note over Auditor: All calls used same tools<br/>No cache invalidation
```

## Tool Profile Comparison

| Agent Type | Tools | Order Sensitive | SDK Used | Notes |
|------------|-------|-----------------|----------|-------|
| Backend | Read, Write, Bash, Glob, Grep | ✅ Yes | ✅ Yes | Frozen at init |
| Frontend | *(none)* | N/A | ❌ No | Direct API calls |
| Test | *(none)* | N/A | ❌ No | Direct API calls |
| Review | Read, Glob, Grep, Bash | ✅ Yes | ✅ Yes (Hybrid) | Different order |
| Hybrid | Type-specific | ✅ Yes | ✅ Yes | Uses tool profiles |

---

Execution Information

Branch: main
Commit: 08bbb8e

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[frankbria]

Closing: v1 tool definitions. With #408 (Agent Adapter Architecture), external agents bring their own tools. CodeFrame's built-in ReactAgent tools are a fallback, not the primary path.