Summary
Tool call activities are created with STARTED state but never completed, causing them to accumulate as orphaned records. On a moderately active instance, this creates hundreds of orphan records requiring periodic cleanup. The issue is in the activity tracking logic, not backend restarts.
Component
Backend / Activity Service / Chat Router
Priority
P2 - Feature impaired (workaround: periodic cleanup script), data accumulates but no functionality loss
Error
Not an error per se, but database shows accumulation:
=== TOOL_CALL ACTIVITIES BY STATE ===
completed : 246
failed : 932 # These are all orphans marked by cleanup
=== CHAT_START ACTIVITIES BY STATE ===
completed : 410
failed : 45 # Normal failure rate
Tool calls have 21% completion vs chat_start at 90%.
Location
- File:
src/backend/routers/chat.py
- Lines: ~200-220 (tool call tracking loop)
- Function: Chat message handler
Root Cause
In the chat endpoint, individual tool_call activities are created for each tool used:
# Each tool creates a STARTED activity
for tool_call in execution_log_simplified:
await activity_service.track_activity(
activity_type=ActivityType.TOOL_CALL, # Created as STARTED
parent_activity_id=chat_activity_id,
...
)
# Only parent activity is completed
await activity_service.complete_activity(
activity_id=chat_activity_id, # NOT the tool_call activities!
...
)
track_activity() creates activities in STARTED state. The code only calls complete_activity() on the parent chat_activity_id, leaving all tool_call activities in STARTED state forever.
Reproduction Steps
- Send a chat message to any agent
- Agent responds using tools (e.g., Read, Write, Bash)
- Check database:
SELECT activity_state, COUNT(*) FROM agent_activities WHERE activity_type = 'tool_call' GROUP BY activity_state
- Observe tool_call activities remain in 'started' state
- Repeat - orphans accumulate with each response
Suggested Fix
Option A - Complete tool_call activities immediately (they're historical records):
for tool_call in execution_log_simplified:
tool_activity_id = await activity_service.track_activity(
agent_name=name,
activity_type=ActivityType.TOOL_CALL,
...
)
# Complete immediately since this is a historical record
await activity_service.complete_activity(
activity_id=tool_activity_id,
status="completed",
details={"tool_name": tool_call.get("tool", "unknown")}
)
Option B - Create with completed state directly:
# Add state parameter to track_activity() or use a new method
await activity_service.record_completed_activity(
activity_type=ActivityType.TOOL_CALL,
...
)
Option C - Don't create individual tool_call activities at all (data already stored in chat message tool_calls JSON column).
Environment
- Trinity version:
d34ca28
- Instance: Production with multiple active agents
Impact
- Database bloat: ~1000 orphaned records per week on active instance
- Requires periodic cleanup via
/cleanup-orphans skill
- Cleanup marks them as
failed with error message, which is misleading
- UI may show stale "in progress" activities
Related
- Activity service:
src/backend/services/activity_service.py
- Activity tracking:
track_activity() always creates with STARTED state
- Chat message already stores tool calls in
tool_calls JSON column (possible duplicate data)
Summary
Tool call activities are created with
STARTEDstate but never completed, causing them to accumulate as orphaned records. On a moderately active instance, this creates hundreds of orphan records requiring periodic cleanup. The issue is in the activity tracking logic, not backend restarts.Component
Backend / Activity Service / Chat Router
Priority
P2 - Feature impaired (workaround: periodic cleanup script), data accumulates but no functionality loss
Error
Not an error per se, but database shows accumulation:
Tool calls have 21% completion vs chat_start at 90%.
Location
src/backend/routers/chat.pyRoot Cause
In the chat endpoint, individual
tool_callactivities are created for each tool used:track_activity()creates activities inSTARTEDstate. The code only callscomplete_activity()on the parentchat_activity_id, leaving all tool_call activities inSTARTEDstate forever.Reproduction Steps
SELECT activity_state, COUNT(*) FROM agent_activities WHERE activity_type = 'tool_call' GROUP BY activity_stateSuggested Fix
Option A - Complete tool_call activities immediately (they're historical records):
Option B - Create with completed state directly:
Option C - Don't create individual tool_call activities at all (data already stored in chat message
tool_callsJSON column).Environment
d34ca28Impact
/cleanup-orphansskillfailedwith error message, which is misleadingRelated
src/backend/services/activity_service.pytrack_activity()always creates withSTARTEDstatetool_callsJSON column (possible duplicate data)