Skip to content

bug: Tool call activities created but never completed, accumulating as orphans #45

Description

@vybe

Summary

Tool call activities are created with STARTED state but never completed, causing them to accumulate as orphaned records. On a moderately active instance, this creates hundreds of orphan records requiring periodic cleanup. The issue is in the activity tracking logic, not backend restarts.

Component

Backend / Activity Service / Chat Router

Priority

P2 - Feature impaired (workaround: periodic cleanup script), data accumulates but no functionality loss

Error

Not an error per se, but database shows accumulation:

=== TOOL_CALL ACTIVITIES BY STATE ===
completed : 246
failed : 932  # These are all orphans marked by cleanup

=== CHAT_START ACTIVITIES BY STATE ===
completed : 410
failed : 45   # Normal failure rate

Tool calls have 21% completion vs chat_start at 90%.

Location

  • File: src/backend/routers/chat.py
  • Lines: ~200-220 (tool call tracking loop)
  • Function: Chat message handler

Root Cause

In the chat endpoint, individual tool_call activities are created for each tool used:

# Each tool creates a STARTED activity
for tool_call in execution_log_simplified:
    await activity_service.track_activity(
        activity_type=ActivityType.TOOL_CALL,  # Created as STARTED
        parent_activity_id=chat_activity_id,
        ...
    )

# Only parent activity is completed
await activity_service.complete_activity(
    activity_id=chat_activity_id,  # NOT the tool_call activities!
    ...
)

track_activity() creates activities in STARTED state. The code only calls complete_activity() on the parent chat_activity_id, leaving all tool_call activities in STARTED state forever.

Reproduction Steps

  1. Send a chat message to any agent
  2. Agent responds using tools (e.g., Read, Write, Bash)
  3. Check database: SELECT activity_state, COUNT(*) FROM agent_activities WHERE activity_type = 'tool_call' GROUP BY activity_state
  4. Observe tool_call activities remain in 'started' state
  5. Repeat - orphans accumulate with each response

Suggested Fix

Option A - Complete tool_call activities immediately (they're historical records):

for tool_call in execution_log_simplified:
    tool_activity_id = await activity_service.track_activity(
        agent_name=name,
        activity_type=ActivityType.TOOL_CALL,
        ...
    )
    # Complete immediately since this is a historical record
    await activity_service.complete_activity(
        activity_id=tool_activity_id,
        status="completed",
        details={"tool_name": tool_call.get("tool", "unknown")}
    )

Option B - Create with completed state directly:

# Add state parameter to track_activity() or use a new method
await activity_service.record_completed_activity(
    activity_type=ActivityType.TOOL_CALL,
    ...
)

Option C - Don't create individual tool_call activities at all (data already stored in chat message tool_calls JSON column).

Environment

  • Trinity version: d34ca28
  • Instance: Production with multiple active agents

Impact

  • Database bloat: ~1000 orphaned records per week on active instance
  • Requires periodic cleanup via /cleanup-orphans skill
  • Cleanup marks them as failed with error message, which is misleading
  • UI may show stale "in progress" activities

Related

  • Activity service: src/backend/services/activity_service.py
  • Activity tracking: track_activity() always creates with STARTED state
  • Chat message already stores tool calls in tool_calls JSON column (possible duplicate data)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions