Feature/agent contribution metric

[dhruvigajjar]

PR: Agent Code Retention Tracking for Q CLI

Description

This PR implements agent code retention tracking to measure how much agent-written code remains in files after 15 minutes. Thismetric helps evaluate the long-term value and quality of agent contributions. The implementation includes graceful shutdown handling, error management, and telemetry to ensure no retention data is lost.

Solution

Implemented a production-ready scheduled retention tracking system that:

• Hooks after fs_write execution - Captures agent-written lines immediately after file operations
• Schedules 15-minute checks - Uses exact timing instead of periodic polling (currently 15 minute )
• Measures actual retention - Checks if agent lines still exist in files after 15 minutes
• Emits unified metrics - Extends existing AgentContribution telemetry with retention fields
• Handles graceful shutdown - Processes all pending checks on /quit or system signals (SIGINT/SIGTERM)
• Manages file errors - Gracefully handles deleted/missing files during retention checks
• Prevents data loss - Flushes pending checks when agent rewrites files
• Provides source attribution - Tracks whether metrics came from scheduled checks, shutdown, or error conditions

Implementation Details

Core Logic Flow

fs_write completes → Extract agent lines → Schedule retention check (15 min)
↓
Each chat interaction → Check due retention → Send metrics for completed checks
↓
On shutdown/quit/file-error → Flush all pending checks → Send metrics with source attribution

Key Changes

Enhanced Line Tracker (line_tracker.rs)
• Added RetentionCheck struct with lines, timestamp, conversation_id, and tool_use_id
• schedule_retention_check() - Schedules check exactly 15 minutes from fs_write
• check_due_retention() - Processes due checks and returns retention results with source attribution
• flush_all_retention_checks() - Immediately processes all pending checks for graceful shutdown
• flush_pending_checks_for_agent_rewrite() - Handles agent overwrite scenarios

fs_write Integration (fs_write.rs)
• extract_agent_lines() - Extracts lines from all write operations (Create, StrReplace, Insert, Append)
• Hooks into existing tool execution flow after line tracker updates
• Fixed null pointer handling for Create operations

Telemetry Extension (telemetry/)
• Extended AgentContribution event with optional retention fields:
• lines_retained: Option - Lines still present after retention check
• total_lines_checked: Option - Total lines that were checked
• source: Option - Attribution for metric source
• send_agent_contribution_metric_with_source() - New method with source tracking
• Single event type handles both immediate contribution and delayed retention metrics
• Backward compatibility maintained for existing telemetry calls

Retention Processing (conversation.rs)
• check_due_retention_metrics() - Called on each chat interaction
• Reads current file content and compares with scheduled checks
• Sends telemetry for completed retention measurements
• flush_all_retention_metrics() - Processes all pending checks immediately
• File error handling - Emits metrics with file_not_found source when files are deleted
• State cleanup - Automatically clears pending checks for missing files

Graceful Shutdown (mod.rs & cli/mod.rs)
• Signal handling - SIGINT/SIGTERM handlers flush all pending retention checks before exit
• Enhanced /quit command - Processes all pending checks before termination
• Cross-platform support - Unix systems get full signal handling, others maintain existing behavior
• Agent rewrite handling - Flushes pending checks when agent overwrites files

Source Attribution

All retention metrics now include source information:
• 15min_check - Regular scheduled retention checks
• quit - User initiated exit via /quit command
• shutdown - Process termination via SIGINT/SIGTERM
• file_not_found - File was deleted before retention check
• agent_rewrite - Agent overwrote file before scheduled check

Files Modified

• crates/chat-cli/src/cli/chat/line_tracker.rs - Retention scheduling logic with graceful shutdown
• crates/chat-cli/src/cli/chat/tools/fs_write.rs - Agent line extraction with null safety
• crates/chat-cli/src/cli/chat/mod.rs - Integration hooks, retention checking, and signal handling
• crates/chat-cli/src/cli/chat/conversation.rs - Retention metric processing with error handling
• crates/chat-cli/src/cli/chat/cli/mod.rs - Enhanced /quit command
• crates/chat-cli/src/telemetry/core.rs - Extended AgentContribution event
• crates/chat-cli/src/telemetry/mod.rs - Updated telemetry method signature with source support
• crates/chat-cli/telemetry_definitions.json - New telemetry field definitions

Enhanced Metric Schema

Regular Contribution (immediate):
rust
AgentContribution {
conversation_id: "conv-123",
tool_use_id: Some("tool-456"),
lines_by_agent: Some(10),
lines_by_user: Some(2),
lines_retained: None,
total_lines_checked: None,
source: None,
}

Retention Metric (15 minutes later):
rust
AgentContribution {
conversation_id: "conv-123",
tool_use_id: Some("tool-456"),
lines_by_agent: None,
lines_by_user: None,
lines_retained: Some(8),
total_lines_checked: Some(10),
source: Some("15min_check"),
}

Graceful Shutdown Metric:
rust
AgentContribution {
conversation_id: "conv-123",
tool_use_id: Some("tool-456"),
lines_retained: Some(8),
total_lines_checked: Some(10),
source: Some("quit"), // or "shutdown", "file_not_found", "agent_rewrite"
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[brandonskiser]

I don't think anything specific to agent contribution should need to wrap all other telemetry around tool uses. This is intended for codewhispererterminal_toolUseSuggested

[dhruvigajjar]

Yes changed it to attribute to agent Contribution metric