[Phase 2.5] GoldenPathRunner._detect_success() has false positive when agent produces no output

[frankbria]

Problem

GoldenPathRunner._detect_success() returns True (success) when the CLI output contains no recognizable success or failure patterns. This caused a false 100% success rate in a run where the agent completed 10 tasks in 1.6s each without generating any code.

Root Cause

In tests/e2e/cli/golden_path_runner.py (line 305-332):

def _detect_success(self, exit_code: int, output: str) -> bool:
    # ...checks success_patterns, failure_patterns...
    
    # If no clear signal, fall back to exit code
    return exit_code == 0  # ← WRONG: defaults to success

When the agent doesn't actually execute (e.g., missing API key, stale run), the CLI returns exit code 0 with output that doesn't match any patterns. The fallback treats this as success.

Observed Behavior

A 31-second run where all 10 tasks "completed" in 1.6s each:

• _detect_success() returned True for all tasks
• test_all_tasks_succeed passed
• But test_files_generated and test_tests_generated correctly failed (0 files generated)

Proposed Fix

Change the default fallback from True to False:

# If no clear signal, assume failure (conservative)
return False

Or better, require an explicit success pattern:

# No success pattern found — treat as failure
for pattern in success_patterns:
    if pattern in output:
        return True
return False

Related

The CLI itself should also be fixed to return non-zero exit codes for FAILED tasks (typer.Exit(1)), but that's a separate issue.

Files

• tests/e2e/cli/golden_path_runner.py — _detect_success() method (line 305)

[traycerai]

Plan

Observations

The issue describes a false positive bug in GoldenPathRunner._detect_success() method in file:tests/e2e/cli/golden_path_runner.py. This file doesn't exist yet in the codebase, indicating it's part of Phase 2.5 testing infrastructure being developed. The current codebase has golden path integration tests in file:tests/core/test_golden_path_integration.py and CLI tests in file:tests/cli/test_v2_cli_integration.py, but lacks an e2e CLI test runner with pattern-based success detection. The bug causes silent failures when the CLI agent produces no output but returns exit code 0.

Approach

The fix requires creating a conservative success detection strategy that treats ambiguous cases (no matching patterns) as failures rather than successes. This prevents false positives when the agent doesn't actually execute. The implementation should follow the "fail-safe" principle: require explicit success signals rather than assuming success from the absence of failure. Additionally, the CLI commands should be updated to return non-zero exit codes for failed tasks, providing a more reliable signal for test automation.

Implementation Steps

1. Create the GoldenPathRunner Test Infrastructure

File: file:tests/e2e/cli/golden_path_runner.py

Create a new test runner class that executes CLI commands and detects success/failure based on output patterns:

• Implement GoldenPathRunner class with methods for running CLI commands
• Add _detect_success(exit_code: int, output: str) -> bool method
• Define success and failure pattern lists (e.g., "✓", "completed", "success" vs "error", "failed", "exception")
• Implement subprocess execution with output capture
• Add timeout handling and process management
• Include logging for debugging test failures

2. Fix the _detect_success() Method Logic

File: file:tests/e2e/cli/golden_path_runner.py (line ~305-332)

Implement conservative success detection logic:

• Primary check: Scan output for explicit success patterns (e.g., "✓", "completed successfully", "task done")
• Secondary check: Scan output for failure patterns (e.g., "error", "failed", "exception", "traceback")
• Failure precedence: If failure patterns found, return False immediately
• Success requirement: Return True only if success patterns found AND no failure patterns
• Conservative fallback: If no patterns match, return False (not True)
• Add detailed logging of which patterns matched for debugging

Example logic flow:

1. Check for failure patterns → if found, return False
2. Check for success patterns → if found, return True  
3. No patterns matched → return False (conservative default)

3. Define Comprehensive Pattern Lists

File: file:tests/e2e/cli/golden_path_runner.py

Create pattern lists based on actual CLI output:

Success patterns:

• "✓" (checkmark character)
• "completed successfully"
• "task completed"
• "run started" (for async operations)
• "generated" (for task generation)
• Specific command success indicators

Failure patterns:

• "error:"
• "failed"
• "exception"
• "traceback"
• "not found"
• "invalid"
• "missing api key"
• "authentication failed"

4. Add Test Cases for Success Detection

File: file:tests/e2e/cli/test_golden_path_runner.py (new file)

Create comprehensive tests for the runner itself:

• Test success detection with clear success output
• Test failure detection with clear failure output
• Test ambiguous case (no patterns) returns False
• Test exit code 0 with no output returns False
• Test exit code 1 with failure patterns returns False
• Test mixed patterns (both success and failure) returns False
• Test timeout handling
• Test subprocess error handling

5. Update CLI Commands to Return Proper Exit Codes

File: file:codeframe/cli.py or relevant v2 CLI module

Ensure CLI commands return non-zero exit codes on failure:

• Identify commands that currently return exit code 0 on failure
• Add raise typer.Exit(1) for failed operations
• Ensure exceptions are caught and converted to exit code 1
• Add exit code documentation to command help text
• Verify task execution failures return non-zero codes

Key commands to check:

• work start / work batch run
• tasks generate
• prd add
• Any command that can fail silently

6. Create E2E Golden Path Test Suite

File: file:tests/e2e/cli/test_golden_path_e2e.py (new file)

Build comprehensive e2e tests using the runner:

• Test complete workflow: init → prd add → tasks generate → work start
• Test success case: agent generates code successfully
• Test failure case: missing API key, invalid input
• Test timeout case: long-running operations
• Verify file generation assertions
• Verify test generation assertions
• Add assertions for actual work artifacts, not just exit codes

7. Add Configuration for Pattern Customization

File: file:tests/e2e/cli/golden_path_runner.py

Make patterns configurable for different test scenarios:

• Allow custom success/failure patterns per test
• Support pattern overrides via constructor parameters
• Add pattern validation (ensure patterns are non-empty)
• Document pattern matching behavior
• Consider regex support for complex patterns

8. Implement Detailed Logging and Diagnostics

File: file:tests/e2e/cli/golden_path_runner.py

Add comprehensive logging for test debugging:

• Log full CLI command being executed
• Log captured stdout and stderr
• Log which patterns matched (or didn't match)
• Log exit code and final success/failure decision
• Add timing information for performance analysis
• Include context about test environment (API keys present, etc.)

Visual Representation

sequenceDiagram
    participant Test as E2E Test
    participant Runner as GoldenPathRunner
    participant CLI as CodeFRAME CLI
    participant Agent as AI Agent

    Test->>Runner: run_command("work start task-123")
    Runner->>CLI: Execute subprocess
    CLI->>Agent: Start task execution
    
    alt Agent executes successfully
        Agent-->>CLI: Generate code + success output
        CLI-->>Runner: exit_code=0, output="✓ Task completed"
        Runner->>Runner: _detect_success()
        Note over Runner: 1. Check failure patterns: None<br/>2. Check success patterns: "✓" found<br/>3. Return True
        Runner-->>Test: Success=True
    else Agent fails
        Agent-->>CLI: Error output
        CLI-->>Runner: exit_code=1, output="Error: failed"
        Runner->>Runner: _detect_success()
        Note over Runner: 1. Check failure patterns: "Error" found<br/>2. Return False immediately
        Runner-->>Test: Success=False
    else Agent produces no output (BUG)
        Agent-->>CLI: No output (stale run)
        CLI-->>Runner: exit_code=0, output=""
        Runner->>Runner: _detect_success()
        Note over Runner: 1. Check failure patterns: None<br/>2. Check success patterns: None<br/>3. Conservative fallback: Return False
        Runner-->>Test: Success=False (FIXED)
    end

Loading

Success Criteria

Criterion	Validation
No false positives	Tests with no output return False
Clear success detected	Tests with success patterns return True
Clear failure detected	Tests with failure patterns return False
Exit codes respected	Non-zero exit codes trigger failure
Patterns configurable	Custom patterns can be provided
Comprehensive logging	Debug output shows pattern matching
CLI returns proper codes	Failed commands return exit code 1
E2E tests pass	Full workflow tests validate behavior

Import In IDE

🤖 Prompt for AI Agents

## Observations

The issue describes a false positive bug in `GoldenPathRunner._detect_success()` method in `file:tests/e2e/cli/golden_path_runner.py`. This file doesn't exist yet in the codebase, indicating it's part of Phase 2.5 testing infrastructure being developed. The current codebase has golden path integration tests in `file:tests/core/test_golden_path_integration.py` and CLI tests in `file:tests/cli/test_v2_cli_integration.py`, but lacks an e2e CLI test runner with pattern-based success detection. The bug causes silent failures when the CLI agent produces no output but returns exit code 0.

## Approach

The fix requires creating a conservative success detection strategy that treats ambiguous cases (no matching patterns) as failures rather than successes. This prevents false positives when the agent doesn't actually execute. The implementation should follow the "fail-safe" principle: require explicit success signals rather than assuming success from the absence of failure. Additionally, the CLI commands should be updated to return non-zero exit codes for failed tasks, providing a more reliable signal for test automation.

## Implementation Steps

### 1. Create the GoldenPathRunner Test Infrastructure

**File**: `file:tests/e2e/cli/golden_path_runner.py`

Create a new test runner class that executes CLI commands and detects success/failure based on output patterns:

- Implement `GoldenPathRunner` class with methods for running CLI commands
- Add `_detect_success(exit_code: int, output: str) -> bool` method
- Define success and failure pattern lists (e.g., "✓", "completed", "success" vs "error", "failed", "exception")
- Implement subprocess execution with output capture
- Add timeout handling and process management
- Include logging for debugging test failures

### 2. Fix the `_detect_success()` Method Logic

**File**: `file:tests/e2e/cli/golden_path_runner.py` (line ~305-332)

Implement conservative success detection logic:

- **Primary check**: Scan output for explicit success patterns (e.g., "✓", "completed successfully", "task done")
- **Secondary check**: Scan output for failure patterns (e.g., "error", "failed", "exception", "traceback")
- **Failure precedence**: If failure patterns found, return `False` immediately
- **Success requirement**: Return `True` only if success patterns found AND no failure patterns
- **Conservative fallback**: If no patterns match, return `False` (not `True`)
- Add detailed logging of which patterns matched for debugging

Example logic flow:
```
1. Check for failure patterns → if found, return False
2. Check for success patterns → if found, return True  
3. No patterns matched → return False (conservative default)
```

### 3. Define Comprehensive Pattern Lists

**File**: `file:tests/e2e/cli/golden_path_runner.py`

Create pattern lists based on actual CLI output:

**Success patterns**:
- "✓" (checkmark character)
- "completed successfully"
- "task completed"
- "run started" (for async operations)
- "generated" (for task generation)
- Specific command success indicators

**Failure patterns**:
- "error:"
- "failed"
- "exception"
- "traceback"
- "not found"
- "invalid"
- "missing api key"
- "authentication failed"

### 4. Add Test Cases for Success Detection

**File**: `file:tests/e2e/cli/test_golden_path_runner.py` (new file)

Create comprehensive tests for the runner itself:

- Test success detection with clear success output
- Test failure detection with clear failure output
- Test ambiguous case (no patterns) returns `False`
- Test exit code 0 with no output returns `False`
- Test exit code 1 with failure patterns returns `False`
- Test mixed patterns (both success and failure) returns `False`
- Test timeout handling
- Test subprocess error handling

### 5. Update CLI Commands to Return Proper Exit Codes

**File**: `file:codeframe/cli.py` or relevant v2 CLI module

Ensure CLI commands return non-zero exit codes on failure:

- Identify commands that currently return exit code 0 on failure
- Add `raise typer.Exit(1)` for failed operations
- Ensure exceptions are caught and converted to exit code 1
- Add exit code documentation to command help text
- Verify task execution failures return non-zero codes

Key commands to check:
- `work start` / `work batch run`
- `tasks generate`
- `prd add`
- Any command that can fail silently

### 6. Create E2E Golden Path Test Suite

**File**: `file:tests/e2e/cli/test_golden_path_e2e.py` (new file)

Build comprehensive e2e tests using the runner:

- Test complete workflow: init → prd add → tasks generate → work start
- Test success case: agent generates code successfully
- Test failure case: missing API key, invalid input
- Test timeout case: long-running operations
- Verify file generation assertions
- Verify test generation assertions
- Add assertions for actual work artifacts, not just exit codes

### 7. Add Configuration for Pattern Customization

**File**: `file:tests/e2e/cli/golden_path_runner.py`

Make patterns configurable for different test scenarios:

- Allow custom success/failure patterns per test
- Support pattern overrides via constructor parameters
- Add pattern validation (ensure patterns are non-empty)
- Document pattern matching behavior
- Consider regex support for complex patterns

### 8. Implement Detailed Logging and Diagnostics

**File**: `file:tests/e2e/cli/golden_path_runner.py`

Add comprehensive logging for test debugging:

- Log full CLI command being executed
- Log captured stdout and stderr
- Log which patterns matched (or didn't match)
- Log exit code and final success/failure decision
- Add timing information for performance analysis
- Include context about test environment (API keys present, etc.)

## Visual Representation

```mermaid
sequenceDiagram
    participant Test as E2E Test
    participant Runner as GoldenPathRunner
    participant CLI as CodeFRAME CLI
    participant Agent as AI Agent

    Test->>Runner: run_command("work start task-123")
    Runner->>CLI: Execute subprocess
    CLI->>Agent: Start task execution
    
    alt Agent executes successfully
        Agent-->>CLI: Generate code + success output
        CLI-->>Runner: exit_code=0, output="✓ Task completed"
        Runner->>Runner: _detect_success()
        Note over Runner: 1. Check failure patterns: None<br/>2. Check success patterns: "✓" found<br/>3. Return True
        Runner-->>Test: Success=True
    else Agent fails
        Agent-->>CLI: Error output
        CLI-->>Runner: exit_code=1, output="Error: failed"
        Runner->>Runner: _detect_success()
        Note over Runner: 1. Check failure patterns: "Error" found<br/>2. Return False immediately
        Runner-->>Test: Success=False
    else Agent produces no output (BUG)
        Agent-->>CLI: No output (stale run)
        CLI-->>Runner: exit_code=0, output=""
        Runner->>Runner: _detect_success()
        Note over Runner: 1. Check failure patterns: None<br/>2. Check success patterns: None<br/>3. Conservative fallback: Return False
        Runner-->>Test: Success=False (FIXED)
    end
```

## Success Criteria

| Criterion | Validation |
|-----------|------------|
| No false positives | Tests with no output return `False` |
| Clear success detected | Tests with success patterns return `True` |
| Clear failure detected | Tests with failure patterns return `False` |
| Exit codes respected | Non-zero exit codes trigger failure |
| Patterns configurable | Custom patterns can be provided |
| Comprehensive logging | Debug output shows pattern matching |
| CLI returns proper codes | Failed commands return exit code 1 |
| E2E tests pass | Full workflow tests validate behavior |

---

Execution Information

Branch: main
Commit: e04112c

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

🔗 Related PRs

#356 - fix: agent execution resilience + Phase 2.5 architecture research [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!