[Phase 5] Engine Performance Tracking and Comparison

[frankbria]

Summary

Track per-engine success rates, token usage, duration, and gate pass rates so users can make informed decisions about which execution engine works best for their project. A lightweight version of Gastown's agent CV system — no permanent identity, no skill routing, just metrics.

Motivation

With #408 (Agent Adapter Architecture), CodeFrame will support multiple engines (Claude Code, Codex, OpenCode, built-in, etc.). The natural question becomes: "Which engine works best for my Python project?" or "Is Claude Code worth the cost vs the built-in agent?"

Gastown tracks full agent CVs with work history, skill demonstration, and quality metrics. That's overkill for CodeFrame's single-developer audience. But basic performance tracking per engine is genuinely useful and trivially implementable since CodeFrame already tracks run outcomes.

Design (single session)

Data to track (per engine, per workspace)

Metric	Source	Description
tasks_attempted	runtime	Total tasks started with this engine
tasks_completed	runtime	Tasks that reached DONE
tasks_failed	runtime	Tasks that reached FAILED
gate_pass_rate	gates	% of runs where gates passed on first try
self_correction_rate	verification loop	% of runs needing self-correction retries
avg_duration_ms	runtime	Average execution time per task
total_tokens	adapter result	Total token usage
avg_tokens_per_task	computed	Tokens / completed tasks

Storage

Add an engine_stats table to the existing SQLite database:

CREATE TABLE engine_stats (
    workspace_id TEXT,
    engine TEXT,
    metric TEXT,
    value REAL,
    updated_at TEXT,
    PRIMARY KEY (workspace_id, engine, metric)
);

Plus a run_engine_log table for per-run detail:

CREATE TABLE run_engine_log (
    run_id TEXT PRIMARY KEY,
    engine TEXT,
    task_id TEXT,
    status TEXT,       -- COMPLETED, FAILED, BLOCKED
    duration_ms INT,
    tokens_used INT,
    gates_passed BOOLEAN,
    self_corrections INT,
    created_at TEXT
);

New CLI commands

cf engines stats                    # Show engine performance comparison
cf engines stats --engine claude-code  # Stats for one engine
cf engines compare                  # Side-by-side comparison table

Example output:

Engine Performance (my-project)
┌──────────────┬──────────┬──────────┬───────────┬──────────────┬─────────────┐
│ Engine       │ Tasks    │ Success  │ Gate Pass │ Avg Duration │ Avg Tokens  │
├──────────────┼──────────┼──────────┼───────────┼──────────────┼─────────────┤
│ claude-code  │ 12       │ 92%      │ 75%       │ 45s          │ 8,200       │
│ built-in     │ 8        │ 75%      │ 50%       │ 120s         │ 15,400      │
│ opencode     │ 3        │ 100%     │ 67%       │ 60s          │ 10,100      │
└──────────────┴──────────┴──────────┴───────────┴──────────────┴─────────────┘

Integration points

• core/runtime.py — after execute_agent() completes, record engine + outcome
• core/verification_loop.py ([Phase 4] Verification Gate Wrapper for External Agents #415) — record gate pass/fail and self-correction count
• Agent adapters ([Phase 4] Agent Adapter Protocol Definition #409) — return TokenUsage in AgentResult

Acceptance Criteria

• Run outcomes recorded with engine name in SQLite
• cf engines stats shows per-engine performance summary
• cf engines compare shows side-by-side table
• Token usage tracked (when adapter reports it)
• Gate pass rate tracked
• Self-correction count tracked
• Stats accumulate across sessions (persistent)
• Works with all registered engines

Dependencies

• Depends on [Phase 4] Engine Registry and Runtime Selection #414 (engine registry) — need engine names
• Benefits from [Phase 4] Verification Gate Wrapper for External Agents #415 (verification wrapper) — gate pass/fail data
• No hard blockers — can start with just the built-in engine

Complexity Note

This is deliberately simpler than Gastown's agent CV system:

• No permanent agent identity (engines don't have names like "Toast")
• No skill routing (we don't auto-select engines based on history)
• No cross-project tracking (stats are per-workspace)
• No audit trails or compliance features
• Just: "which engine works best here?"

[traycerai]

Plan

Observations

The workspace uses a standalone SQLite database at .codeframe/state.db, managed in file:codeframe/core/workspace.py via _init_database() and _ensure_schema_upgrades(). Schema changes are made with raw SQLite—no ORM. file:codeframe/core/runtime.py's execute_agent() already receives the engine param and the final AgentState (which has gate_results and attempt_count). ReactAgent tracks _total_tokens_used internally. CLI sub-apps are registered in file:codeframe/cli/app.py using typer and rich tables—the same pattern the new engines sub-app must follow.

Approach

Add two new SQLite tables (run_engine_log, engine_stats) via the existing schema upgrade path in workspace.py, then wire recording logic into execute_agent() in runtime.py, and finally create a new engines Typer sub-app (a new file file:codeframe/cli/engines_commands.py) registered in app.py. No new libraries are needed—typer, rich, and sqlite3 are already used.

---

Implementation Steps

1. Extend the Database Schema

In file:codeframe/core/workspace.py, add the two new tables inside both _init_database() and _ensure_schema_upgrades() so existing workspaces get them automatically.

In _init_database(), add two new CREATE TABLE IF NOT EXISTS blocks after the run_logs block:

• run_engine_log: columns run_id TEXT PRIMARY KEY, engine TEXT, task_id TEXT, workspace_id TEXT, status TEXT (COMPLETED/FAILED/BLOCKED), duration_ms INTEGER, tokens_used INTEGER, gates_passed INTEGER (1/0/NULL), self_corrections INTEGER, created_at TEXT, with a FOREIGN KEY (run_id) REFERENCES runs(id).

• engine_stats: columns workspace_id TEXT, engine TEXT, metric TEXT, value REAL, updated_at TEXT, with PRIMARY KEY (workspace_id, engine, metric).

Also add performance indexes:

• idx_run_engine_log_workspace_engine ON run_engine_log(workspace_id, engine)
• idx_engine_stats_workspace ON engine_stats(workspace_id, engine)

In _ensure_schema_upgrades(), add idempotent CREATE TABLE IF NOT EXISTS calls for both tables (same DDL), so they appear in existing workspaces the first time get_workspace() is called.

---

2. Create an Engine Stats Helper Module

Create file:codeframe/core/engine_stats.py with the following functions (all using get_db_connection(workspace) exactly as runtime.py does):

• record_run(workspace, run_id, engine, task_id, status, duration_ms, tokens_used, gates_passed, self_corrections) — inserts a row into run_engine_log and then calls _update_aggregate_stats().

• _update_aggregate_stats(workspace, engine) — private function that computes all aggregate metrics from run_engine_log for the given engine+workspace (tasks_attempted, tasks_completed, tasks_failed, gate_pass_rate, self_correction_rate, avg_duration_ms, total_tokens, avg_tokens_per_task) and upserts each as a row in engine_stats using INSERT OR REPLACE.

• get_engine_stats(workspace, engine=None) → returns a dict[str, dict[str, float]] keyed by engine name. If engine is provided, filters to that engine only. Reads from the engine_stats table.

• get_run_log(workspace, engine=None, limit=100) → returns raw per-run records from run_engine_log for display in a detailed view.

---

3. Wire Recording into execute_agent()

In file:codeframe/core/runtime.py, update execute_agent() to capture and record metrics after the agent completes. The timing, status, token count, and gate results are all already available in that function.

At the top of the try block, capture start_time using _utc_now() (already a helper in that module).

After the final status-based complete_run / fail_run / block_run call (around line 866), add a metrics-recording block:

• Compute duration_ms from start_time to _utc_now().
• Derive tokens_used: for the react engine read react_agent._total_tokens_used; for plan, sum tokens from any token_usage records if available, otherwise 0.
• Derive gates_passed: check state.gate_results — if non-empty, 1 if all passed, 0 if any failed; None if no gates ran.
• Derive self_corrections: use state.attempt_count - 1 for the plan engine (since first attempt is not a retry), 0 for react (ReactAgent handles retries internally and doesn't expose the count in the current AgentStatus return).
• Call engine_stats.record_run(workspace, run.id, engine, run.task_id, state.status.value, duration_ms, tokens_used, gates_passed, self_corrections) — import engine_stats lazily inside the function to avoid circular imports, matching the pattern already used for other imports in execute_agent().

Wrap the call in a try/except Exception with a logger.warning so a metrics failure never breaks task execution.

---

4. Create the engines CLI Sub-App

Create file:codeframe/cli/engines_commands.py following the exact same pattern as file:codeframe/cli/metrics_commands.py:

engines_app = typer.Typer(name="engines", help="Engine performance tracking", no_args_is_help=True)

stats command:

• Signature: engine: Optional[str] = typer.Option(None, "--engine", "-e"), plus a --format option (text/json).
• Resolves the workspace using get_workspace(Path.cwd()) (same pattern as work commands in app.py).
• Calls engine_stats.get_engine_stats(workspace, engine=engine).
• In text mode, renders a rich.table.Table with columns: Engine, Tasks, Success %, Gate Pass %, Self-Correction %, Avg Duration, Avg Tokens — one row per engine.
• Shows "No engine stats recorded yet" if table is empty.

compare command:

• Same as stats but always shows all engines side-by-side (no --engine filter), sorted by success rate descending.
• Renders the same rich.table.Table layout.

Both commands import from codeframe.core.workspace and codeframe.core.engine_stats. Handle FileNotFoundError (no workspace) with a clear error message and raise typer.Exit(1).

---

5. Register the New Sub-App

In file:codeframe/cli/app.py, at the imports section add:

from codeframe.cli.engines_commands import engines_app

In the sub-app registration block (around line 4780), add:

app.add_typer(engines_app, name="engines")

This makes cf engines stats and cf engines compare available as documented in the ticket.

---

Data Flow Summary

sequenceDiagram
    participant CLI as cf work start
    participant Runtime as runtime.execute_agent()
    participant Agent as ReactAgent / Agent
    participant EngStats as engine_stats.record_run()
    participant DB as .codeframe/state.db

    CLI->>Runtime: execute_agent(workspace, run, engine="react")
    Runtime->>Agent: agent.run(task_id)
    Agent-->>Runtime: AgentState (status, gate_results, _total_tokens_used)
    Runtime->>Runtime: compute duration_ms, tokens_used, gates_passed
    Runtime->>EngStats: record_run(workspace, run_id, engine, ...)
    EngStats->>DB: INSERT INTO run_engine_log
    EngStats->>EngStats: _update_aggregate_stats()
    EngStats->>DB: INSERT OR REPLACE INTO engine_stats (upsert each metric)

    CLI->>CLI: cf engines stats
    CLI->>EngStats: get_engine_stats(workspace)
    EngStats->>DB: SELECT FROM engine_stats
    DB-->>CLI: metrics per engine → Rich table

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The workspace uses a standalone SQLite database at `.codeframe/state.db`, managed in `file:codeframe/core/workspace.py` via `_init_database()` and `_ensure_schema_upgrades()`. Schema changes are made with raw SQLite—no ORM. `file:codeframe/core/runtime.py`'s `execute_agent()` already receives the `engine` param and the final `AgentState` (which has `gate_results` and `attempt_count`). `ReactAgent` tracks `_total_tokens_used` internally. CLI sub-apps are registered in `file:codeframe/cli/app.py` using `typer` and `rich` tables—the same pattern the new `engines` sub-app must follow.

## Approach

Add two new SQLite tables (`run_engine_log`, `engine_stats`) via the existing schema upgrade path in `workspace.py`, then wire recording logic into `execute_agent()` in `runtime.py`, and finally create a new `engines` Typer sub-app (a new file `file:codeframe/cli/engines_commands.py`) registered in `app.py`. No new libraries are needed—`typer`, `rich`, and `sqlite3` are already used.

---

## Implementation Steps

### 1. Extend the Database Schema

In `file:codeframe/core/workspace.py`, add the two new tables inside **both** `_init_database()` and `_ensure_schema_upgrades()` so existing workspaces get them automatically.

**In `_init_database()`**, add two new `CREATE TABLE IF NOT EXISTS` blocks after the `run_logs` block:

- **`run_engine_log`**: columns `run_id TEXT PRIMARY KEY`, `engine TEXT`, `task_id TEXT`, `workspace_id TEXT`, `status TEXT` (COMPLETED/FAILED/BLOCKED), `duration_ms INTEGER`, `tokens_used INTEGER`, `gates_passed INTEGER` (1/0/NULL), `self_corrections INTEGER`, `created_at TEXT`, with a `FOREIGN KEY (run_id) REFERENCES runs(id)`.

- **`engine_stats`**: columns `workspace_id TEXT`, `engine TEXT`, `metric TEXT`, `value REAL`, `updated_at TEXT`, with `PRIMARY KEY (workspace_id, engine, metric)`.

Also add performance indexes:
- `idx_run_engine_log_workspace_engine ON run_engine_log(workspace_id, engine)`
- `idx_engine_stats_workspace ON engine_stats(workspace_id, engine)`

**In `_ensure_schema_upgrades()`**, add idempotent `CREATE TABLE IF NOT EXISTS` calls for both tables (same DDL), so they appear in existing workspaces the first time `get_workspace()` is called.

---

### 2. Create an Engine Stats Helper Module

Create `file:codeframe/core/engine_stats.py` with the following functions (all using `get_db_connection(workspace)` exactly as `runtime.py` does):

- **`record_run(workspace, run_id, engine, task_id, status, duration_ms, tokens_used, gates_passed, self_corrections)`** — inserts a row into `run_engine_log` and then calls `_update_aggregate_stats()`.

- **`_update_aggregate_stats(workspace, engine)`** — private function that computes all aggregate metrics from `run_engine_log` for the given engine+workspace (tasks_attempted, tasks_completed, tasks_failed, gate_pass_rate, self_correction_rate, avg_duration_ms, total_tokens, avg_tokens_per_task) and upserts each as a row in `engine_stats` using `INSERT OR REPLACE`.

- **`get_engine_stats(workspace, engine=None)`** → returns a `dict[str, dict[str, float]]` keyed by engine name. If `engine` is provided, filters to that engine only. Reads from the `engine_stats` table.

- **`get_run_log(workspace, engine=None, limit=100)`** → returns raw per-run records from `run_engine_log` for display in a detailed view.

---

### 3. Wire Recording into `execute_agent()`

In `file:codeframe/core/runtime.py`, update `execute_agent()` to capture and record metrics after the agent completes. The timing, status, token count, and gate results are all already available in that function.

**At the top of the `try` block**, capture `start_time` using `_utc_now()` (already a helper in that module).

**After the final status-based `complete_run` / `fail_run` / `block_run` call** (around line 866), add a metrics-recording block:

- Compute `duration_ms` from `start_time` to `_utc_now()`.
- Derive `tokens_used`: for the `react` engine read `react_agent._total_tokens_used`; for `plan`, sum tokens from any `token_usage` records if available, otherwise `0`.
- Derive `gates_passed`: check `state.gate_results` — if non-empty, `1` if all passed, `0` if any failed; `None` if no gates ran.
- Derive `self_corrections`: use `state.attempt_count - 1` for the `plan` engine (since first attempt is not a retry), `0` for `react` (ReactAgent handles retries internally and doesn't expose the count in the current `AgentStatus` return).
- Call `engine_stats.record_run(workspace, run.id, engine, run.task_id, state.status.value, duration_ms, tokens_used, gates_passed, self_corrections)` — import `engine_stats` lazily inside the function to avoid circular imports, matching the pattern already used for other imports in `execute_agent()`.

Wrap the call in a `try/except Exception` with a `logger.warning` so a metrics failure never breaks task execution.

---

### 4. Create the `engines` CLI Sub-App

Create `file:codeframe/cli/engines_commands.py` following the exact same pattern as `file:codeframe/cli/metrics_commands.py`:

```
engines_app = typer.Typer(name="engines", help="Engine performance tracking", no_args_is_help=True)
```

**`stats` command**:
- Signature: `engine: Optional[str] = typer.Option(None, "--engine", "-e")`, plus a `--format` option (`text`/`json`).
- Resolves the workspace using `get_workspace(Path.cwd())` (same pattern as `work` commands in `app.py`).
- Calls `engine_stats.get_engine_stats(workspace, engine=engine)`.
- In text mode, renders a `rich.table.Table` with columns: Engine, Tasks, Success %, Gate Pass %, Self-Correction %, Avg Duration, Avg Tokens — one row per engine.
- Shows "No engine stats recorded yet" if table is empty.

**`compare` command**:
- Same as `stats` but always shows all engines side-by-side (no `--engine` filter), sorted by success rate descending.
- Renders the same `rich.table.Table` layout.

Both commands import from `codeframe.core.workspace` and `codeframe.core.engine_stats`. Handle `FileNotFoundError` (no workspace) with a clear error message and `raise typer.Exit(1)`.

---

### 5. Register the New Sub-App

In `file:codeframe/cli/app.py`, at the imports section add:
```python
from codeframe.cli.engines_commands import engines_app
```

In the sub-app registration block (around line 4780), add:
```python
app.add_typer(engines_app, name="engines")
```

This makes `cf engines stats` and `cf engines compare` available as documented in the ticket.

---

### Data Flow Summary

```mermaid
sequenceDiagram
    participant CLI as cf work start
    participant Runtime as runtime.execute_agent()
    participant Agent as ReactAgent / Agent
    participant EngStats as engine_stats.record_run()
    participant DB as .codeframe/state.db

    CLI->>Runtime: execute_agent(workspace, run, engine="react")
    Runtime->>Agent: agent.run(task_id)
    Agent-->>Runtime: AgentState (status, gate_results, _total_tokens_used)
    Runtime->>Runtime: compute duration_ms, tokens_used, gates_passed
    Runtime->>EngStats: record_run(workspace, run_id, engine, ...)
    EngStats->>DB: INSERT INTO run_engine_log
    EngStats->>EngStats: _update_aggregate_stats()
    EngStats->>DB: INSERT OR REPLACE INTO engine_stats (upsert each metric)

    CLI->>CLI: cf engines stats
    CLI->>EngStats: get_engine_stats(workspace)
    EngStats->>DB: SELECT FROM engine_stats
    DB-->>CLI: metrics per engine → Rich table
```

---

Execution Information

Branch: main
Commit: 705b160

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

🔗 Related PRs

#126 - Add token tracking to WorkerAgent execute_task method [merged]
#151 - Integrate QualityTracker with workflow system [merged]
#152 - Implement agent maturity level system [merged]
#363 - feat: wire ReactAgent into runtime with --engine react CLI flag [merged]
#387 - [Phase 2.5-F] Wire engine parameter through v2 API routes (#354) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!