Proof commands need end-to-end integration tests

[frankbria]

Summary

The PROOF9 quality system (`cf proof run`, `cf proof capture`, `cf proof waive`, etc.) is a flagship differentiator highlighted in the README and VISION. However, the proof commands shipped in #422 without integration-level CLI tests validating the full capture → enforce → waive → status loop.

Unit tests for the core module exist, but there are no tests in `tests/cli/` that exercise `proof_commands.py` through the Typer CLI runner, and no test that verifies the closed loop (capture a glitch → run proof → see it enforced).

Acceptance Criteria

• CLI integration test: `cf proof capture` creates a REQ and persists it
• CLI integration test: `cf proof run` evaluates the workspace against open REQs
• CLI integration test: `cf proof waive` marks a REQ waived with expiry
• CLI integration test: `cf proof status` shows correct summary
• At least one test covering the closed loop: capture → run → see obligation enforced

References

• `codeframe/cli/proof_commands.py` — 6 commands
• `tests/cli/` — existing CLI test pattern to follow
• `tests/core/test_proof*.py` — unit tests that already exist (add integration layer on top)

Why Pre-Launch

PROOF9 is marketed as the core differentiator. If it silently fails in the CLI for a new user, it undermines the product's main claim.

[coderabbitai]

🔗 Related PRs

#304 - test(cli): add comprehensive v2 CLI integration test suite [merged]
#317 - feat(cli): add interactive PRD generation via Socratic discovery [merged]
#369 - test: add CLI integration tests for ReactAgent runtime parameters [merged]
#386 - [Phase 2.5-F] Harden verification gates and add batch-level QA [merged]
#447 - feat(proof): PROOF9 quality memory system (#422) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[traycerai]

Plan

Observations

• tests/cli/ does not exist — it must be created from scratch.
• The existing CLI tests live in tests/core/test_proof9.py under TestCLI, using typer.testing.CliRunner and tmp_path + create_or_load_workspace. That is the pattern to follow.
• proof_app is registered on the main app in file:codeframe/cli/app.py via app.add_typer(proof_app, name="proof").
• _run_gate in file:codeframe/core/proof/runner.py is the seam to mock for cf proof run tests (same approach used in TestRunner in file:tests/core/test_proof9.py).
• pytest.ini already declares the integration and v2 markers; the new file should use @pytest.mark.integration and @pytest.mark.v2.
• typer, pytest, and unittest.mock are all available in the project.

Approach

Create a single new file tests/cli/test_proof_commands.py (plus the required tests/cli/__init__.py) that exercises every acceptance criterion through the Typer CliRunner, using tmp_path as the isolated workspace and patching _run_gate where real subprocess execution is not desired. This mirrors the pattern already established in TestCLI inside test_proof9.py and keeps the integration layer cleanly separated from the unit layer.

---

Implementation Steps

1. Create tests/cli/__init__.py

Create an empty tests/cli/__init__.py so pytest discovers the package correctly (matching the pattern of tests/__init__.py).

---

2. Create tests/cli/test_proof_commands.py

This file contains all integration tests. Structure it as follows:

Module-level setup

• Import pytest, Path, patch from unittest.mock, CliRunner from typer.testing, and app from codeframe.cli.app.
• Apply pytestmark = pytest.mark.integration and pytest.mark.v2 at module level.

Shared fixture: workspace_dir

• A pytest.fixture that accepts tmp_path and calls create_or_load_workspace(tmp_path) (same as the unit-test fixture in test_proof9.py), then yields tmp_path. This gives each test a clean, isolated workspace directory to pass as --workspace.

Shared fixture: cli_runner

• A pytest.fixture returning CliRunner() so every test reuses the same runner instance.

---

AC1 — cf proof capture creates a REQ and persists it

test_capture_creates_req_and_persists

• Invoke app with ["proof", "capture", "-w", str(workspace_dir), "--title", ..., "--description", ..., "--where", "src/auth.py", "--severity", "high", "--source", "qa"].
• Assert result.exit_code == 0.
• Assert "REQ-0001" appears in result.output.
• After the CLI call, directly call ledger.get_requirement(workspace, "REQ-0001") and assert it is not None and its title matches — confirming persistence, not just output.

test_capture_invalid_severity_exits_nonzero

• Invoke with --severity "extreme" (invalid enum value).
• Assert result.exit_code != 0.

---

AC2 — cf proof run evaluates the workspace against open REQs

test_run_with_passing_obligations

• First invoke capture to create REQ-0001.
• Patch codeframe.core.proof.runner._run_gate to return (True, "ok").
• Invoke ["proof", "run", "-w", str(workspace_dir), "--full"].
• Assert result.exit_code == 0.
• Assert "PASS" appears in result.output.
• Assert "All obligations satisfied" appears in result.output.

test_run_with_failing_obligations

• Same setup, but patch _run_gate to return (False, "failed").
• Assert result.exit_code == 1 (the command calls raise typer.Exit(1) on failure).
• Assert "FAIL" appears in result.output.

test_run_invalid_gate_exits_nonzero

• Invoke ["proof", "run", "-w", str(workspace_dir), "--gate", "nonexistent"].
• Assert result.exit_code != 0.
• Assert "Unknown gate" appears in result.output.

test_run_no_obligations_exits_zero

• Invoke run on an empty workspace (no REQs captured).
• Assert result.exit_code == 0.
• Assert "No applicable obligations found" appears in result.output.

---

AC3 — cf proof waive marks a REQ waived with expiry

test_waive_marks_req_waived

• Capture REQ-0001 first.
• Invoke ["proof", "waive", "REQ-0001", "-w", str(workspace_dir), "--reason", "No automated test yet", "--expires", "2027-01-01"].
• Assert result.exit_code == 0.
• Assert "waived" appears in result.output.
• Directly call ledger.get_requirement(workspace, "REQ-0001") and assert req.status == ReqStatus.WAIVED and req.waiver.reason == "No automated test yet".

test_waive_nonexistent_req_exits_nonzero

• Invoke waive for "REQ-9999" (never captured).
• Assert result.exit_code == 1.

test_waive_invalid_date_exits_nonzero

• Invoke with --expires "not-a-date".
• Assert result.exit_code == 1.
• Assert "Invalid date format" appears in result.output.

---

AC4 — cf proof status shows correct summary

test_status_empty_workspace

• Invoke ["proof", "status", "-w", str(workspace_dir)] on a fresh workspace.
• Assert result.exit_code == 0.
• Assert "No proof requirements" appears in result.output.

test_status_shows_open_count

• Capture one REQ.
• Invoke status.
• Assert result.exit_code == 0.
• Assert "1" and "Open" appear in result.output.

test_status_shows_waived_count

• Capture a REQ, then waive it.
• Invoke status.
• Assert "Waived" count is 1 in the output.

test_status_expired_waiver_reverts_to_open

• Capture a REQ, then directly call ledger.waive_requirement with a past expiry date (e.g., date(2020, 1, 1)).
• Invoke status.
• Assert "Expired" appears in result.output (the command prints the expired waiver notice) and the open count is 1.

---

AC5 — Closed-loop: capture → run → see obligation enforced

test_closed_loop_capture_run_enforced

This is the flagship integration test:

1. Invoke capture with all required flags → assert exit_code == 0 and "REQ-0001" in output.
2. Patch _run_gate to return (False, "assertion failed").
3. Invoke run --full → assert exit_code == 1 (obligation enforced / blocked).
4. Assert "FAIL" and "REQ-0001" appear in the run output.
5. Invoke status → assert "Open" count is 1 (REQ is still open because run failed).
6. Now patch _run_gate to return (True, "all green").
7. Invoke run --full again → assert exit_code == 0.
8. Assert "PASS" and "All obligations satisfied" in output.

This single test validates the full capture → enforce → pass loop through the CLI.

---

3. Verify tests/cli/ is picked up by pytest

pytest.ini sets testpaths = tests, which recursively discovers tests/cli/. No configuration change is needed, but confirm the new __init__.py is present so imports resolve correctly.

---

File Summary

File	Action
file:tests/cli/__init__.py	Create (empty)
file:tests/cli/test_proof_commands.py	Create — all integration tests described above

No changes to file:codeframe/cli/proof_commands.py, file:codeframe/core/proof/runner.py, or any other source file are required.

Import In IDE

🤖 Prompt for AI Agents

## Observations

- `tests/cli/` **does not exist** — it must be created from scratch.
- The existing CLI tests live in `tests/core/test_proof9.py` under `TestCLI`, using `typer.testing.CliRunner` and `tmp_path` + `create_or_load_workspace`. That is the pattern to follow.
- `proof_app` is registered on the main `app` in `file:codeframe/cli/app.py` via `app.add_typer(proof_app, name="proof")`.
- `_run_gate` in `file:codeframe/core/proof/runner.py` is the seam to mock for `cf proof run` tests (same approach used in `TestRunner` in `file:tests/core/test_proof9.py`).
- `pytest.ini` already declares the `integration` and `v2` markers; the new file should use `@pytest.mark.integration` and `@pytest.mark.v2`.
- `typer`, `pytest`, and `unittest.mock` are all available in the project.

## Approach

Create a single new file `tests/cli/test_proof_commands.py` (plus the required `tests/cli/__init__.py`) that exercises every acceptance criterion through the Typer `CliRunner`, using `tmp_path` as the isolated workspace and patching `_run_gate` where real subprocess execution is not desired. This mirrors the pattern already established in `TestCLI` inside `test_proof9.py` and keeps the integration layer cleanly separated from the unit layer.

---

## Implementation Steps

### 1. Create `tests/cli/__init__.py`

Create an empty `tests/cli/__init__.py` so pytest discovers the package correctly (matching the pattern of `tests/__init__.py`).

---

### 2. Create `tests/cli/test_proof_commands.py`

This file contains all integration tests. Structure it as follows:

#### Module-level setup

- Import `pytest`, `Path`, `patch` from `unittest.mock`, `CliRunner` from `typer.testing`, and `app` from `codeframe.cli.app`.
- Apply `pytestmark = pytest.mark.integration` and `pytest.mark.v2` at module level.

#### Shared fixture: `workspace_dir`

- A `pytest.fixture` that accepts `tmp_path` and calls `create_or_load_workspace(tmp_path)` (same as the unit-test fixture in `test_proof9.py`), then yields `tmp_path`. This gives each test a clean, isolated workspace directory to pass as `--workspace`.

#### Shared fixture: `cli_runner`

- A `pytest.fixture` returning `CliRunner()` so every test reuses the same runner instance.

---

#### AC1 — `cf proof capture` creates a REQ and persists it

**`test_capture_creates_req_and_persists`**

- Invoke `app` with `["proof", "capture", "-w", str(workspace_dir), "--title", ..., "--description", ..., "--where", "src/auth.py", "--severity", "high", "--source", "qa"]`.
- Assert `result.exit_code == 0`.
- Assert `"REQ-0001"` appears in `result.output`.
- After the CLI call, directly call `ledger.get_requirement(workspace, "REQ-0001")` and assert it is not `None` and its `title` matches — confirming persistence, not just output.

**`test_capture_invalid_severity_exits_nonzero`**

- Invoke with `--severity "extreme"` (invalid enum value).
- Assert `result.exit_code != 0`.

---

#### AC2 — `cf proof run` evaluates the workspace against open REQs

**`test_run_with_passing_obligations`**

- First invoke `capture` to create `REQ-0001`.
- Patch `codeframe.core.proof.runner._run_gate` to return `(True, "ok")`.
- Invoke `["proof", "run", "-w", str(workspace_dir), "--full"]`.
- Assert `result.exit_code == 0`.
- Assert `"PASS"` appears in `result.output`.
- Assert `"All obligations satisfied"` appears in `result.output`.

**`test_run_with_failing_obligations`**

- Same setup, but patch `_run_gate` to return `(False, "failed")`.
- Assert `result.exit_code == 1` (the command calls `raise typer.Exit(1)` on failure).
- Assert `"FAIL"` appears in `result.output`.

**`test_run_invalid_gate_exits_nonzero`**

- Invoke `["proof", "run", "-w", str(workspace_dir), "--gate", "nonexistent"]`.
- Assert `result.exit_code != 0`.
- Assert `"Unknown gate"` appears in `result.output`.

**`test_run_no_obligations_exits_zero`**

- Invoke `run` on an empty workspace (no REQs captured).
- Assert `result.exit_code == 0`.
- Assert `"No applicable obligations found"` appears in `result.output`.

---

#### AC3 — `cf proof waive` marks a REQ waived with expiry

**`test_waive_marks_req_waived`**

- Capture `REQ-0001` first.
- Invoke `["proof", "waive", "REQ-0001", "-w", str(workspace_dir), "--reason", "No automated test yet", "--expires", "2027-01-01"]`.
- Assert `result.exit_code == 0`.
- Assert `"waived"` appears in `result.output`.
- Directly call `ledger.get_requirement(workspace, "REQ-0001")` and assert `req.status == ReqStatus.WAIVED` and `req.waiver.reason == "No automated test yet"`.

**`test_waive_nonexistent_req_exits_nonzero`**

- Invoke `waive` for `"REQ-9999"` (never captured).
- Assert `result.exit_code == 1`.

**`test_waive_invalid_date_exits_nonzero`**

- Invoke with `--expires "not-a-date"`.
- Assert `result.exit_code == 1`.
- Assert `"Invalid date format"` appears in `result.output`.

---

#### AC4 — `cf proof status` shows correct summary

**`test_status_empty_workspace`**

- Invoke `["proof", "status", "-w", str(workspace_dir)]` on a fresh workspace.
- Assert `result.exit_code == 0`.
- Assert `"No proof requirements"` appears in `result.output`.

**`test_status_shows_open_count`**

- Capture one REQ.
- Invoke `status`.
- Assert `result.exit_code == 0`.
- Assert `"1"` and `"Open"` appear in `result.output`.

**`test_status_shows_waived_count`**

- Capture a REQ, then waive it.
- Invoke `status`.
- Assert `"Waived"` count is `1` in the output.

**`test_status_expired_waiver_reverts_to_open`**

- Capture a REQ, then directly call `ledger.waive_requirement` with a past expiry date (e.g., `date(2020, 1, 1)`).
- Invoke `status`.
- Assert `"Expired"` appears in `result.output` (the command prints the expired waiver notice) and the open count is `1`.

---

#### AC5 — Closed-loop: capture → run → see obligation enforced

**`test_closed_loop_capture_run_enforced`**

This is the flagship integration test:

1. Invoke `capture` with all required flags → assert `exit_code == 0` and `"REQ-0001"` in output.
2. Patch `_run_gate` to return `(False, "assertion failed")`.
3. Invoke `run --full` → assert `exit_code == 1` (obligation enforced / blocked).
4. Assert `"FAIL"` and `"REQ-0001"` appear in the run output.
5. Invoke `status` → assert `"Open"` count is `1` (REQ is still open because run failed).
6. Now patch `_run_gate` to return `(True, "all green")`.
7. Invoke `run --full` again → assert `exit_code == 0`.
8. Assert `"PASS"` and `"All obligations satisfied"` in output.

This single test validates the full capture → enforce → pass loop through the CLI.

---

### 3. Verify `tests/cli/` is picked up by pytest

`pytest.ini` sets `testpaths = tests`, which recursively discovers `tests/cli/`. No configuration change is needed, but confirm the new `__init__.py` is present so imports resolve correctly.

---

## File Summary

| File | Action |
|---|---|
| `file:tests/cli/__init__.py` | Create (empty) |
| `file:tests/cli/test_proof_commands.py` | Create — all integration tests described above |

No changes to `file:codeframe/cli/proof_commands.py`, `file:codeframe/core/proof/runner.py`, or any other source file are required.

---

Execution Information

Branch: main
Commit: 4308b0a

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.