test: cover campaign outcome evidence edge cases - #36
Draft
cursor[bot] wants to merge 14 commits into
Draft
Conversation
Adds a pure campaign_outcome package for deriving explicit campaign outcome truth from runner-fed evidence. Final review now consumes a FinalReviewReadModel projected from CampaignOutcome instead of treating report_ok as campaign success. report_ok remains report-generation evidence only. Wires runner finalization through evaluate_campaign_outcome and project_final_review, adds outcome-based exit-code handling, and preserves measurement truth separately from report/artifact success. Adds focused evaluator/projection tests plus runner hardening regressions for report_ok false-success prevention and report-failure preservation.
Normalize explicit report status before report_ok, tighten recommendation authority to fully successful measurement evidence, preserve evaluator failure detail in projection, distinguish report/export failures from analysis failures, and add adversarial coverage for conflicting post-run truth.
Extract shared private post-run rendering helpers so legacy and read-model final-review entrypoints share presentation logic without changing outcome semantics, public renderer signatures, or UI behavior.
Preserve backend-startup failure domains during no-success finalization, exit interrupted campaigns with code 130, and align artifact test fixtures with canonical artifact constants.
Split campaign outcome synthesis into ordered private gates, reduce post-run review helper complexity, and remove duplicate metrics DTO drift by reusing the final-review metrics snapshot contract.
Manually port useful Cursor Bugbot regression coverage for pre-measurement aborts, fatal measurement failures, and campaign evidence aggregation while leaving stale or redundant bot-generated behavior out of the branch.
Add targeted docstrings for the campaign outcome contracts, evaluator, projector, runner evidence aggregation, and post-run UI rendering to clarify ownership of outcome truth and presentation boundaries.
Carry synthesized abort reasons through campaign outcome evaluation, treat explicit complete report status as authoritative, and route KeyboardInterrupt through evaluator-backed final review before exiting 130.
Use a single module import style for campaign outcome evaluator tests to satisfy CodeQL import hygiene while preserving behavior.
Guarantee interrupted campaigns exit 130 even if final review rendering fails, keep normal completion unreachable after interruption, narrow boundary-invalid measurement classification to OOM-backed evidence, and remove remaining artifact fixture drift.
Prioritize measurement failures over lifecycle-complete no-success states, share final-review rendering across normal and interrupted paths, and stop duplicating status truth inside evidence summaries.
Require enough OOM-backed aggregate capacity before treating invalid cycles as explained, remove the unreachable lifecycle-complete no-success outcome gate, and guarantee interrupted closeout exits 130 even when evidence fetch fails.
Project artifact display mode from campaign outcome truth so failed, aborted, and insufficient-evidence runs show diagnostics-oriented review instead of a full artifact table, while preserving full artifacts for success, partial, and degraded outcomes.
Co-authored-by: Joshua L. Davis Sr. <Mad-Labs42@users.noreply.github.com>
23 tasks
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Summary
Adds focused regression coverage for campaign outcome evaluation/projection edge cases.
Risky behavior now covered:
NO_EVIDENCE/INSUFFICIENT_EVIDENCEand cannot produce success-style review authority.Test files added/updated:
test_campaign_outcome_slice1.pyWhy these tests materially reduce regression risk:
Why
Recent campaign outcome code added new truth/projection paths where weak evidence or missing failure detail could regress into misleading final-review output.
Scope
Risk
Test-only change. It affects regression coverage for the campaign outcome evaluator/projection seam.
Validation
.\.venv\Scripts\python.exe .agent\scripts\helpers\verify_dev_contract.py --quickor CI equivalent)Ran:
QUANTMAP_LAB_ROOT=/tmp/quantmap-lab python3 -m ruff check test_campaign_outcome_slice1.py test_runner_campaign_outcome_hardening.py test_cli_ux_post_run_review.py test_cli_ux_yolo_review.pyQUANTMAP_LAB_ROOT=/tmp/quantmap-lab python3 -m pytest --cov=src.campaign_outcome --cov=src.runner --cov=src.ui --cov-report=term-missing test_campaign_outcome_slice1.py test_runner_campaign_outcome_hardening.py test_cli_ux_post_run_review.py test_cli_ux_yolo_review.pyAgent Surface
.agent/scripts/agent_surface_audit.pyis present in this branch, agent surface audit passedNotes
Dev dependencies were installed in the automation environment before validation because
pytestwas initially unavailable.