feat(BACK-001): counterfactual provider and evidence export - #13
Merged
Merged
Conversation
- Add backtest_harness.evidence module with run_counterfactual, build_failure_evidence, to_verification_result, to_evidence_chain_link - Extend provider_receipts with canonical_hash (ADR-021 byte-for-byte compatibility) and PROVIDER_VERSION - Add tearsheet overloads for precise mypy typing - Add conformance tests against verdict-core contracts (VerificationResult, EvidenceChainLink, ProviderReceipt) - Update ARCHITECTURE.md with BACK-001 evidence flow documentation Refs #11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements [BACK-001] "Backtest counterfactual provider and evidence export" (issue #11). Exposes Monte Carlo, fee, walk-forward, and tear-sheet results as verification / counterfactual evidence compatible with Verdict Core's
VerificationResultandEvidenceChainLinkcontracts (ADR-021 / ADR-024).Changes
src/backtest_harness/evidence.py(new): deterministic, seededrun_counterfactual, explicitbuild_failure_evidencefor failure/timeout/denied/cancelled/unknown states, and Core adaptersto_verification_result/to_evidence_chain_link.src/backtest_harness/provider_receipts.py: addedcanonical_hash(byte-for-byte identical toverdict-core.canonical_hash) andPROVIDER_VERSION.src/backtest_harness/analytics.py: precisetearsheetoverloads so mypy narrows theas_dict=Truereturn todict.docs/ARCHITECTURE.md: documented the BACK-001 counterfactual evidence flow.test_evidence.py,test_provider_receipts.py(conformance mirror of core),test_core_conformance.py(boundary conformance against realverdict.contracts.VerificationResult/EvidenceChainLink/verdict.provider_receipts.ProviderReceipt).Acceptance-criteria evidence
run_counterfactualrecordsdataset_ref,seed,fee_config["model"],code_version, andtrade_returns_hashin the receipt; results are bound byevidence_refs=(results_hash,). Pure in-process NumPy only; RNG state is saved/restored so callers are unaffected (test_does_not_leak_global_rng_state). Identical args → identical bundle (test_same_seed_reproduces_identical_evidence); different seed → different results.test_core_conformance.pyround-trips the receipt throughverdict.provider_receipts.ProviderReceipt.from_dict, exports throughverdict.contracts.VerificationResult.from_dict(statuspassed/failed) andverdict.contracts.EvidenceChainLink.from_dict.build_failure_evidenceallowlists non-success outcomes;successcannot be fabricated;to_verification_resultmaps onlysuccess→passed, failure/denied/error →failed, timeout/unknown →unknown, cancelled/skipped →skipped.to_verification_resultderives status solely from the recordedoutcome; tampered advisory detail fields (status: "passed") are ignored (test_advisory_details_cannot_upgrade_status).to_evidence_chain_linkrejects empty decision/policy/envelope and cannot mint authority (test_provider_cannot_mint_decision_authority)._reject_sensitiverejectsapi_key/authorization/password/secret/tokenat the receipt boundary (test_sensitive_provenance_is_rejected,test_receipt_rejects_sensitive_metadata).run_id/dataset_ref, non-positive equity/sim counts, bad walk-forward splits, unknown fee model, fee_trades without config, malformed payloads, tampered digests.Verification
uv run python -m pytest -q→ 66 passed (incl. core conformance).uv run ruff check .→ All checks passed.uv run ruff format --check .→ formatted.uv run mypy src/→ no issues.git diff --check→ clean.Closes #11.