Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ trading research systems. The compatible Python import namespace is
| --- | --- |
| `backtest_harness.fee_models` | Protocol and implementations for exchange fee calculations. |
| `backtest_harness.monte_carlo` | Equity-path sampling and risk percentile estimation. |
| `backtest_harness.analytics` | Tear-sheet statistics and walk-forward index splitting. |
| `backtest_harness.provider_receipts` | Deterministic, privacy-safe evaluation receipts (ADR-021). |
| `backtest_harness.evidence` | Counterfactual provider evidence and Core-compatible export (BACK-001). |

## Data flow

Expand Down Expand Up @@ -44,6 +47,34 @@ flowchart TD
LLM[verdict] --> KT
```

## Counterfactual evidence (BACK-001)

`backtest_harness.evidence` exposes deterministic, counterfactual backtest
evaluations as verification evidence compatible with Verdict Core's
`VerificationResult` and `EvidenceChainLink` contracts:

- **`run_counterfactual`** — Seeded, reproducible evaluation of a historical
returns series. Records inputs, random seed, fee model, code version, and
dataset reference in a receipt whose `evidence_refs` bind it to the canonical
hash of produced results (Monte Carlo, tear-sheet, walk-forward, fees). Pure
in-process math; no network, no live execution.
- **`build_failure_evidence`** — Explicit receipts for failure, timeout,
denied, cancelled, and unknown states. The absence of results is auditable;
fabricated success is impossible because the bundle carries none.
- **`to_verification_result`** — Export as a Core `VerificationResult` payload.
Status derives only from the recorded outcome; a non-success outcome can
never map to `"passed"`, and advisory detail fields cannot override the
mapping.
- **`to_evidence_chain_link`** — Export as a Core `EvidenceChainLink` payload.
All decision-authority fields (decision, policy, envelope hash, model,
timestamp) must be supplied by the caller; this provider is evidence-only and
cannot grant or fabricate authority.

The receipts implement ADR-021 (Deterministic Provider Evaluation Receipts):
identical evaluation inputs produce identical `inputs_hash` and `config_hash`,
enabling deterministic replay. Sensitive keys (`api_key`, `authorization`,
`password`, `secret`, `token`) are rejected at the receipt boundary.

## Design principles

- **Auditability over cleverness**: Backtest assumptions should be inspectable.
Expand Down
19 changes: 17 additions & 2 deletions src/backtest_harness/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,22 +27,37 @@
"""

from backtest_harness.analytics import split_walk_forward, tearsheet
from backtest_harness.evidence import (
build_failure_evidence,
run_counterfactual,
to_evidence_chain_link,
to_verification_result,
)
from backtest_harness.fee_models import (
BoundedProfitFeeModel,
FeeModel,
FlatMakerTakerModel,
)
from backtest_harness.monte_carlo import MonteCarloSimulator
from backtest_harness.provider_receipts import build_backtest_receipt
from backtest_harness.provider_receipts import (
PROVIDER_VERSION,
build_backtest_receipt,
canonical_hash,
)

__version__ = "0.2.0"
__version__ = PROVIDER_VERSION

__all__ = [
"BoundedProfitFeeModel",
"FeeModel",
"FlatMakerTakerModel",
"MonteCarloSimulator",
"build_backtest_receipt",
"build_failure_evidence",
"canonical_hash",
"run_counterfactual",
"split_walk_forward",
"tearsheet",
"to_evidence_chain_link",
"to_verification_result",
]
10 changes: 9 additions & 1 deletion src/backtest_harness/analytics.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

from collections.abc import Iterator
from math import sqrt
from typing import Any
from typing import Any, Literal, overload

import numpy as np

Expand All @@ -34,6 +34,14 @@ def _to_returns(returns: Any) -> np.ndarray:
return arr


@overload
def tearsheet(
returns: Any, periods_per_year: int = _TRADING_PERIODS, as_dict: Literal[True] = True
) -> dict[str, float | int]: ...
@overload
def tearsheet(
returns: Any, periods_per_year: int = _TRADING_PERIODS, as_dict: Literal[False] = False
) -> None: ...
def tearsheet(
returns: Any, periods_per_year: int = _TRADING_PERIODS, as_dict: bool = True
) -> dict | None:
Expand Down
Loading
Loading