Scope (minimum v1)
Deterministic benchmark harness runs the same published task via DIRECT API model vs Verdict route, reports cost / latency / completion / regression / failover. Reproducible when seeded.
Do not duplicate
This issue surfaces the existing benchmarking.py + reproducible_benchmarks.py modules (already shipped) and the PR #22 final V1-007 ticket.
Deferred (per audit, not for V1)
- Multi-model tournament scoring
- Provider-rotation benchmarks
- GNN evaluation harness
Refs: evidence/V1_READINESS_AUDIT_2026-08-03.md #239
Scope (minimum v1)
Deterministic benchmark harness runs the same published task via DIRECT API model vs Verdict route, reports cost / latency / completion / regression / failover. Reproducible when seeded.
Do not duplicate
This issue surfaces the existing benchmarking.py + reproducible_benchmarks.py modules (already shipped) and the PR #22 final V1-007 ticket.
Deferred (per audit, not for V1)
Refs: evidence/V1_READINESS_AUDIT_2026-08-03.md #239