feat: add the Phase 1 product-quality baseline scorecard

[blisspixel]

Phase 1 of the Quality Proof plan asks for one dated, revision-bound, aggregate-safe artifact recording what the product costs and covers, plus an explicit ledger of what it does not measure. This emits that artifact.

Network-free, corpus-free, and deliberately not a gate — a scorecard that blocks a merge becomes a number people tune rather than a number people read.

Scope was the design question

Most Phase 1 quantities already have an owner, and recomputing them would create a second set of numbers to keep consistent:

Quantity	Existing owner
Component latency and allocation	scripts/characterize_performance.py
Claim lineage and provenance	scripts/check_default_claim_audit.py
Corpus-derived catalog coverage	validation/catalog_baseline.py
Claim-family precision, benefit, safety	the frozen preregistration

So this measures only what nothing else measures, names the owner for what is covered, and records the rest as unmeasured with the reason.

What it refuses to measure

Some Phase 1 metrics cannot be produced honestly without the corpus or the network, and the important call was not to fake them.

Classified-versus-unclassified observable surface is the clearest case: computing it over synthetic fixtures would measure the fixtures, because the unclassified share would be whatever the fixture author chose to include. Same for CT marginal gain, end-to-end p50/p95, and degraded-source rate.

Phase 1 requires unmeasured channels to be named, because an absent metric reads as a passing one. The ledger is a first-class output and a test asserts it is never empty.

The one real gap: MCP context cost

This is the quantity with no existing owner, and it is the evidence the deferred core-versus-advanced profile decision depends on. The first run is more interesting than expected:

22 tools, 81,880 bytes of discovery + 9,996 bytes of instruction preamble = 91,876 bytes of session context before an agent does any work.

Where it goes:

Component	Bytes	Share
outputSchema	51,358	63.9%
description	21,867	27.2%
inputSchema	4,696	5.9%
annotations	1,945	2.4%
name	441	0.5%

This reframes the deferred decision. The assumed lever was cutting the tool count — but tool count is not where the bytes are. Dropping every output schema would cut discovery 63%, while the five largest tools carry only 47%. Cross-tool definition duplication is measured too and recovers just 6,424 bytes across 5 shared bodies, so a shared-definition scheme is not the lever either.

The without-output-schema figure is reported as an upper bound on headroom, not a proposal: structured output is part of the contract clients validate against. The honest read is that the headroom lives in the structured-output contract for the largest tools (reevaluate_domain alone is 14,729 bytes because it replays the full JSON result), which is a product decision needing a client-benefit measurement — exactly what the roadmap already gates it behind.

Catalog surface also puts a number on the known date backlog: 26 of 1,070 detection rules carry a verification date.

Measurement honesty

Byte counts are exact. The token figure is labeled an order-of-magnitude estimate with its divisor named, because the tokenizer belongs to the client and this project should not pretend to know it.

Every number binds to a code commit and a catalog digest, so a later comparison is meaningful.

Tests

17 contract tests covering the reconciliation properties that would make the artifact misleading if broken: per-tool costs ranked largest-first, headroom never exceeding the payload, rule counts summing to their totals, the unmeasured ledger never empty, JSON round-tripping, and rendering surviving a checkout with no git metadata. Duplicate accounting is tested to charge only the repeats and to keep same-name-different-body definitions separate.

Verification

• uv run python scripts/check.py — all 26 stages pass
• uv run pytest tests/test_quality_scorecard.py -q — 17 passed
• pyright strict clean

The dated baseline memo lands in a follow-up so its revision binding points at the merged commit rather than a revision where the harness did not exist.

[Copilot]

Pull request overview

Adds a Phase 1, revision-bound “product-quality baseline scorecard” artifact (JSON or markdown memo) intended to be aggregate-safe, network-free, and corpus-free, plus contract tests and documentation pointers so the artifact can be reproduced and compared over time.

Changes:

• Introduces scripts/quality_scorecard.py to emit a scorecard capturing MCP discovery/context cost and the shipped catalog’s static surface, plus explicit “measured elsewhere” and “unmeasured” ledgers.
• Adds contract tests to assert reconciliation/ordering properties and to prevent an empty “unmeasured” ledger.
• Updates roadmap/docs/changelog to reference the new harness and record the initial baseline numbers.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_quality_scorecard.py	Adds contract tests covering scorecard structure, reconciliation properties, and markdown rendering behavior.
scripts/quality_scorecard.py	Implements the scorecard generator and markdown memo renderer, including MCP context-cost measurement and catalog surface accounting.
ROADMAP.md	Notes the new Phase 1 baseline artifact harness in the plan narrative.
docs/README.md	Adds a pointer to the new scorecard harness alongside other key docs/artifacts.
CHANGELOG.md	Records the addition of the scorecard harness and summarizes initial measured baseline figures.

Suppressed comments (2)

scripts/quality_scorecard.py:219

• _catalog_surface() currently calls load_fingerprints(), which includes custom fingerprints (and may reflect local operator config). For a revision-bound, aggregate-safe "shipped catalog" surface, load only the built-in generated artifact instead.

    fingerprints = load_fingerprints()

scripts/quality_scorecard.py:290

• catalog_digest_sha256 is computed from _CATALOG_DIR (YAML directory), but the runtime catalog comes from the generated JSON artifact. This can make the scorecard claim it measured a different revision of the catalog than it actually used (and may be None in environments where the YAML sources aren’t present).

    return {
        "commit": _git("rev-parse", "HEAD"),
        "working_tree_dirty": bool(status) if status is not None else None,
        "catalog_digest_sha256": _directory_digest(_CATALOG_DIR),
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.