docs: freeze the 2026-08-05 Phase 1 product-quality baseline - #75
Merged
Conversation
Records the first dated baseline from scripts/quality_scorecard.py, bound to commit 8a0536b and the shipped catalog digest. Network-free, corpus-free, protocol and catalog counts only. Also fixes a real defect the commit-receipt gate caught. The memo printed the catalog digest as a bare backticked hex string, which is exactly the form test_backticked_commit_receipts_exist resolves as a git commit across CHANGELOG.md, docs/, and validation/. A content hash is not a commit, so the gate failed. The ambiguity was not only mechanical: a reader seeing two backticked hex values on one line would reasonably take both for commits. The digest now carries its sha256: prefix, and a regression test asserts the memo never emits a non-commit hash in receipt form. Full check.py passes all 26 stages.
There was a problem hiding this comment.
Pull request overview
Freezes a dated, aggregate-safe “Phase 1 product-quality baseline” artifact generated by scripts/quality_scorecard.py, and hardens the scorecard memo format so catalog digests cannot be misread (or mechanically interpreted) as git commit receipts.
Changes:
- Add a committed baseline memo under
validation/with MCP discovery/context byte counts and catalog surface counts. - Update
scripts/quality_scorecard.pyto prefix the catalog digest withsha256:in the rendered memo receipt line. - Add a regression test ensuring the memo never emits the catalog digest as a bare backticked hex “commit receipt”.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
validation/2026-08-05-quality-baseline-scorecard.md |
Adds the frozen Phase 1 baseline memo (aggregate-safe context/cost + catalog-surface counts). |
tests/test_quality_scorecard.py |
Adds a regression test preventing bare backticked digest hex from appearing in the memo. |
scripts/quality_scorecard.py |
Changes memo rendering to include sha256: prefix for the catalog digest receipt. |
.gitignore |
Whitelists the new committed baseline memo under validation/. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| assert entry["metric"] in memo | ||
|
|
||
|
|
||
| def test_markdown_memo_never_prints_a_non_commit_hash_as_a_commit_receipt(scorecard: dict[str, Any]) -> None: |
Comment on lines
+167
to
+172
| memo = _render_markdown(scorecard) | ||
| receipts = set(re.findall(r"`([0-9a-f]{7,40})`", memo)) | ||
| digest = scorecard["revisions"]["catalog_digest_sha256"] or "" | ||
|
|
||
| assert digest[:12] not in receipts | ||
| assert f"`sha256:{digest[:12]}`" in memo |
| | **Session context before first call** | **91,876 bytes** | | ||
| | Order-of-magnitude tokens | ~22,969 | | ||
|
|
||
| Where the discovery payload goes: |
| | `annotations` | 1,945 | 2.4% | | ||
| | `name` | 441 | 0.5% | | ||
|
|
||
| Output schemas are 63.1% of discovery. Removing them entirely would leave 30,170 bytes, which is an upper bound on headroom rather than a proposal: structured output is part of the contract clients validate against. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the first dated baseline from
scripts/quality_scorecard.py, bound to commit8a0536band the shipped catalog digest. Network-free, corpus-free, protocol and catalog counts only — no target names, records, tenant IDs, or rows.This is the Phase 1 promotion evidence: "one dated aggregate-safe scorecard, reproduction commands, environment, revision digests, and no target rows."
Frozen numbers
A real defect the gate caught
The memo originally printed the catalog digest as a bare backticked hex string — exactly the form
test_backticked_commit_receipts_existresolves as a git commit acrossCHANGELOG.md,docs/, andvalidation/. A content hash is not a commit, so the gate failed.The ambiguity was not only mechanical. A reader seeing two backticked hex values on one line would reasonably take both for commit receipts:
Now:
A regression test asserts the memo never emits a non-commit hash in receipt form, so a future field carrying a digest cannot silently reintroduce this.
This also identifies the intermittent
pytest+covfailure I saw twice earlier in this work as very likely the same test rather than a Hypothesis flake — that test shells out togit cat-fileand is therefore sensitive to repository state. I did not capture the output of those two runs, so I am not claiming it retroactively; every backticked receipt currently in the docs resolves.Verification
uv run python scripts/check.py— all 26 stages passuv run pytest tests/test_quality_scorecard.py tests/test_documentation_integrity.py -q— 28 passedCHANGELOG.md,docs/, andvalidation/resolves as a commit