feat: gate the Markdown rot that broke correlation.md for three weeks - #76
Merged
Conversation
On 2026-07-11 one commit converted 285 lines of working $-delimited math in docs/correlation.md into \( ... \) and \[ ... \], and replaced five sections with one-paragraph pointer stubs kept only to preserve an anchor. Both survived every gate for three weeks. PR #71 repaired the document and deliberately left the gate as follow-up. This is that gate. Neither failure was catchable by an existing check. GitHub does not render \( or \[ as math; it treats them as escaped literal brackets, drops the backslash, and prints raw LaTeX in running prose. Nothing validated delimiters at all. check_section_links.py verifies that a referenced section number resolves, which a pointer stub satisfies perfectly, so a hollowed-out section reads as a live cross-reference target while carrying nothing. Three checks, all skipping fenced code and inline code spans so documenting the patterns stays legal: math-delimiter reject \( and \[ outside code display-math reject an unclosed $$ block, which swallows the text after it pointer-stub reject a leaf section under 60 words whose body or heading is redirect language The stub heuristic was designed against evidence rather than intuition. Word count alone does not work: the real stubs ran 28 to 32 words while this repository has legitimate sections at 8 words ("Setup") and 11 ("Reporting vulnerabilities"). Redirect vocabulary is the separator. Validated by replaying 921b64a, the commit that introduced the stubs: 5 of 5 caught, 0 false positives across all 72 current Markdown files. Container headings that introduce subsections are exempt, since an empty body above a subsection is ordinary structure. The gate found a live defect on its first run. docs/statistical-assurance.md rendered its signed marginal entropy formula as literal bracketed LaTeX, in a file the correlation.md repair never touched. Fixed here. My own earlier grep for this pattern missed it, which is the argument for a real checker over an ad-hoc search. Also makes two test-quality fixes found while wiring this up. Finding.render raised ValueError on a path outside the repository root even though check_paths accepts arbitrary files; it now falls back to the absolute path. And the scorecard tests recomputed the MCP surface four times per module, spinning up four asyncio event loops, each allocating a Windows socket pair for its self-pipe. Under parallel workers that churn contributed to an intermittent socketpair failure in an unrelated MCP test. They now share one module-scoped measurement, which is also correct on its own terms since the measurement is deterministic for a revision. Wired into scripts/check.py and ci.yml together so local and CI parity holds. Full check.py passes all 27 stages.
There was a problem hiding this comment.
Pull request overview
Adds a new “doc-rot” gate to prevent Markdown regressions that silently render incorrectly on GitHub (broken LaTeX delimiters, unclosed $$ blocks, and hollow “pointer stub” sections), and wires it into both local scripts/check.py and CI. Also updates tests and fixes one existing doc rendering defect found by the new gate.
Changes:
- Add
scripts/check_doc_rot.py+tests/test_doc_rot.pyto enforce and test Markdown render/content integrity. - Integrate the new gate into
scripts/check.pyand.github/workflows/ci.yml. - Reduce test churn by reusing the scorecard’s MCP measurement, and fix a docs math block in
docs/statistical-assurance.md.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_quality_scorecard.py | Reuses a module-scoped MCP measurement to avoid repeated asyncio loop creation. |
| tests/test_doc_rot.py | Adds end-to-end and unit tests covering the new doc-rot checker behavior. |
| scripts/check.py | Registers the new doc-rot stage in the local check pipeline. |
| scripts/check_doc_rot.py | Implements Markdown scanning for broken math delimiters, unbalanced $$, and pointer stubs. |
| docs/statistical-assurance.md | Fixes display-math rendering by switching to $$ ... $$. |
| CHANGELOG.md | Documents the new gate and the doc fix it surfaced. |
| .github/workflows/ci.yml | Runs the new doc-rot checker in CI for parity with local checks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+45
to
+46
| _INLINE_CODE = re.compile(r"`[^`]*`") | ||
| _HEADING = re.compile(r"^(#{2,6})\s+(.*)$") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-07-11 one commit converted 285 lines of working
$-delimited math indocs/correlation.mdinto\( ... \)and\[ ... \], and replaced five sections with one-paragraph pointer stubs kept only to preserve an anchor. Both survived every gate for three weeks. PR #71 repaired the document and deliberately left the gate as follow-up. This is that gate.Why nothing caught it
Neither failure was catchable by an existing check:
\(or\[as math. It treats them as escaped literal brackets, drops the backslash, and prints raw LaTeX in running prose. Nothing validated delimiters at all.check_section_links.pyverifies that a referenced section number resolves — which a pointer stub satisfies perfectly. A hollowed-out section reads as a live cross-reference target while carrying nothing.Three checks
math-delimiter\(and\[outside codedisplay-math$$block, which swallows the text after itpointer-stubAll three skip fenced code blocks and inline code spans, so documenting the patterns stays legal — this PR's own files do it.
The stub heuristic was designed against evidence
Word count alone does not work. The real stubs ran 28 to 32 words, while this repository has legitimate sections at 8 words (
Setup) and 11 (Reporting vulnerabilities). Redirect vocabulary is what separates a terse section from a hollow one.Validated by replaying
921b64a, the commit that introduced the stubs:Container headings that introduce subsections are exempt, since an empty body above a subsection is ordinary structure.
It found a live defect on its first run
docs/statistical-assurance.mdrendered its signed marginal entropy formula as literal bracketed LaTeX — in a file the correlation.md repair never touched, still broken on main:Fixed here. Worth noting: my own earlier ad-hoc grep for this pattern missed it because the pattern was malformed. That is the argument for a real checker over a search.
Two test-quality fixes found while wiring this up
Finding.renderraisedValueErroron a path outside the repository root, even thoughcheck_pathsaccepts arbitrary files. It now falls back to the absolute path.socketpair()failure in an unrelated MCP test. They now share one module-scoped measurement, which is also correct on its own terms since the measurement is deterministic for a revision.Verification
uv run python scripts/check.py— all 27 stages pass (26 + newdoc-rot)uv run pytest tests/test_doc_rot.py -q— 13 passedscripts/check.pyandci.ymltogether so local/CI parity holds--explain,--plain,--json,--direct-probes), all six MCP client targets, and the version all match