test: add accessibility testing guidance#390
Conversation
Add accessibility testing guidance across TDD, browser testing, CI, references, and test-engineer coverage analysis. Add a11y eval prompts and tune skill descriptions so routing remains stable at 100% rank-1.
nucliweb
left a comment
There was a problem hiding this comment.
Thanks for this, @summergan. The accessibility guidance itself is genuinely good and fills a real gap: the a11y testing patterns in references/testing-patterns.md, the accessibility-tree and keyboard-focus additions to browser-testing-with-devtools, the axe/pa11y CI guidance, the test-engineer coverage, and the three new a11y eval cases all hold together. That part I would happily see land.
My blocker is elsewhere, and I want to be concrete about it. Beyond the a11y work, this rewrites ~15 skill descriptions, and about 11 of them have nothing to do with accessibility. Those 11 inject the exact wording of that skills own eval prompt into its description:
| Skill (unrelated to a11y) | Added to description | That skills eval prompt |
|---|---|---|
| documentation-and-adrs | "queue design rationale" | "Document the architecture decision behind the queue design" |
| security-and-hardening | "file upload handlers" | "Audit this file upload handler for vulnerabilities" |
| git-workflow-and-versioning | "splitting a messy working tree into clean atomic commits" | "Split this messy working tree into clean atomic commits" |
| debugging-and-error-recovery | "a test passed yesterday and fails today" | "This test passed yesterday and fails today, figure out what broke" |
| source-driven-development | "Next.js or Stripe" | "Verify against the official Next.js docs..." / "...Stripe integration" |
| incremental-implementation | "thin slices behind a feature flag" | "Ship this change behind a feature flag in thin slices" |
That is what the "100% rank-1" comes from, and it is why the number is misleading rather than reassuring. evals/README.md calls this out directly: "paraphrase how users actually talk; dont copy the description (thats gaming the eval)." Copying the prompt into the description is the same failure in reverse: once a description contains its own eval prompt, Tier 2 is no longer measuring independent routing, it is measuring that the copy was done. The metric that made Tier 2 useful stops meaning anything.
There is also a production cost separate from the metric: each of these edits changes how that skill triggers for every user, and some narrow it. security-and-hardening goes from the broad "handling user input, authentication, data storage..." to leading with "file upload handlers", a single case that happens to be its eval prompt. That is a real routing regression dressed up as a metric win.
The clean line, and it is a clean line: a description edit that adds vocabulary for new a11y content in this PR is legitimate (that is exactly what test-engineer, browser-testing-with-devtools, ci-cd-and-automation, and test-driven-development do, and it is the same good move #358 made for the frontend/perf gaps). A description edit that adds the eval prompt of an unrelated skill should come out.
Suggested split: keep the accessibility guidance and its four a11y description edits; revert the ~11 unrelated description rewrites (documentation-and-adrs, security-and-hardening, git-workflow-and-versioning, debugging-and-error-recovery, source-driven-development, incremental-implementation, doubt-driven-development, observability-and-instrumentation, spec-driven-development, shipping-and-launch, using-agent-skills). If any of those descriptions has a real vocabulary gap worth fixing, that is its own focused PR with prompts that paraphrase rather than copy, not a rider on an accessibility change.
With those reverted, this is a clean, well-scoped a11y PR and Im a yes.
|
Coordination note: this overlaps #389 (same author, five shared files including the |
|
The accessibility content is welcome, it's coverage the testing skills should have. What gives me pause is the rank-1 rate jumping from 86% to 100%: that means this also retuned skill descriptions catalog-wide, and a perfect score across 76 prompts reads more like descriptions fitted to the eval prompts than a real routing win (the eval README calls out that exact anti-pattern). Can you split this in two: the a11y guidance on its own, and the description changes separately, each with a note on the real routing failure it fixes? 20 files for "a11y testing guidance" is a lot of surface, and I want to be sure the description tuning is honest improvement rather than test-fitting. |
|
+1 on splitting the accessibility work from the unrelated description tuning. The a11y guidance is the part worth preserving, and isolating it gives us a chance to tighten one important boundary in its verification model. |
Add accessibility testing guidance across TDD, browser testing, CI, references, and test-engineer coverage analysis. Add a11y eval prompts and tune skill descriptions so routing remains stable at 100% rank-1.