Skip to content

test: add accessibility testing guidance#390

Closed
summergan wants to merge 1 commit into
addyosmani:mainfrom
summergan:feature/accessibility-testing-guidance
Closed

test: add accessibility testing guidance#390
summergan wants to merge 1 commit into
addyosmani:mainfrom
summergan:feature/accessibility-testing-guidance

Conversation

@summergan

Copy link
Copy Markdown

Add accessibility testing guidance across TDD, browser testing, CI, references, and test-engineer coverage analysis. Add a11y eval prompts and tune skill descriptions so routing remains stable at 100% rank-1.

Add accessibility testing guidance across TDD, browser testing, CI, references, and test-engineer coverage analysis. Add a11y eval prompts and tune skill descriptions so routing remains stable at 100% rank-1.

@nucliweb nucliweb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @summergan. The accessibility guidance itself is genuinely good and fills a real gap: the a11y testing patterns in references/testing-patterns.md, the accessibility-tree and keyboard-focus additions to browser-testing-with-devtools, the axe/pa11y CI guidance, the test-engineer coverage, and the three new a11y eval cases all hold together. That part I would happily see land.

My blocker is elsewhere, and I want to be concrete about it. Beyond the a11y work, this rewrites ~15 skill descriptions, and about 11 of them have nothing to do with accessibility. Those 11 inject the exact wording of that skills own eval prompt into its description:

Skill (unrelated to a11y) Added to description That skills eval prompt
documentation-and-adrs "queue design rationale" "Document the architecture decision behind the queue design"
security-and-hardening "file upload handlers" "Audit this file upload handler for vulnerabilities"
git-workflow-and-versioning "splitting a messy working tree into clean atomic commits" "Split this messy working tree into clean atomic commits"
debugging-and-error-recovery "a test passed yesterday and fails today" "This test passed yesterday and fails today, figure out what broke"
source-driven-development "Next.js or Stripe" "Verify against the official Next.js docs..." / "...Stripe integration"
incremental-implementation "thin slices behind a feature flag" "Ship this change behind a feature flag in thin slices"

That is what the "100% rank-1" comes from, and it is why the number is misleading rather than reassuring. evals/README.md calls this out directly: "paraphrase how users actually talk; dont copy the description (thats gaming the eval)." Copying the prompt into the description is the same failure in reverse: once a description contains its own eval prompt, Tier 2 is no longer measuring independent routing, it is measuring that the copy was done. The metric that made Tier 2 useful stops meaning anything.

There is also a production cost separate from the metric: each of these edits changes how that skill triggers for every user, and some narrow it. security-and-hardening goes from the broad "handling user input, authentication, data storage..." to leading with "file upload handlers", a single case that happens to be its eval prompt. That is a real routing regression dressed up as a metric win.

The clean line, and it is a clean line: a description edit that adds vocabulary for new a11y content in this PR is legitimate (that is exactly what test-engineer, browser-testing-with-devtools, ci-cd-and-automation, and test-driven-development do, and it is the same good move #358 made for the frontend/perf gaps). A description edit that adds the eval prompt of an unrelated skill should come out.

Suggested split: keep the accessibility guidance and its four a11y description edits; revert the ~11 unrelated description rewrites (documentation-and-adrs, security-and-hardening, git-workflow-and-versioning, debugging-and-error-recovery, source-driven-development, incremental-implementation, doubt-driven-development, observability-and-instrumentation, spec-driven-development, shipping-and-launch, using-agent-skills). If any of those descriptions has a real vocabulary gap worth fixing, that is its own focused PR with prompts that paraphrase rather than copy, not a rider on an accessibility change.

With those reverted, this is a clean, well-scoped a11y PR and Im a yes.

@nucliweb

Copy link
Copy Markdown
Collaborator

Coordination note: this overlaps #389 (same author, five shared files including the test-driven-development description line). I've put the joint merge-order plan on #389 so it lives in one place. cc @addyosmani @federicobartoli

@addyosmani

Copy link
Copy Markdown
Owner

The accessibility content is welcome, it's coverage the testing skills should have. What gives me pause is the rank-1 rate jumping from 86% to 100%: that means this also retuned skill descriptions catalog-wide, and a perfect score across 76 prompts reads more like descriptions fitted to the eval prompts than a real routing win (the eval README calls out that exact anti-pattern).

Can you split this in two: the a11y guidance on its own, and the description changes separately, each with a note on the real routing failure it fixes? 20 files for "a11y testing guidance" is a lot of surface, and I want to be sure the description tuning is honest improvement rather than test-fitting.

@federicobartoli

Copy link
Copy Markdown
Collaborator

+1 on splitting the accessibility work from the unrelated description tuning. The a11y guidance is the part worth preserving, and isolating it gives us a chance to tighten one important boundary in its verification model.
The current examples occasionally describe semantic assertions as proof of actual assistive-technology behavior. For example, the component test named exposes the email field and validation error to assistive tech verifies that a dynamically rendered error has role="alert". That is useful coverage, and role="alert" is reasonable in this scenario, but the assertion proves the intended alert semantics—not that VoiceOver or NVDA actually announced the message.
I would not expect an AI agent to operate a screen reader as part of this workflow. Instead, the skill should make the evidence boundary explicit:
component tests verify role, accessible name, state, and live-region markup;
browser tests verify keyboard/focus behavior and accessibility-tree exposure;
when actual browser/assistive-technology interoperability matters, the agent identifies a manual verification step, records the relevant browser/AT combination and expected result, and does not claim it was verified unless it was actually performed.
Could the standalone a11y change reflect this distinction in three places?
Rename the component test so it claims alert semantics rather than confirmed AT exposure.
Separate “live-region markup/update” from “actual announcement” in the TDD layer map.
Update the eval expectation so role/label/live-region assertions are accepted as semantic evidence, while actual screen-reader behavior is either supported by AT evidence or explicitly handed off for manual verification.
This keeps the workflow executable by an agent without letting it overstate what automated evidence proves. WAI-ARIA APG similarly recommends testing the browser/assistive-technology combinations relevant to the target audience: https://www.w3.org/WAI/ARIA/apg/practices/read-me-first/#browser_and_AT_support

@summergan summergan closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants