feat(ax-bridge): walker candidate diagnostics on --debug (#660)#691
Merged
Conversation
Issue #660 PR A added top-level milestone events; PR B added the AX messaging timeout. The remaining empirical work — landing the SpringBoard-explicit walker change — was deferred in shaun0927's 2026-04-27 status comment because the failure-tree data needed to write it correctly was not available. This change extends the existing --debug machinery so the next live capture on a ko-KR Flutter cold-launch with the UNUserNotificationCenter sheet visible produces exactly that data. Stdout (the JSON dump payload) is unchanged — the additions are stderr JSON-line events only. New events: - walker_app_windows_enumerated: every AXWindow under Simulator.app with title/identifier/role/subrole. Proves whether a SpringBoard-hosted overlay renders in a sibling window the walker never visits. - walker_top_candidates: top-5 ContentCandidates by score with role, label, depth, score, and appSemanticsCount. Reveals the scoring landscape so a walker change can be evaluated against actual data rather than speculation. - walker_overlay_roles_seen: every AXSheet/AXAlert/AXSystemDialog/ AXSystemFloatingWindow/AXDialog/AXSystemAlert encountered during the walk regardless of whether it won the score race. This is the load- bearing signal: if the count is non-zero on a failing capture, the overlay subtree IS reachable from the matched window and the walker just needs to widen its content selection. If the count is zero, the overlay must live in a sibling window or a separate macOS process and the change has to enumerate AXWindows / CGWindowListCopyWindowInfo. - walker_winner / walker_winner_none: the chosen content root with its role/label/score, or an explicit "no winner" event matching the DEVICE_CONTENT_ROOT_EMPTY exit path. Bounded with WALKER_DEBUG_CANDIDATE_CAP=256 and WALKER_DEBUG_OVERLAY_CAP=32 so a degenerate tree cannot blow stderr volume. All emission gated on debugEnabled. Verified locally: - swiftc -O compiles clean - 51 unit tests across ax-bridge-content-root, ax-bridge-wrapper, ax-bridge-help, native-accessibility-bridge, and accessibility-bridge pass - Live dump --debug against a booted ko-KR iPhone 17 Pro emits all four new events; stdout payload unchanged outside expected app-state churn Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This was referenced Apr 29, 2026
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces enhanced diagnostic logging for the accessibility walker in ax-bridge.swift to help investigate issues with SpringBoard-hosted overlays. Key changes include enumerating all Simulator windows, tracking specific overlay-related roles, and caching metadata for content candidates to emit detailed debug events. These diagnostics are gated by a debug flag and include safety caps to limit resource usage. I have no feedback to provide.
9 tasks
shaun0927
added a commit
that referenced
this pull request
Apr 29, 2026
* test(integration): add gated ko-KR push-permission live suite (#660) Issue #660 plan item: "Add integration test gated by a ko-KR simulator availability check in CI". Encodes the three Exit criteria from the issue body as Jest assertions: 1. app_handle_alert { action: 'dismiss' } returns dismissed: true, strategy: 'ax-scan' 2. visibleButtons contains the localized labels (허용, 허용 안 함) 3. ax-bridge-native dump --debug exits 0 and emits the walker debug events added in #691 The suite is gated behind explicit opt-in (OPENSAFARI_KOKR_PUSH_DIALOG=1 + OSF_DEVICE_ID + OSF_PERMISSION_BUNDLE) because the cold-launch step requires the host shell to have Full Disk Access for the simulator's TCC sandbox so `simctl privacy reset notifications` succeeds. Without the FDA grant, the suite skips with a structured warning instead of failing CI on a host environment issue. See docs/recipes/transient-simctl-errors.md for the FDA setup the suite expects. Default behavior (no opt-in) is a single-test "suite skipped" assertion, so the file integrates cleanly into the existing tests/integration set without needing a CI flag flip. When a focused session has the FDA grant, opting in turns the same file into a binary verification of the issue's Exit criteria. The walker_* event assertions in the second test depend on the diagnostic events landed in #691; opt-in runs against a develop branch that includes #691 will pass cleanly, runs against an earlier develop will hit a partial-fail on those events but only when opt-in is set. Verified locally: - Default-skip run: 1 test passes, prints opt-in skip warning - ESLint clean Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(660): throw on opt-in env failure; canonicalize visibleButtons Address gemini-code-assist (3) and chatgpt-codex-connector (1) review items on PR #692: - gemini@:204, :232, :246: once OPENSAFARI_KOKR_PUSH_DIALOG=1 has been flipped, environmental failures (TCC reset error, empty tool body, missing permission sheet) are setup errors that the operator alone can resolve and must surface as test failures rather than silent console.warn skips. Skip-with-warn behavior is preserved for the default no-opt-in path. - codex@:253: collectVisibleButtonLabels in app_handle_alert annotates non-ASCII whitespace as "<original> (norm: <normalized>)" (slice 2 of #642), and SpringBoard ko-KR permission sheets use NBSP between the syllables of 허용 안 함. Strict equality against the raw constant would falsely fail when the diagnostic suffix is present or when whitespace is U+00A0. Introduces stripDiagnosticAnnotation + normalizeWhitespace + canonicalLabel mirroring the regex from src/tools/app-handle-alert.ts so the assertion compares against canonicalized labels. Verified locally: - Default-skip run passes 1 test with the structured warning - ESLint clean Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(660): cover U+2060 in canonicalLabel whitespace normalization (codex P2) Address chatgpt-codex-connector P2 on PR #692 (commit d9f1599, line 80): JavaScript's `\s` regex class does NOT match U+2060 WORD JOINER, even though `src/tools/app-handle-alert-labels.ts` explicitly treats it as fancy whitespace alongside U+00A0 / U+202F / U+2007 / U+2028 / U+2029. A SpringBoard label like "허용안함" (with U+2060 between syllables) would slip past `\s+` normalization untouched and never equal the plain ASCII-spaced constant `허용 안 함`, producing a false-regression report on a passing behavior. Mirror the FANCY_WHITESPACE codepoint set from the source file 1:1 and strip those characters before applying the standard `\s+` collapse. The escape sequence form is used so the regex character class is unambiguous in source — avoids the editor-level invisibility of literal control characters. Verified locally: - Default-skip run passes 1 test - ESLint clean Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--debugmachinery so the next live ko-KR capture produces the failure-tree data shaun0927's 2026-04-27 status comment identified as the prerequisite to safely landing a SpringBoard-explicit walker change. Stdout (the JSON dump payload) is unchanged.findMatchingWindowandfindDeviceContentRecursively:walker_app_windows_enumerated— every AXWindow under Simulator.appwalker_top_candidates— top-5 candidates by score with role/label/depth/score/appSemanticsCountwalker_overlay_roles_seen— every overlay-suspect role (AXSheet, AXAlert, AXSystemDialog, AXSystemFloatingWindow, AXDialog, AXSystemAlert) encountered during the walkwalker_winner/walker_winner_none— the chosen content root or an explicit no-winner eventWALKER_DEBUG_CANDIDATE_CAP=256andWALKER_DEBUG_OVERLAY_CAP=32.Why this is safe
debugEnabled, which only flips when--debugor--verboseis passed.findDeviceContentRecursivelyhappy path is preserved verbatim — the patch only adds tracking arrays and a single emission call after the recursion returns.Why this advances #660 without speculative Swift
The remaining plan items in #660 (SpringBoard explicit entry-point + integration test + Exit criteria) all require the failure tree from a live ko-KR cold-launch. PR A (#685) added the JSON-stderr framework. PR B (#686) bounded AX messaging at 1.5s. This PR closes the diagnostic gap so a future focused session can:
--debugstderr against a sheet-visible cold-launchwalker_overlay_roles_seen.countto decide whether the overlay subtree is reachable from the matched windowwalker_app_windows_enumeratedto confirm whether SpringBoard hosts the sheet in a sibling window the current walker never visitsTest plan
swiftc -O src/native/ax-bridge.swiftcompiles cleantests/unit/ax-bridge-{content-root,wrapper,help}.test.ts,tests/unit/native-accessibility-bridge.test.ts, andtests/unit/accessibility-bridge.test.tspassdump --debugon a booted ko-KR iPhone 17 Pro / iOS 26.4 emits all four new events🤖 Generated with Claude Code