iPhone 17 Pro simulator taps use wrong coordinate space and AX post-tap probe fails

[shaun0927]

Summary

OpenSafari's app_tap_element and app_double_tap cannot reliably drive a Flutter app on an iPhone 17 Pro / iOS 26.4 simulator. Two failure surfaces are entangled and both must be closed before the App Review verification flow (settings, notification opt-in, IAP, account deletion, UGC controls) can run autonomously:

1. AX dump empty on Flutter sub-screens (e.g. the in-app Ducats Shop route). ax-bridge-native dump exits non-zero with DEVICE_CONTENT_ROOT_EMPTY; the matched device window has descendants but every candidate the walker scores reports appSemanticsCount: 0.
2. Coordinate space mismatch between the AX frames returned by the bridge (in macOS-screen-points relative to the device-content-root, ≈ 697×1515 on this combination) and the iOS-points the SimHID dispatcher consumes (402×874 on iPhone 17 Pro). Even when the AX tree is populated, taps issued at the AX-frame center can land outside the visible target.

Environment

• App under test: Flutter app com.omofictions.omofictionsApp
• Simulator: iPhone 17 Pro, iOS 26.4
• Device UDID: D7D26213-C3E9-4623-BCCB-984CDF5D0793
• OpenSafari binary: /Users/jh0927/opensafari/dist/ax-bridge-native
• Date: 2026-04-29 KST

Reproduction

Tested live in this session against the booted device above. PR #691's walker --debug events confirm both surfaces; quoted output is verbatim from the failing dump invocation.

Surface A — AX dump empty on the Ducats Shop screen

1. Cold-launch the app: xcrun simctl launch <UDID> com.omofictions.omofictionsApp. App restores to the in-app Ducats Shop route.
2. Run ax-bridge-native dump --device <UDID> --max-depth 12 --debug.
3. Bridge exits 1 with DEVICE_CONTENT_ROOT_EMPTY. The new walker debug events from feat(ax-bridge): walker candidate diagnostics on --debug (#660) #691 show the precise failure shape:

   walker_app_windows_enumerated count=2 (iPhone 17 Pro device window + AXMenuBar)
   walker_top_candidates totalCandidates=8, all 5 top entries have appSemanticsCount=0
     [0] depth=1 role=AXGroup score=5 (matches iOSContentGroup trait)
     [1..4] depth=1-2 role=AXButton/AXGroup score=0..-5
   walker_overlay_roles_seen count=0
   walker_winner depth=1 role=AXGroup score=5 appSemanticsCount=0
   content_root_empty
   

4. The screen renders normally (5 product cards, "Buy Ducats" button, bullet copy, "Terms of Service" link), so the AX subtree exists in pixels but Flutter has not exposed Semantics widgets to the macOS AX bridge. ensureSemanticsActive (the simctl spawn defaults write com.apple.Accessibility ApplicationAccessibilityEnabled -int 1 path) does not repopulate the tree on its own; the dump still exits non-zero after the flag flip.

Surface B — AX-frame to iOS-point coordinate mismatch

1. On a screen where AX is populated (e.g. the bottom-tab Home page, per the original report), app_tree returns the My element at logical frame x=522.75, y=1317.39, width=174.25, height=138.67, center (609.875, 1386.73) — these are macOS-screen-points relative to the device-content-root origin.
2. The simulator screenshot is 1206 × 2622 px (iPhone 17 Pro @3x).
3. iPhone 17 Pro iOS-points: 402 × 874. Ratio of AX-frame size (697×1515) to iOS-points (402×874): 697/402 ≈ 1.733, 1515/874 ≈ 1.733.
4. sim-hid-bridge.swift:336-349 documents that IndigoHIDMessageForMouseNSEvent consumes coordinates in iOS-points, divided by mainScreenScale when the runtime reports mainScreenSize in pixels (the Xcode 26+ regression fixed in test(input): SimulatorKitHID 통합 테스트 — 실 시뮬레이터 Settings.app 자동화 검증 #491).
5. app-tap-element.ts:211-212 builds rawCenterX/rawCenterY directly from match.frame.x + match.frame.width/2 and forwards to backend.tap(deviceId, centerX, centerY, …) without any macOS-pt → iOS-pt conversion. On iPhone 17 Pro at the Simulator zoom in use, the dispatched coordinates are ~1.733× larger than the iOS target on each axis.

Observed

• app_tap_element returns status: tapped but the UI does not change.
• verified: false, effect: "verification_unavailable".
• The post-tap context probe surfaces POST_TAP_CONTEXT_PROBE_FAILED because the dump that follows the tap hits the same DEVICE_CONTENT_ROOT_EMPTY exit as the pre-tap dump.
• Subsequent app_tree calls intermittently fail with the same DEVICE_CONTENT_ROOT_EMPTY error and PR feat(ax-bridge): walker candidate diagnostics on --debug (#660) #691's walker events confirm appSemanticsCount=0 across all candidates.

Expected

• app_tap_element(label: "My") converts the AX frame from macOS-points to the iOS-point input space the SimHID backend expects, and the dispatched tap lands on the visible bottom-tab center.
• Post-tap verification recovers the AX tree on Flutter screens that don't cold-expose semantics, OR returns a typed diagnostic that distinguishes "Flutter Semantics not yet active" from "AX server crashed" from "app backgrounded".

Root-cause hypothesis

Surface A

ensureSemanticsActive's simctl-defaults activation (ApplicationAccessibilityEnabled) writes the flag but does not currently force Flutter's SemanticsBinding.ensureSemantics callback on a screen the framework already considers settled. The Dart VM Service fallback (Strategy C in src/native/semantics-activator.ts:14-19) is the only mechanism that materialises the tree from a settled state via debugDumpSemanticsTreeInTraversalOrder, and that path requires a debug or profile build. Production / release builds — the App Review verification target — exhaust the activator's strategy ladder and surface DEVICE_CONTENT_ROOT_EMPTY to the wrapper.

Surface B

app-tap-element.ts:211-212 treats match.frame as iOS-points but the AX bridge reports it in macOS-screen-points after subtracting the device-content-root origin (src/native/ax-bridge.swift:234-235). On a Simulator window scaled so 1 iOS-pt ≠ 1 macOS-pt — observed at 1.733× on iPhone 17 Pro / iOS 26.4 in this session — the conversion factor is missing. The bridge already knows the device-content-root size in macOS-pt; SimHID already knows iOS-pt size via getScreenSize. Joining the two in either layer (bridge emits the conversion factor on every dump, OR app_tap_element queries the size and divides) closes the gap.

Plan

• WU1 — Diagnostic surfacing (no behaviour change): when the wrapper observes DEVICE_CONTENT_ROOT_EMPTY, surface the walker --debug data (top candidates + overlay roles + window enumeration) in the thrown error message instead of the bare Command failed: … tail. Forces the diagnostic onto the user's tool output without requiring a manual re-run. Depends on PR feat(ax-bridge): walker candidate diagnostics on --debug (#660) #691.
• WU2 — Surface A: production-build Semantics activation. Investigate whether the simctl spawn defaults write com.apple.Accessibility ApplicationAccessibilityEnabled -int 1 flag combined with a process re-foregrounding nudge (or a user-initiated tap on a non-Semantics surface) reliably fires SemanticsBinding.ensureSemantics on release builds. If it does, retry once after the activation step in ensureSemanticsActive Strategy B. If it does not, document the limitation in a recipe and surface a typed FLUTTER_SEMANTICS_INACTIVE error code so the wrapper can offer a structured fallback message rather than the current opaque DEVICE_CONTENT_ROOT_EMPTY.
• WU3 — Surface B: AX-frame to iOS-point conversion. Emit the device-content-root macOS-pt size from ax-bridge.swift on the dump root (alongside chromeOnly), have app-tap-element.ts look up iOS-pt size via simctl runtime info and scale centerX / centerY per axis before calling backend.tap. Apply the same conversion to app_double_tap and any other coordinate-from-AX-frame call site.
• WU4 — Live integration test for both surfaces, opt-in gated similar to test(integration): gated ko-KR push-permission live suite (#660) #692. Asserts a tap on a known label lands on the visible target (verified via post-tap AX recovery + screenshot diff), and that the dump on a Flutter sub-screen returns a non-empty tree under each tier of the activator ladder.
• WU5 — Recipe documenting the AX-frame coordinate space and the production-build Semantics-activation guarantee level (debug vs profile vs release).

Exit criteria

• app_tap_element { label: "My" } on a populated Home tab dispatches a tap whose post-tap context probe confirms the navigation (Home → My).
• app_tree on the Ducats Shop sub-screen returns a tree whose flattened node count exceeds MIN_NODE_THRESHOLD (5) on both debug and release builds, OR the wrapper surfaces a typed FLUTTER_SEMANTICS_INACTIVE error so the caller can branch.
• Walker --debug data is included verbatim in the error returned by ax-bridge wrapper when DEVICE_CONTENT_ROOT_EMPTY fires (no manual re-run required).
• WU4's gated integration test passes against an iPhone 17 Pro / iOS 26.4 simulator with the omofictions app cold-launched.

Related

• arch: AX bridge GUI-less path — decouple AX reads from Simulator.app foreground #573 — kept AX tree live for backgrounded Simulator (foreground dependency)
• ax-bridge-native: 세션 중 크래시 / DEVICE_CONTENT_ROOT_EMPTY — health-recover 로직 부재 #643 — self-healing recovery wrapper for the AX bridge
• feat(ax-bridge): enumerate UNUserNotificationCenter permission sheet (#651 follow-up) #660 — UNUserNotificationCenter sheet enumeration (parallel SpringBoard concern; feat(ax-bridge): walker candidate diagnostics on --debug (#660) #691 added the walker debug events this issue uses for diagnosis)
• Rescue coordinate taps with AX button press #690 — AX rescue path for non-modal coordinate taps
• test(input): SimulatorKitHID 통합 테스트 — 실 시뮬레이터 Settings.app 자동화 검증 #491 — pixel/point regression in mainScreenSize (the upstream of the SimHID coordinate-space contract)

[shaun0927]

Sprint 1 / autopilot status (2026-05-07)

WU3-impl (TS-side macOS-pt → iOS-pt coordinate conversion using deviceContentMacOSPt from #695) is being landed in this session. PR will be linked from this comment chain on open.

WU2 (production-build Flutter Semantics activation) is not landing in this autopilot session:

• WU2 requires reproducing the failing case live: a release-build of com.omofictions.omofictionsApp cold-launched on iPhone 17 Pro / iOS 26.4 with the Ducats Shop sub-screen visible, then exercising the simctl spawn defaults write com.apple.Accessibility ApplicationAccessibilityEnabled -int 1 activation strategy with a re-foregrounding nudge to confirm whether SemanticsBinding.ensureSemantics actually fires on a settled release build.
• Without that live signal, the choice between (a) retrying activation in ensureSemanticsActive Strategy B vs. (b) surfacing a typed FLUTTER_SEMANTICS_INACTIVE error code is speculation. The Issue Plan explicitly anticipates this fork.
• Same FDA-granted host precondition as feat(ax-bridge): enumerate UNUserNotificationCenter permission sheet (#651 follow-up) #660 applies.

WU4 (live integration test for both surfaces) is gated on WU2 (Surface A) plus the live host. WU3-impl alone unlocks the Surface B regression test, which can ship as a focused follow-up.

WU5 (recipe + docs) will land once WU3 is verified merged on develop.

This issue stays OPEN. Closing it requires WU2 / WU4 / WU5 against a live FDA-granted host.

🤖 Generated with Claude Code