Apps that render to a canvas (games, Flutter apps with no semantics, some webviews) expose an empty accessibility tree — the agent is blind there. The fix: detect the empty-tree case and fall back to sending an annotated screenshot to a vision-capable model, letting it answer with tap coordinates.
Where to look
accessibility/UiTreeBuilder.kt — where an empty/thin tree is detectableaccessibility/UiObserveRenderer.kt + screenshot pipeline (already exists for the legend overlay)ChatSession.phoneCapabilitySuffix — the act grammar the model is taught
Open design questions
- Coordinate-grid overlay on the screenshot vs raw pixels + normalized coords?
- When to trigger fallback automatically vs a new
act look --vision verb?
This is the hardest and highest-impact open problem in the repo. Fame awaits. 🐉
Apps that render to a canvas (games, Flutter apps with no semantics, some webviews) expose an empty accessibility tree — the agent is blind there. The fix: detect the empty-tree case and fall back to sending an annotated screenshot to a vision-capable model, letting it answer with tap coordinates.
Where to look
accessibility/UiTreeBuilder.kt— where an empty/thin tree is detectableaccessibility/UiObserveRenderer.kt+ screenshot pipeline (already exists for the legend overlay)ChatSession.phoneCapabilitySuffix— the act grammar the model is taughtOpen design questions
act look --visionverb?This is the hardest and highest-impact open problem in the repo. Fame awaits. 🐉