browser_click: accept param aliases and resolve natural-language descriptions#631
Open
esslamali15 wants to merge 1 commit into
Open
browser_click: accept param aliases and resolve natural-language descriptions#631esslamali15 wants to merge 1 commit into
esslamali15 wants to merge 1 commit into
Conversation
…rategy ladder
Small local models call browser_click with target=/text=/element= instead
of selector=, and pass natural descriptions like 'first video result'.
Both previously dead-ended ('No selector provided' / invalid CSS), which
in practice broke every browse-and-click workflow driven by a local model.
- Accept the common param aliases
- Resolve through a ladder: CSS -> exact text -> fuzzy text -> YouTube
video-title special case -> role-based keyword match, using count() so
non-matching strategies fail in milliseconds
- Search child iframes after the main frame (consent dialogs, ad frames)
- On failure, include the visible clickable elements in the error so the
model can self-correct on the next turn
esslamali15
requested review from
ANarayan,
jonsaadfalcon and
robbym-dev
as code owners
July 8, 2026 22:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Local models (qwen3, llama, etc.) driving the orchestrator call
browser_clickwithtarget=/text=/element=instead ofselector=, and pass descriptions like 'first video result' rather than CSS. Both dead-end today (No selector provided/ Playwright invalid-selector error), which in practice breaks most browse-and-click workflows — the model gets an unhelpful error, retries the same thing, and eventually apologizes to the user.Fix
selector|target|text|element|queryall accepted.count()so non-matching rungs cost milliseconds; only a matching strategy gets a click timeout.Testing
On a live install (qwen3:8b orchestrator): the previously-failing flow navigate to YouTube search → click 'first video result' now resolves in ~0.1s via the ladder and lands on the correct watch page; a nonsense selector returns the candidate list instead of a bare error.
🤖 Generated with Claude Code