Skip to content

Skill should require observing page state after every action, not just performing it #90

Description

@kiki830621

Problem

使用者原話
「我覺得safari-browser的使用skill應該要每作一個動作之後,就看依次網頁有什麼變化才對」

The skill teaches how to perform actions (click, js, fill, upload) but does not
require observing the resulting page state before the next action. In practice an
agent chains actions on the assumption that each one did what was intended. On a fragile
site that assumption breaks silently, and the divergence is only noticed several steps
later — by which time the recovery is expensive and the cause is hard to attribute.

Type

docs

Expected

After each state-changing action, the skill should require reading back a cheap state
signal before continuing — at minimum location.pathname, plus the specific field or
row the action was supposed to affect. Gate the next step on that observation, not on
the action's return value.

Actual

Nothing in the skill mandates a post-action observation, so the natural agent behaviour
is fire-and-continue.

Evidence from one session (a government portal: frameset + iframe, server-rendered forms)

Four distinct failure shapes, all of which a post-action read would have caught immediately:

  1. A text-matched button hit the wrong element. Selecting 確認 by visible text
    matched a different button than the modal's, and the click navigated away from the
    form to a PDF preview endpoint. The next four commands ran against the wrong page
    before the mismatch surfaced.
  2. set URL silently failed. On a blank POST-result page, AppleScript reported
    success and location.pathname stayed on the old value. Retries in a loop were needed;
    nothing errored.
  3. .value disagreed with the screen. For a widget-backed date field, reading the
    input returned the string previously assigned while the widget rendered a different
    value. The read was confidently wrong in both directions — assignment did not reach
    the widget, and read-back did not reflect it.
  4. A click returned undefined while succeeding. Navigation began before the return
    value could be collected, so a script gating on === "opened" aborted a step that had
    in fact worked.

The common shape: the signal came loose from the state. Return values, "no error", and
element .value are all unreliable on their own.

Impact

  • Agents chain several actions onto a page that is no longer the page they think it is.
  • Recovery cost is superlinear: the further the divergence propagates, the harder it is to
    tell which step went wrong.
  • Worse on sites with short sessions — wasted steps consume the working window before the
    task completes.

Suggested direction (not prescriptive)

  • A short "observe after acting" section in the skill, with the cheap default read
    (location.pathname + the target element) and the rule gate on state, never on return
    value
    .
  • Note that text-matched selectors are not identity: scope to a container, or verify the
    element's onclick / id before clicking.
  • Note that widget-backed fields need the widget's own API for both read and write; the
    underlying input.value is not authoritative.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions