diff --git a/agents/agy-reviewer.md b/agents/agy-reviewer.md index 7ee5b813..5a166753 100644 --- a/agents/agy-reviewer.md +++ b/agents/agy-reviewer.md @@ -1,6 +1,6 @@ --- name: agy-reviewer -description: Cross-family review by Gemini via the agy CLI, for a genuinely different-provider perspective on a diff, design or finding. Use to satisfy the cross-family leg of the review ladder, or whenever a second opinion should not come from another Claude. Returns the verdict and findings plus a path to the full review. +description: Cross-family review by Gemini via the agy CLI, for a genuinely different-provider perspective on a diff, design or finding. Use to obtain the cross-family leg of the review ladder, or whenever a second opinion should not come from another Claude. Agy reviews are independent opinions, not sandboxed certification. Returns the verdict and findings plus a path to the full review. tools: Bash, Read, Write, Glob, Grep model: haiku effort: low @@ -34,8 +34,10 @@ read of the same diff. Agy holds its own `agy` Agent Fabric seat, so a Gemini finding is recorded against the Google family rather than borrowing another provider's identity. Fabric carries the coordination and -the record; this dispatch is the call itself. You do not need to bootstrap, request or verify -the seat before reviewing, and a missing seat does not block a review. +the record; this dispatch is the call itself. The dispatcher records this route as `prompt_only`, +because agy is not sandboxed against writes. It remains a genuine independent opinion, but is not +certification eligible. You do not need to bootstrap, request or verify the seat before reviewing, +and a missing seat does not block a review. ## Procedure @@ -61,9 +63,15 @@ including work already completed. That makes the prompt load-bearing. Tell Gemini plainly what it may and may not do: > You MAY read files under ``. You MUST NOT run shell commands or write -> any file. Those are auto-denied, and one denied call discards your entire -> answer including work already done. If you cannot answer without one, say so -> in prose instead of attempting it. +> any file. If you cannot answer without one, say so in prose instead of +> attempting it. + +Both halves of that matter, and they are separate facts. A tool the local agy +permissions do **not** grant is auto-denied, and one denied call discards the +entire turn including work already completed, so attempting one is expensive. +A tool the permissions **do** grant simply runs: the instruction is a request, +not a sandbox, so never treat a review's silence about writing as proof that +it did not write. Also ask it to state at the top whether its reads succeeded. That is cheap, and it catches a review written blind that no status field would reveal. diff --git a/skills/orchestrate/evals/test_cf_dispatch.py b/skills/orchestrate/evals/test_cf_dispatch.py index b33c49fe..2afa966b 100644 --- a/skills/orchestrate/evals/test_cf_dispatch.py +++ b/skills/orchestrate/evals/test_cf_dispatch.py @@ -422,8 +422,8 @@ def test_agy_direct_route_dispatches_json_sandbox_and_file_prompt(): assert record["provider_family"] == "google" assert record["endpoint_provider"] == "agy" assert record["effort"] == "medium" - assert record["read_only_guarantee"] == "enforced" - assert record["certification_eligible"] is True + assert record["read_only_guarantee"] == "prompt_only" + assert record["certification_eligible"] is False assert out.read_text(encoding="utf-8") == "AGY OK" args = args_file.read_text(encoding="utf-8").splitlines() assert args[args.index("--output-format") + 1] == "json" diff --git a/skills/orchestrate/references/cli-headless.md b/skills/orchestrate/references/cli-headless.md index 4ad0cff3..4fb712d5 100644 --- a/skills/orchestrate/references/cli-headless.md +++ b/skills/orchestrate/references/cli-headless.md @@ -1,6 +1,6 @@ # CLI headless reference (dated layer) -Verified locally on macOS, 2026-06-07. Model IDs, flags, auth, and safety modes drift. Always run +Verified locally on macOS, 2026-08-05. Model IDs, flags, auth, and safety modes drift. Always run ` --help` / model discovery before depending on a chain. ## Contents @@ -18,8 +18,9 @@ Verified locally on macOS, 2026-06-07. Model IDs, flags, auth, and safety modes Headless CLIs are how external work reaches another provider. Fabric carries the request, the reply and the activity record around that call; it does not -run the provider itself. A verifier that is not certifying must enforce -read-only or planning mode; advisory claims require independent verification. +run the provider itself. Certification requires an enforced read-only +boundary. A prompt-only review can still provide a genuine independent +opinion, but it is not certification eligible. ## Harness-conditioned rule @@ -81,8 +82,12 @@ or `oauth_safe_mode`. - `agy`: `--sandbox --output-format json --disable-slash-commands`, with `--model`/`--effort` as separate flags and repeatable `--add-dir` for read material (also settable as a colon-separated `CF_DISPATCH_AGY_ADD_DIR`). - stdout and stderr stay separate so a permission denial cannot masquerade as - an empty success. `--dangerously-skip-permissions` is refused. + These flags do not enforce read-only access, so the dispatcher reports + `prompt_only`: the prompt asks agy not to mutate, but local permissions can + still allow writes. A write probe under agy 1.1.10's dispatcher flags + succeeded and created the file, and `--mode plan` did the same. stdout and + stderr stay separate so a permission denial cannot masquerade as an empty + success. `--dangerously-skip-permissions` is refused. - `cursor`: `--mode ask --sandbox enabled`; current help documents ask as read-only, while current headless plan mode can exit without an answer. - `kiro`: disabled by default in the dispatcher. Enable only with `CF_DISPATCH_ENABLE_KIRO=1`; no hard @@ -91,8 +96,9 @@ or `oauth_safe_mode`. disabled (`--available-tools=''`); repo inspection cannot currently be guaranteed read-only from local help. -If any adapter cannot enforce the promised safety level, log the failure and fail over. Do not silently -downgrade certification. For large prompts, prefer `--prompt-file`; enforced adapters use +If any adapter cannot enforce the promised safety level, record its actual guarantee and keep +certification ineligible. Do not silently upgrade `prompt_only` to `enforced`. For large +prompts, prefer `--prompt-file`; enforced adapters use stdin/file-backed input where supported to avoid shell argument limits. Orchestrated runs always pass `--out /` and list it in the manifest. Omitting `--out` creates one declared ephemeral output for a @@ -212,8 +218,8 @@ every implementation leg, check `git -C log --oneline ..HEAD`, A worker has no conversation context, so the brief carries everything: absolute path and branch, what has already been verified so it is not redone, ordered parts, out-of-scope paths, how to verify, and that it must not push, open a pull request or merge. Always -include the scepticism clause — *verify every claim in this brief before relying on it; if -something here is wrong, saying so is more valuable than following it* — which has +include the scepticism clause: *verify every claim in this brief before relying on it; if +something here is wrong, saying so is more valuable than following it*, which has repeatedly produced the most valuable output of a run. ## Runtime routing diff --git a/skills/orchestrate/scripts/cf_dispatch.sh b/skills/orchestrate/scripts/cf_dispatch.sh index 27c73ef9..db3dd017 100755 --- a/skills/orchestrate/scripts/cf_dispatch.sh +++ b/skills/orchestrate/scripts/cf_dispatch.sh @@ -440,7 +440,11 @@ run_one() { # $1 tool $2 model $3 effort -> writes clean answer to OUT, echoes ${model:+--model "$model"} "$(cat "$PROMPT_TMP")" "$raw" 2>"$diag"; rc=$? fi ;; agy) - guarantee="enforced" + # agy --sandbox does not enforce read-only writes. On agy 1.1.10 a + # write probe under these dispatcher flags returned SUCCESS and + # created the file; --mode plan did the same, so only the prompt + # discourages mutation. + guarantee="prompt_only" if ! require_cmd agy "$diag"; then status="tool_not_found" rc=127