Repackage pilotfish as a Claude Code plugin#11
Conversation
|
Heads-up: I've pushed three more commits to this branch. Two are substantive, one is taste — flagging the split explicitly so you can review (or reject) them independently, and I'm happy to pull any of them out into their own PR if you'd rather this branch stayed just the repackage.
Net effect: the shipped configuration is now exactly the one Anthropic benchmarked — a frontier orchestrator with Sonnet 5 workers, 96% of all-frontier performance at 46% of cost. The three Sonnet roles stay three files because the
Guard behaviour is unchanged by the routing commit (verified at bytecode level — only human-readable strings moved). Test suite is green: 21 tests, up from 19. |
… hook
BREAKING CHANGE: pilotfish is now a plugin, not a global-config merge.
/plugin marketplace add Nanako0129/pilotfish
/plugin install pilotfish@pilotfish
/pilotfish
Nothing is written into ~/.claude/ or into your projects, and uninstalling removes
every trace. templates/, install/, VERSION, RELEASING.md and the four-way version-stamp
coupling are gone -- all of that existed to manage a hand-merged global config.
The orchestration policy becomes a skill, so it loads when you type /pilotfish rather
than into every session of every project. Its self-disabling preamble ("if you are a
subagent, ignore this section") is deleted: a skill never reaches subagents, so the
confusion it patched cannot occur.
THE GUARD
A policy is a request; a missing capability is a fact. A PreToolUse hook now enforces
what the policy could only ask for. The discriminator is the presence of `agent_type` in
the hook payload -- absent in the main session, present in a subagent (verified against
real captured payloads):
* subagents may not set run_in_background
* subagents may not detach (nohup / setsid / disown / trailing &)
* the built-in Explore agent is denied; recon routes to `scout`
The main session keeps every capability. Its backgrounding is the path that works, and
blocking it would disable the orchestrator. The guard fails open.
WHY SUBAGENTS MAY NOT DETACH (established by experiment, not assumption)
A foreground command exceeding its `timeout` is not killed -- the harness promotes it to
a background task and promises a completion notification. Whether that promise is kept
depends on how the orchestrator spawned the agent:
* spawned with run_in_background=true -> the promoted process survives the turn
boundary AND the agent returning, runs to completion, has its stdout captured, and
its notification re-invokes the agent. Observed surviving 59s/69s/81s past a turn
boundary and 141s past agent termination, zero signals across four trials.
* spawned in the foreground -> SIGTERMed ~4s after the agent returns. Work
destroyed, captured output truncated mid-stream. Reproduced exactly.
nohup/setsid dodge that SIGTERM by escaping the process group -- which is why the pattern
gets adopted -- but they also escape the harness's task tracking: no task id, no captured
output, no notification. Detaching launders a destroyed result into an orphaned one.
So subagents no longer detach. They run in the foreground with an explicit timeout and
hand back anything that cannot finish inside one; long-running processes belong to the
orchestrator, the only context whose background tasks are both tracked and notified. This
also makes spawning agents with run_in_background=true load-bearing for correctness, not
merely for cost.
WHY EXPLORE IS BLOCKED RATHER THAN SHADOWED
Plugin agents are namespaced (pilotfish:Explore), so a plugin cannot shadow the built-in
Explore the way a user-level ~/.claude/agents/Explore.md can -- tested, and the override
silently did not take. Since v2.1.198 the built-in inherits the main-session model, so
every background search bills at frontier rates. The guard blocks it and routes recon to
`scout` (haiku), preserving the behaviour without depending on shadowing.
Five roles now, not six.
WHAT CANNOT BE PORTED
A plugin cannot set the main-session model. `model: "best"` + fallbackModel is now a
documented manual step (/model best) rather than something an installer writes. That is
the same property as the clean uninstall, stated honestly.
TESTS
16 dependency-free tests. test_guard.py pins both asymmetries -- including that the main
session KEEPS run_in_background/detach -- and that the guard does not false-positive on
&&, 2>&1, & inside quotes, or substrings. test_plugin.py checks the manifests parse, the
hook script exists and is executable (a hook pointing at a missing script fails silently,
which would turn every guarantee back into a suggestion), every role pins its own model,
and the skill routes only to roles that exist. Both guards were also verified live in a
real session via --plugin-dir, not only in unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scout moves Haiku -> Sonnet (effort low); executor moves Opus/medium -> Sonnet/high; verifier stays Opus and goes to effort high. mech-executor (Sonnet/low) and security-executor (Opus/high) are unchanged. The shipped configuration is now exactly the one Anthropic benchmarked: a frontier orchestrator with Sonnet 5 workers, at 96% of all-frontier performance for 46% of the cost. Scout output is unverified input to everything downstream -- the verifier gate covers executor work, not reconnaissance -- so a wrong file:line from a scout becomes an executor confidently editing the wrong thing, with nothing to catch it. Haiku 4.5 -> Sonnet 5 is a large enough capability gap that this stops being theoretical. On subscriptions Haiku was never the cheap option it looked like: the weekly limit is a shared all-models bucket plus an additional Sonnet-only bucket, so Haiku draws on the scarce one while Sonnet draws on dedicated headroom. Opus on executor was over-insurance. Quality on cheap executors is bought back by the verifier more cheaply than by upgrading the executor itself; paying Opus rates for routine implementation *and* running a verifier was paying twice for the same guarantee. The verifier's effort bump is where that budget goes instead. The three Sonnet roles stay three files: the Agent tool has no effort parameter, so effort is settable only in frontmatter, and collapsing them would run bulk mechanical work at effort high for nothing. Guard behaviour is unchanged; only the text of its Explore denial, which names the tier scout is pinned to. Docs and the zh-TW mirrors updated to match. The 74% Haiku row is dropped from the savings table -- quoting savings for a tier we no longer ship would be false advertising. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fnkD9CdpQUvFTPQGz9GzC
215 lines -> 89, in both languages. The README had grown into a manual: a table of contents, a full install walkthrough, a trust section, a fallback-story table, and a twelve-row FAQ. Almost none of that is what a reader needs in the first thirty seconds, and all of it competed with the three things that are: what pilotfish is, what it buys you (cost and speed), and the shape of the thing. Nothing is deleted, only relocated. The fallback story, the tuning knobs, the FAQ, and the design rationale live in docs/design.md, which the README now links to. Install collapses into the intro (it was always two commands); uninstall is one clause; trust & security folds into the guard section, where the "read the ~100 lines yourself" argument actually lands. The Why section now leads with cost AND speed -- the speed argument was never made explicitly, though it is half the point of routing bulk work to a faster model at low effort and overlapping independent agents. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fnkD9CdpQUvFTPQGz9GzC
A heredoc body is data on some program's stdin — the calling shell never interprets it — so an `&`, `nohup`, or `setsid` inside one is prose, not a control operator. The guard scanned the whole command string, so a subagent running `python3 - <<'EOF'` was denied for "backgrounding a command" if the script it piped in merely contained an ampersand. An `&` in a code comment was enough. Found in the wild: it blocked a verifier agent mid-run, on a comment reading "# strip docstrings & constant string values". This is the failure mode the guard's own docstring warns about and that test_legitimate_commands_are_not_false_positives exists to prevent — a guard that blocks legitimate work gets switched off, and then it protects nothing. strip_heredocs() blanks the body space-for-space, preserving every offset, and deliberately leaves the line that OPENS the heredoc intact — that is where a real trailing `&` or `nohup` would sit. It runs before strip_quotes, which would otherwise eat a quoted delimiter (`<<'EOF'`) and leave the body looking like live shell. Two tests, in opposite directions: heredoc bodies are data (ALLOW), and `cat <<EOF &`, `nohup cat <<EOF`, and a detach after the terminator still DENY — because an over-broad blanking fix would silently disable the whole shell-detach leg for any command containing a heredoc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fnkD9CdpQUvFTPQGz9GzC
The guard was a Python `PreToolUse` hook invoked bare, relying on its `#!/usr/bin/env python3` shebang and the Unix executable bit. Native Windows honors neither, so it could never launch there at all — the plugin was Unix-only by construction, not merely Python-dependent. And Python was never the real problem. Per Claude Code's own docs no interpreter is guaranteed on a machine running it: not Python, and not even `node`, since the native standalone installer never puts `node` on PATH. There is no runtime a plugin can ship a hook against and still run everywhere Claude Code runs. pilotfish's premise is that it should. The failure mode settles it. A guard hook must fail open, or a bug in the guard locks you out of your own session. But a *missing interpreter* also fails open, silently — so on exactly the machines where the hook cannot run you get no enforcement and no signal that enforcement is missing. A rule you believe is enforced is a rule you stop watching. A door that isn't there beats a door you wrongly believe is locked. So the two rules go back to policy, stated as genuine prohibitions: - Never invoke the built-in `Explore`; use `pilotfish:scout`. The cost finding is unchanged — it inherits the main-session model and bills every search at frontier rates — but nothing now stops a session that ignores it. - Subagents must never detach a process. `scout` is the one exception that survives as real enforcement: its `tools: Read, Glob, Grep` allowlist gives it no Bash to detach with. Both prompts previously said detaching "is blocked for subagents". That claim was false the moment the hook went, and worse than silence: a model told the door is locked stops holding it shut. `tests/test_plugin.py` now pins both halves — that no role is told *to* detach, and that every shell-capable role is told *not* to — plus that SKILL.md still carries both rules, so neither can be deleted without a test going red. Also fixes an effort table in docs/design.md that still claimed executor/verifier run at `medium`; 0d15f22 moved both to `high`. The plugin is now markdown and JSON. Nothing to install, nothing to interpret, runs anywhere Claude Code runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011pT25EttAtiXYfsvutWyhE
|
Maintainer note: I rebased this branch onto v1.2.0 ( The current branch also carries the v1.2 phase gates and compatibility requirements:
Validation completed after the rebase:
GitHub currently reports the PR as mergeable with a clean merge state. |
|
@dromsak, thanks for the work here. The Plugin packaging direction is solid, and most of the technical work carries forward into v2 regardless of how the activation question lands. The reason this can’t merge as-is is straightforward: pilotfish’s core contract is that orchestration is globally active—it is present in every session without requiring the user to invoke it. This PR replaces that behavior with explicit invocation, and that contract should not be dropped before we know whether the Plugin platform can preserve it. That is what #12 is for. v1 remains the supported baseline while the Plugin path is evaluated for equivalent ambient activation. The result will determine whether v2 succeeds v1 or exists alongside it as a Session-mode option. The following work carries forward regardless:
A focused PR against No rework is needed on this PR. I’ll leave it open for a few days in case there is any additional context you want to preserve, then close it as superseded by #12. PR #11 will remain linked there as the design provenance for the Plugin work and the findings that carry forward. Thanks again—this materially shaped the v2 plan even though the activation boundary is not something I can adopt yet. |
|
Closing this PR as superseded by #12. The Plugin packaging work remains valuable design provenance for the v2 activation decision. The marketplace packaging, Session-mode skill, namespaced roles, runtime preflight, Windows/interpreter portability findings, and model/effort routing research all carry forward. The unresolved boundary is the activation contract: pilotfish v1 remains ambient global orchestration, while this branch requires explicit Skill invocation. Future Plugin work will continue under #12, so no further rework is requested on this branch. @dromsak, thank you for the substantial implementation and research that shaped the v2 direction. |
Proposal
Repackage pilotfish as a Claude Code plugin.
This is a proposal, not a fait accompli — it's a big change and I'd rather you shape it than merge it as-is. The underlying bug it protects against is also fixable in the current v1.x architecture without this replatform — see my comment on #10 and the
fix-subagent-detachbranch. Take that and leave this if the plugin shape isn't for you.Important
This PR changed course, and you should know why. It originally proposed a
PreToolUseguard hook as its centrepiece — "the rules are now enforced, not requested". I've removed it. I ran this plugin on a Windows machine and the hook could not launch: it was a Python script invoked bare via its#!/usr/bin/env python3shebang and the Unix executable bit, and native Windows honors neither. Python was never the real problem — per Claude Code's docs no interpreter is guaranteed on a machine running it, not Python and not evennode(the native standalone installer never putsnodeonPATH). There is no runtime a plugin can ship a hook against and still run everywhere Claude Code runs. So this PR is now just the replatform, and pilotfish stays pure markdown + JSON. Details in Why no guard below.This resolves #8 (install) — and the update pain behind it
#8 is the install flow breaking because the agent-facing runbook trips WebFetch's prompt-injection protection, and the fallback is a clone-and-paste ritual. Your own last comment there was "what if we turned it into a cc plugin? That should simplify." — this is that.
/plugin installis native, first-party Claude Code — it can't be flagged as adversarial remote content, because it isn't fetching a document at all. The pinned-clone workaround goes away./plugin, no re-fetch, no re-merge. The four-file version-stamp coupling that made releases fragile is gone too.So #8 isn't worked around, it's designed out. If you merge this, it closes #8.
Why a plugin
The install stops being a merge into
~/.claude/. No policy block in your globalCLAUDE.md, no six files copied into~/.claude/agents/, nocurl | sh-shaped trust problem, no WebFetch prompt-injection caveat. Uninstalling removes every trace.The policy stops being an ambient context tax. Today the orchestration block loads into every session of every project, whether you're orchestrating or not — which is also why it needs its self-disabling preamble ("If you are running as a subagent role… ignore this section entirely"). As a skill, it loads when you type
/pilotfishand not before, and it never reaches subagents at all. The preamble's job disappears rather than being done better.The version-stamp machinery disappears.
VERSION+ the policy comment + two README clone-pins + a test enforcing all four move together +RELEASING.md— all of it exists to manage a hand-merged global config. A plugin has one version, in its manifest.Why no guard
A guard hook is genuinely the right shape for two of pilotfish's rules. #6 proved it: subagents kept re-delegating, and the fix wasn't a firmer instruction, it was
disallowedTools: Agent, Workflow. Capability removal beat persuasion, decisively. I built the same thing for detaching and for the built-inExplore, and it worked — on Linux and macOS.It cannot be made portable:
node— the native standalone installer never puts it onPATH. There is no safe assumption to make here.#!shebang nor the Unix executable bit. The idiomatic hook — ship a script, mark it executable, name it inhooks.json— is Unix-only by construction. It doesn't degrade on Windows; it never launches.And the failure mode is what settles it. A guard hook must fail open — a malformed payload has to allow the call, or a bug in the guard locks a user out of their own session. But a missing interpreter also fails open, and silently: on exactly the machines where the hook can't run, you get no enforcement and no signal that enforcement is missing. A rule you believe is enforced is a rule you stop watching. A door that isn't there beats a door you wrongly believe is locked.
So the two rules go back to policy, stated as prohibitions the model can read:
nohup,setsid,disown, trailing&,run_in_background)Explore; usepilotfish:scoutscoutis read-onlytools: Read, Glob, Grepgives it no BashdisallowedTools: Agent, Workflow(#6, unchanged)The two capability-removal mechanisms that are portable (a
toolsallowlist,disallowedTools) are both still in use. Only the hook is gone.One thing I want to flag as a real regression, not paper over: the
Explorecost problem is unsolved. Since v2.1.198 the built-in inherits the main-session model, so every search it runs from a Fable/Opus session bills at frontier rates. A plugin cannot shadow it back to a cheap tier — plugin agents are namespaced, sopilotfish:Exploreis a different agent from the built-in and overrides nothing (I tested this; the override silently did not take). Shadowing is unavailable, a hook is unshippable, so all that's left is the skill telling the orchestrator, in as many words, not to use it. Nothing stops a session that ignores that.Consequence, unchanged from the original proposal: five roles, not six.
Explore.mdis gone;scoutis the recon role.What's removed
templates/,install/,VERSION,RELEASING.md,tests/test_policy.py— all superseded.docs/andCHANGELOG.mdcarry over.What can't be ported
A plugin cannot set your main-session model.
model: "best"+fallbackModelis now a documented one-line manual step (/model best) rather than something the installer writes. Honest trade: the plugin gains a clean uninstall and loses the ability to touch your settings — which are the same property.Tests
8 tests, dependency-free, green (
python3 -m unittest discover -s tests— a contributor dep; the shipped plugin has none).tests/test_plugin.pypins the wiring: manifests parse; every role pins its own model in frontmatter (a role that stops doing so silently inherits the main-session model — the entire cost problem this project exists to solve); the skill routes only to namespaced roles that exist (a barescoutdoesn't resolve on a fresh install).It also pins the two now-unenforced rules, in both directions:
SKILL.mdstill carries both rules, so neither can be silently deleted;Credit
The plugin-with-guard-hooks shape isn't novel — Rylaa/fable5-orchestrator got there first. pilotfish deliberately doesn't follow it, for the portability reason above, and the README says so.
🤖 Generated with Claude Code