Skip to content

Repackage pilotfish as a Claude Code plugin#11

Closed
dromsak wants to merge 6 commits into
Nanako0129:mainfrom
dromsak:v2-plugin
Closed

Repackage pilotfish as a Claude Code plugin#11
dromsak wants to merge 6 commits into
Nanako0129:mainfrom
dromsak:v2-plugin

Conversation

@dromsak

@dromsak dromsak commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Proposal

Repackage pilotfish as a Claude Code plugin.

/plugin marketplace add Nanako0129/pilotfish
/plugin install pilotfish@pilotfish
/pilotfish

This is a proposal, not a fait accompli — it's a big change and I'd rather you shape it than merge it as-is. The underlying bug it protects against is also fixable in the current v1.x architecture without this replatform — see my comment on #10 and the fix-subagent-detach branch. Take that and leave this if the plugin shape isn't for you.

Important

This PR changed course, and you should know why. It originally proposed a PreToolUse guard hook as its centrepiece — "the rules are now enforced, not requested". I've removed it. I ran this plugin on a Windows machine and the hook could not launch: it was a Python script invoked bare via its #!/usr/bin/env python3 shebang and the Unix executable bit, and native Windows honors neither. Python was never the real problem — per Claude Code's docs no interpreter is guaranteed on a machine running it, not Python and not even node (the native standalone installer never puts node on PATH). There is no runtime a plugin can ship a hook against and still run everywhere Claude Code runs. So this PR is now just the replatform, and pilotfish stays pure markdown + JSON. Details in Why no guard below.

This resolves #8 (install) — and the update pain behind it

#8 is the install flow breaking because the agent-facing runbook trips WebFetch's prompt-injection protection, and the fallback is a clone-and-paste ritual. Your own last comment there was "what if we turned it into a cc plugin? That should simplify." — this is that.

  • Install: there is no runbook to fetch and no imperative document for an AI to misclassify. /plugin install is native, first-party Claude Code — it can't be flagged as adversarial remote content, because it isn't fetching a document at all. The pinned-clone workaround goes away.
  • Update: today an update means re-running the idempotent installer, which re-hits the exact same fetch problem. A plugin updates through the marketplace on a version bump — /plugin, no re-fetch, no re-merge. The four-file version-stamp coupling that made releases fragile is gone too.

So #8 isn't worked around, it's designed out. If you merge this, it closes #8.

Why a plugin

The install stops being a merge into ~/.claude/. No policy block in your global CLAUDE.md, no six files copied into ~/.claude/agents/, no curl | sh-shaped trust problem, no WebFetch prompt-injection caveat. Uninstalling removes every trace.

The policy stops being an ambient context tax. Today the orchestration block loads into every session of every project, whether you're orchestrating or not — which is also why it needs its self-disabling preamble ("If you are running as a subagent role… ignore this section entirely"). As a skill, it loads when you type /pilotfish and not before, and it never reaches subagents at all. The preamble's job disappears rather than being done better.

The version-stamp machinery disappears. VERSION + the policy comment + two README clone-pins + a test enforcing all four move together + RELEASING.md — all of it exists to manage a hand-merged global config. A plugin has one version, in its manifest.

Why no guard

A guard hook is genuinely the right shape for two of pilotfish's rules. #6 proved it: subagents kept re-delegating, and the fix wasn't a firmer instruction, it was disallowedTools: Agent, Workflow. Capability removal beat persuasion, decisively. I built the same thing for detaching and for the built-in Explore, and it worked — on Linux and macOS.

It cannot be made portable:

  • A hook is a script, and a script needs an interpreter Claude Code doesn't guarantee. Not Python. Not node — the native standalone installer never puts it on PATH. There is no safe assumption to make here.
  • Native Windows honors neither a #! shebang nor the Unix executable bit. The idiomatic hook — ship a script, mark it executable, name it in hooks.json — is Unix-only by construction. It doesn't degrade on Windows; it never launches.

And the failure mode is what settles it. A guard hook must fail open — a malformed payload has to allow the call, or a bug in the guard locks a user out of their own session. But a missing interpreter also fails open, and silently: on exactly the machines where the hook can't run, you get no enforcement and no signal that enforcement is missing. A rule you believe is enforced is a rule you stop watching. A door that isn't there beats a door you wrongly believe is locked.

So the two rules go back to policy, stated as prohibitions the model can read:

Rule Status
Subagents must never detach (nohup, setsid, disown, trailing &, run_in_background) policy — not enforced
Never invoke the built-in Explore; use pilotfish:scout policy — not enforced
scout is read-only genuinely enforcedtools: Read, Glob, Grep gives it no Bash
Roles can't re-delegate genuinely enforceddisallowedTools: Agent, Workflow (#6, unchanged)

The two capability-removal mechanisms that are portable (a tools allowlist, disallowedTools) are both still in use. Only the hook is gone.

One thing I want to flag as a real regression, not paper over: the Explore cost problem is unsolved. Since v2.1.198 the built-in inherits the main-session model, so every search it runs from a Fable/Opus session bills at frontier rates. A plugin cannot shadow it back to a cheap tier — plugin agents are namespaced, so pilotfish:Explore is a different agent from the built-in and overrides nothing (I tested this; the override silently did not take). Shadowing is unavailable, a hook is unshippable, so all that's left is the skill telling the orchestrator, in as many words, not to use it. Nothing stops a session that ignores that.

Consequence, unchanged from the original proposal: five roles, not six. Explore.md is gone; scout is the recon role.

What's removed

templates/, install/, VERSION, RELEASING.md, tests/test_policy.py — all superseded. docs/ and CHANGELOG.md carry over.

What can't be ported

A plugin cannot set your main-session model. model: "best" + fallbackModel is now a documented one-line manual step (/model best) rather than something the installer writes. Honest trade: the plugin gains a clean uninstall and loses the ability to touch your settings — which are the same property.

Tests

8 tests, dependency-free, green (python3 -m unittest discover -s tests — a contributor dep; the shipped plugin has none).

tests/test_plugin.py pins the wiring: manifests parse; every role pins its own model in frontmatter (a role that stops doing so silently inherits the main-session model — the entire cost problem this project exists to solve); the skill routes only to namespaced roles that exist (a bare scout doesn't resolve on a fresh install).

It also pins the two now-unenforced rules, in both directions:

  • no role is told to detach, and every shell-capable role is told not to;
  • SKILL.md still carries both rules, so neither can be silently deleted;
  • no prompt claims detaching is "blocked." It was, in the first draft of this PR — and when the hook went, that sentence became a lie that left no rule behind at all. A model told the door is locked stops holding it shut. Both negative controls verified: reintroduce the false claim, or delete a rule, and a test goes red.

Credit

The plugin-with-guard-hooks shape isn't novel — Rylaa/fable5-orchestrator got there first. pilotfish deliberately doesn't follow it, for the portability reason above, and the README says so.


🤖 Generated with Claude Code

@dromsak

dromsak commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Heads-up: I've pushed three more commits to this branch. Two are substantive, one is taste — flagging the split explicitly so you can review (or reject) them independently, and I'm happy to pull any of them out into their own PR if you'd rather this branch stayed just the repackage.

0d15f22 — routing: the execution tier moves to Sonnet, Haiku is dropped.
scout Haiku → Sonnet/low, executor Opus/medium → Sonnet/high, verifier stays Opus and goes to effort high. mech-executor and security-executor are untouched. Three reasons:

  • Scout output is unverified input to everything downstream. The verifier gate covers executor work, not reconnaissance — so a wrong file:line from a scout becomes an executor confidently editing the wrong thing, with nothing to catch it. Haiku 4.5 → Sonnet 5 is a large enough capability gap that this stops being theoretical, and recon is the one place cheapness compounds into error rather than just costing a retry.
  • On subscriptions, Haiku was never the cheap option it appeared to be. The weekly limit is a shared all-models bucket plus an additional Sonnet-only bucket — so Haiku draws on the scarce one while Sonnet draws on dedicated headroom. For the subscriber this plugin is aimed at, moving recon to Sonnet can cost less of the resource that actually runs out.
  • Opus on executor was over-insurance: quality is bought back by the verifier more cheaply than by upgrading the executor, so paying Opus rates for routine implementation and running a verifier was paying twice for the same guarantee. The verifier's effort bump is where that budget goes instead.

Net effect: the shipped configuration is now exactly the one Anthropic benchmarked — a frontier orchestrator with Sonnet 5 workers, 96% of all-frontier performance at 46% of cost. The three Sonnet roles stay three files because the Agent tool has no effort parameter; effort is settable only in frontmatter, so collapsing them would run bulk mechanical work at effort: high for nothing.

95d4bc7 — guard bugfix: an & inside a heredoc body was denied.
This is a real false positive, found in the wild — it blocked an agent mid-run on a comment reading # strip docstrings & constant string values. A heredoc body is data on some program's stdin; the calling shell never interprets it, so &/nohup/setsid in there is prose. strip_heredocs() blanks the body space-for-space and deliberately leaves the opening line intact, since that's where a real trailing & would sit. Two tests in opposite directions: bodies are data (ALLOW), and cat <<EOF & / nohup cat <<EOF / a detach after the terminator still DENY — an over-broad fix here would silently disable the whole shell-detach leg for any command containing a heredoc.

39cf3a0 — README cut from 215 lines to 89, in both languages.
This one is taste, not correctness, and it's the one to push back on if you disagree — no hard feelings, and it lifts out cleanly. The argument: the README had grown a table of contents, an install walkthrough, a trust section, a fallback table and a twelve-row FAQ, all competing with the three things a reader needs in the first thirty seconds — what it is, what it saves (cost and speed; the speed argument was never actually made), and the diagram. Nothing is deleted, only relocated to docs/design.md, which the README now links to. The zh-TW README was rebuilt to match rather than dropped, reusing the existing translation verbatim wherever the English was unchanged.

Guard behaviour is unchanged by the routing commit (verified at bytecode level — only human-readable strings moved). Test suite is green: 21 tests, up from 19.

@dromsak dromsak changed the title Repackage pilotfish as a Claude Code plugin, with a guard hook that enforces the rules Repackage pilotfish as a Claude Code plugin Jul 13, 2026
dromsak and others added 6 commits July 14, 2026 18:29
… hook

BREAKING CHANGE: pilotfish is now a plugin, not a global-config merge.

  /plugin marketplace add Nanako0129/pilotfish
  /plugin install pilotfish@pilotfish
  /pilotfish

Nothing is written into ~/.claude/ or into your projects, and uninstalling removes
every trace. templates/, install/, VERSION, RELEASING.md and the four-way version-stamp
coupling are gone -- all of that existed to manage a hand-merged global config.

The orchestration policy becomes a skill, so it loads when you type /pilotfish rather
than into every session of every project. Its self-disabling preamble ("if you are a
subagent, ignore this section") is deleted: a skill never reaches subagents, so the
confusion it patched cannot occur.

THE GUARD

A policy is a request; a missing capability is a fact. A PreToolUse hook now enforces
what the policy could only ask for. The discriminator is the presence of `agent_type` in
the hook payload -- absent in the main session, present in a subagent (verified against
real captured payloads):

  * subagents may not set run_in_background
  * subagents may not detach (nohup / setsid / disown / trailing &)
  * the built-in Explore agent is denied; recon routes to `scout`

The main session keeps every capability. Its backgrounding is the path that works, and
blocking it would disable the orchestrator. The guard fails open.

WHY SUBAGENTS MAY NOT DETACH (established by experiment, not assumption)

A foreground command exceeding its `timeout` is not killed -- the harness promotes it to
a background task and promises a completion notification. Whether that promise is kept
depends on how the orchestrator spawned the agent:

  * spawned with run_in_background=true  -> the promoted process survives the turn
    boundary AND the agent returning, runs to completion, has its stdout captured, and
    its notification re-invokes the agent. Observed surviving 59s/69s/81s past a turn
    boundary and 141s past agent termination, zero signals across four trials.
  * spawned in the foreground            -> SIGTERMed ~4s after the agent returns. Work
    destroyed, captured output truncated mid-stream. Reproduced exactly.

nohup/setsid dodge that SIGTERM by escaping the process group -- which is why the pattern
gets adopted -- but they also escape the harness's task tracking: no task id, no captured
output, no notification. Detaching launders a destroyed result into an orphaned one.

So subagents no longer detach. They run in the foreground with an explicit timeout and
hand back anything that cannot finish inside one; long-running processes belong to the
orchestrator, the only context whose background tasks are both tracked and notified. This
also makes spawning agents with run_in_background=true load-bearing for correctness, not
merely for cost.

WHY EXPLORE IS BLOCKED RATHER THAN SHADOWED

Plugin agents are namespaced (pilotfish:Explore), so a plugin cannot shadow the built-in
Explore the way a user-level ~/.claude/agents/Explore.md can -- tested, and the override
silently did not take. Since v2.1.198 the built-in inherits the main-session model, so
every background search bills at frontier rates. The guard blocks it and routes recon to
`scout` (haiku), preserving the behaviour without depending on shadowing.

Five roles now, not six.

WHAT CANNOT BE PORTED

A plugin cannot set the main-session model. `model: "best"` + fallbackModel is now a
documented manual step (/model best) rather than something an installer writes. That is
the same property as the clean uninstall, stated honestly.

TESTS

16 dependency-free tests. test_guard.py pins both asymmetries -- including that the main
session KEEPS run_in_background/detach -- and that the guard does not false-positive on
&&, 2>&1, & inside quotes, or substrings. test_plugin.py checks the manifests parse, the
hook script exists and is executable (a hook pointing at a missing script fails silently,
which would turn every guarantee back into a suggestion), every role pins its own model,
and the skill routes only to roles that exist. Both guards were also verified live in a
real session via --plugin-dir, not only in unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scout moves Haiku -> Sonnet (effort low); executor moves Opus/medium ->
Sonnet/high; verifier stays Opus and goes to effort high. mech-executor
(Sonnet/low) and security-executor (Opus/high) are unchanged. The shipped
configuration is now exactly the one Anthropic benchmarked: a frontier
orchestrator with Sonnet 5 workers, at 96% of all-frontier performance for
46% of the cost.

Scout output is unverified input to everything downstream -- the verifier
gate covers executor work, not reconnaissance -- so a wrong file:line from
a scout becomes an executor confidently editing the wrong thing, with
nothing to catch it. Haiku 4.5 -> Sonnet 5 is a large enough capability gap
that this stops being theoretical.

On subscriptions Haiku was never the cheap option it looked like: the weekly
limit is a shared all-models bucket plus an additional Sonnet-only bucket,
so Haiku draws on the scarce one while Sonnet draws on dedicated headroom.

Opus on executor was over-insurance. Quality on cheap executors is bought
back by the verifier more cheaply than by upgrading the executor itself;
paying Opus rates for routine implementation *and* running a verifier was
paying twice for the same guarantee. The verifier's effort bump is where
that budget goes instead.

The three Sonnet roles stay three files: the Agent tool has no effort
parameter, so effort is settable only in frontmatter, and collapsing them
would run bulk mechanical work at effort high for nothing.

Guard behaviour is unchanged; only the text of its Explore denial, which
names the tier scout is pinned to. Docs and the zh-TW mirrors updated to
match. The 74% Haiku row is dropped from the savings table -- quoting
savings for a tier we no longer ship would be false advertising.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fnkD9CdpQUvFTPQGz9GzC
215 lines -> 89, in both languages. The README had grown into a manual:
a table of contents, a full install walkthrough, a trust section, a
fallback-story table, and a twelve-row FAQ. Almost none of that is what a
reader needs in the first thirty seconds, and all of it competed with the
three things that are: what pilotfish is, what it buys you (cost and
speed), and the shape of the thing.

Nothing is deleted, only relocated. The fallback story, the tuning knobs,
the FAQ, and the design rationale live in docs/design.md, which the README
now links to. Install collapses into the intro (it was always two
commands); uninstall is one clause; trust & security folds into the guard
section, where the "read the ~100 lines yourself" argument actually lands.

The Why section now leads with cost AND speed -- the speed argument was
never made explicitly, though it is half the point of routing bulk work to
a faster model at low effort and overlapping independent agents.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fnkD9CdpQUvFTPQGz9GzC
A heredoc body is data on some program's stdin — the calling shell never
interprets it — so an `&`, `nohup`, or `setsid` inside one is prose, not a
control operator. The guard scanned the whole command string, so a subagent
running `python3 - <<'EOF'` was denied for "backgrounding a command" if the
script it piped in merely contained an ampersand. An `&` in a code comment
was enough.

Found in the wild: it blocked a verifier agent mid-run, on a comment reading
"# strip docstrings & constant string values".

This is the failure mode the guard's own docstring warns about and that
test_legitimate_commands_are_not_false_positives exists to prevent — a guard
that blocks legitimate work gets switched off, and then it protects nothing.

strip_heredocs() blanks the body space-for-space, preserving every offset,
and deliberately leaves the line that OPENS the heredoc intact — that is
where a real trailing `&` or `nohup` would sit. It runs before strip_quotes,
which would otherwise eat a quoted delimiter (`<<'EOF'`) and leave the body
looking like live shell.

Two tests, in opposite directions: heredoc bodies are data (ALLOW), and
`cat <<EOF &`, `nohup cat <<EOF`, and a detach after the terminator still
DENY — because an over-broad blanking fix would silently disable the whole
shell-detach leg for any command containing a heredoc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fnkD9CdpQUvFTPQGz9GzC
The guard was a Python `PreToolUse` hook invoked bare, relying on its
`#!/usr/bin/env python3` shebang and the Unix executable bit. Native
Windows honors neither, so it could never launch there at all — the
plugin was Unix-only by construction, not merely Python-dependent.

And Python was never the real problem. Per Claude Code's own docs no
interpreter is guaranteed on a machine running it: not Python, and not
even `node`, since the native standalone installer never puts `node` on
PATH. There is no runtime a plugin can ship a hook against and still run
everywhere Claude Code runs. pilotfish's premise is that it should.

The failure mode settles it. A guard hook must fail open, or a bug in the
guard locks you out of your own session. But a *missing interpreter* also
fails open, silently — so on exactly the machines where the hook cannot
run you get no enforcement and no signal that enforcement is missing. A
rule you believe is enforced is a rule you stop watching. A door that
isn't there beats a door you wrongly believe is locked.

So the two rules go back to policy, stated as genuine prohibitions:

- Never invoke the built-in `Explore`; use `pilotfish:scout`. The cost
  finding is unchanged — it inherits the main-session model and bills
  every search at frontier rates — but nothing now stops a session that
  ignores it.
- Subagents must never detach a process. `scout` is the one exception
  that survives as real enforcement: its `tools: Read, Glob, Grep`
  allowlist gives it no Bash to detach with.

Both prompts previously said detaching "is blocked for subagents". That
claim was false the moment the hook went, and worse than silence: a model
told the door is locked stops holding it shut. `tests/test_plugin.py` now
pins both halves — that no role is told *to* detach, and that every
shell-capable role is told *not* to — plus that SKILL.md still carries
both rules, so neither can be deleted without a test going red.

Also fixes an effort table in docs/design.md that still claimed
executor/verifier run at `medium`; 0d15f22 moved both to `high`.

The plugin is now markdown and JSON. Nothing to install, nothing to
interpret, runs anywhere Claude Code runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pT25EttAtiXYfsvutWyhE

Copy link
Copy Markdown
Owner

Maintainer note: I rebased this branch onto v1.2.0 (cf9f854) and resolved the conflicts while preserving the plugin direction. I’ve left the original PR description unchanged; for review context, the rebased head is now 71d92bc.

The current branch also carries the v1.2 phase gates and compatibility requirements:

  • seven namespaced plugin roles, including the plan/security review roles introduced in v1.2
  • skill invocation is /pilotfish:pilotfish
  • a Claude Code runtime preflight requiring >= 2.1.207 before delegation or writes
  • the current suite contains 16 tests

Validation completed after the rebase:

  • python3 -m unittest discover -s tests -v — 16 passed
  • strict plugin and marketplace validation — passed
  • JSON, Markdown relative links, and git diff --check — passed

GitHub currently reports the PR as mergeable with a clean merge state.

Copy link
Copy Markdown
Owner

@dromsak, thanks for the work here. The Plugin packaging direction is solid, and most of the technical work carries forward into v2 regardless of how the activation question lands.

The reason this can’t merge as-is is straightforward: pilotfish’s core contract is that orchestration is globally active—it is present in every session without requiring the user to invoke it. This PR replaces that behavior with explicit invocation, and that contract should not be dropped before we know whether the Plugin platform can preserve it.

That is what #12 is for. v1 remains the supported baseline while the Plugin path is evaluated for equivalent ambient activation. The result will determine whether v2 succeeds v1 or exists alongside it as a Session-mode option.

The following work carries forward regardless:

  • Plugin packaging and marketplace installation
  • Session-mode Skill as the opt-in baseline
  • Namespaced agents, runtime preflight, and Windows/interpreter portability findings
  • Model and effort routing, accepted independently for v1

A focused PR against main containing only the routing change would be welcome. If you would rather not extract it, I can carry it forward with explicit attribution.

No rework is needed on this PR. I’ll leave it open for a few days in case there is any additional context you want to preserve, then close it as superseded by #12. PR #11 will remain linked there as the design provenance for the Plugin work and the findings that carry forward.

Thanks again—this materially shaped the v2 plan even though the activation boundary is not something I can adopt yet.

Copy link
Copy Markdown
Owner

Closing this PR as superseded by #12.

The Plugin packaging work remains valuable design provenance for the v2 activation decision. The marketplace packaging, Session-mode skill, namespaced roles, runtime preflight, Windows/interpreter portability findings, and model/effort routing research all carry forward.

The unresolved boundary is the activation contract: pilotfish v1 remains ambient global orchestration, while this branch requires explicit Skill invocation. Future Plugin work will continue under #12, so no further rework is requested on this branch.

@dromsak, thank you for the substantial implementation and research that shaped the v2 direction.

@Nanako0129 Nanako0129 closed this Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installation problem

2 participants