PRD 0011: teach the herd the agent protocol (#415)

The herd read paint instead of speaking protocol. Three symptoms, one cause:
state came from regex against a screen capture, a prompt left no evidence it
was ever submitted, and the roster stopped at the edge of the box.

Phase 1 — believe the engine. `herd hooks install claude` writes Claude Code's
own lifecycle hooks so state comes from the engine (`authority: hook`), merging
into the user's settings file rather than clobbering it. Sessions are launched
with MOSHCODE_HERD_NAME/DIR so a hook can name what it is reporting for, and a
hook fired outside a herd session does nothing and exits 0. `herd doctor`
checks substrate, manifest drift, stale reports, and — for the first time —
says what is wrong with rules.json instead of ignoring it. blocked gains
sub-kinds (permission/question/menu) that ride in --json and notifications.

Phase 2 — the ledger. Every prompt mints a task: id, transitions with
timestamps, and the output captured as a screen delta, in 0600 JSONL capped at
500 tasks per session. `herd tasks`, `herd task`, `herd log`, `herd stats` read
it back; blocked time is reported as what it is, human latency. The write goes
where the watcher's notification decision already happens, not in a second
poller. `wait --any/--all` replaces the polling loop every fan-out script had.

Phase 3 — the protocol. `herd serve` exposes the herd over A2A v0.3.0
(discovery, message/send, tasks/get, tasks/cancel) behind the moshcode login,
with no unauthenticated mode, loopback included, and --agent sessions withheld
unless --expose-autonomous. `herd remote add` puts a deployed agent on the
roster — a2a or a bare POST endpoint — and prompt/read/wait/kill work on it
unchanged, with auth from MOSHCODE_REMOTE_<NAME>_TOKEN and never in the
manifest.

Phase 4 — `herd eval` runs a dataset across engines with either the dataset's
own patterns or an engine as judge, exiting 0/4/5 so CI can tell a worse agent
from a broken box. gradient joins the tool table with its own runtime check.

119 new tests. Decisions taken while building are recorded at the end of the
PRD rather than edited into its requirements.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: ralyodio

.github/workflows/herd-eval.yml

@@ -0,0 +1,109 @@
+# NOT managed by the sh1pt Actions Fleet — hand-written for PRD 0011 R13.
+# The two fleet-managed workflows (ci.yml, test.yml) carry a pack hash and are
+# reverted by the next fleet sync, so this lives in its own file rather than as
+# a job added to one of them.
+#
+# What this gates: "the agent still passes the dataset", as a red/green check
+# next to `npm test`. It needs a real engine with real credentials, which a
+# runner does not have by default — so the whole job is a no-op until a
+# credential secret exists, and says so rather than going green by accident. A
+# gate that fails on every fork because nobody added a secret is a gate people
+# turn off.
+name: herd eval
+
+on:
+  workflow_dispatch:
+    inputs:
+      threshold:
+        description: "Score every engine has to reach (0-1)"
+        default: "0.8"
+      engines:
+        description: "Comma-separated engines to compare"
+        default: "claude"
+  pull_request:
+    paths:
+      - "evals/**"
+      - "src/herd-eval.mjs"
+      - "src/herd-tasks.mjs"
+      - ".github/workflows/herd-eval.yml"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: herd-eval-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  eval:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v7
+
+      # The `secrets` context is not available in a job-level `if`, so the
+      # check is a step that publishes an output the rest of the job reads.
+      - id: creds
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: |
+          if [ -n "$ANTHROPIC_API_KEY" ]; then
+            echo "have=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "have=false" >> "$GITHUB_OUTPUT"
+            echo "::notice title=herd eval skipped::no engine credentials on this runner — add ANTHROPIC_API_KEY to turn this check on"
+          fi
+
+      - uses: pnpm/action-setup@v6
+        if: steps.creds.outputs.have == 'true'
+
+      - uses: actions/setup-node@v7
+        if: steps.creds.outputs.have == 'true'
+        with:
+          node-version: '22'
+          cache: pnpm
+
+      - run: pnpm install --frozen-lockfile
+        if: steps.creds.outputs.have == 'true'
+
+      # The herd needs somewhere to run its sessions. Without tmux it would
+      # fall back to script(1), which works, but tmux is one apt away and is
+      # the substrate people actually use.
+      - run: sudo apt-get update && sudo apt-get install -y tmux
+        if: steps.creds.outputs.have == 'true'
+
+      - run: npm install -g @anthropic-ai/claude-code
+        if: steps.creds.outputs.have == 'true'
+
+      # `rules` as the judge, not an engine: the dataset carries its own
+      # expectations, and a judge that is itself an LLM would make a flaky
+      # check out of a deterministic one. Exit 4 is "below the threshold" and
+      # exit 5 is "the harness could not run" — the job distinguishes them so a
+      # broken runner does not get filed as a worse agent.
+      - name: run the dataset
+        if: steps.creds.outputs.have == 'true'
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: |
+          set +e
+          node bin/moshcode.mjs herd eval \
+            --dataset evals/moshcode.jsonl \
+            --engines "${{ inputs.engines || 'claude' }}" \
+            --threshold "${{ inputs.threshold || '0.8' }}" \
+            --json > eval.json
+          code=$?
+          set -e
+          cat eval.json
+          case "$code" in
+            0) echo "::notice title=herd eval::every engine is at or above the threshold" ;;
+            4) echo "::error title=herd eval::an engine scored below the threshold"; exit 1 ;;
+            5) echo "::error title=herd eval::the harness could not run (infrastructure, not the agent)"; exit 1 ;;
+            *) echo "::error title=herd eval::unexpected exit $code"; exit 1 ;;
+          esac
+
+      - uses: actions/upload-artifact@v4
+        if: steps.creds.outputs.have == 'true' && always()
+        with:
+          name: herd-eval-report
+          path: eval.json
+          if-no-files-found: ignore

README.md

@@ -263,9 +263,9 @@ moshcode agents claude -d --name api     # and an agent
 
 ```sh
 $ moshcode ps
-  api    claude  blocked   ~/src/coinpay   3m
-  logs   shell   idle      ~/src/coinpay   3m
-  work   shell   idle      ~/src/coinpay   3m
+  api    claude  blocked   ~/src/coinpay   3m   screen
+  logs   shell   idle      ~/src/coinpay   3m   screen
+  work   shell   idle      ~/src/coinpay   3m   screen
 
 ⚠ 1 waiting on you — moshcode attach api
 ```
@@ -317,9 +317,9 @@ Every session carries a state: `working`, `blocked`, `done`, `idle`, or
 `unknown`. `blocked` means a human decision is the only thing missing.
 
 ```
-  api       claude  blocked   ~/src/coinpay        12m
-  web       codex   working   ~/src/ugig.net        4m
-  audit     opencode  done    ~/src/moshpit-dns     1h
+  api       claude  blocked   ~/src/coinpay        12m   hook
+  web       codex   working   ~/src/ugig.net        4m   screen
+  audit     opencode  done    ~/src/moshpit-dns     1h   runtime
 ```
 
 State comes from one authority per session, never two. An engine that reports
@@ -416,6 +416,120 @@ await herdWait("api"); await herdWait("web");
 say(herdRead("api", { lines: 20 }));
 ```
 
+Fanning work out is easy; joining on it used to be a hand-rolled polling loop.
+`--any` returns on the first session to get there, `--all` when the last one
+has, and both take the same `--state` and `--timeout` as a single wait:
+
+```sh
+moshcode wait --any api web docs           # --json names the winner
+moshcode wait --all api web --state done
+```
+
+```js
+const first = await herdWait(["api", "web", "docs"], { any: true });
+await herdWait(["api", "web"], { states: ["done"] });
+```
+
+### Let the engine say what it is doing
+
+Reading a screen works and it rots — engines change their wording between
+releases and nothing tells you. When an engine has lifecycle hooks, install
+them once and its state comes from the engine itself:
+
+```sh
+moshcode herd hooks install claude
+✓ claude — 3 hooks installed (stop, notification, prompt-submit)
+```
+
+`moshcode ps` then reads `hook` in its last column instead of `screen`. The
+file is **merged, never clobbered** — your own hooks stay, and `hooks remove`
+takes out only what moshcode put in. A hook that fires outside a herd session
+does nothing and exits 0, so installing one cannot break an engine you run by
+hand, and the screen rules stay as the fallback for everything else.
+`moshcode herd doctor` says what is installed, what has drifted, and — for the
+first time — what is wrong with your `rules.json` instead of ignoring it.
+
+### What happened while you slept
+
+Every prompt through the herd mints a **task**: an id, its state transitions
+with timestamps, and the output it produced. `ps` still answers "now"; this
+answers "what happened".
+
+```sh
+$ moshcode herd tasks api
+  t-01  22:14  done      4m     "port the auth routes"
+  t-02  22:19  blocked   6h11   "run the migration"
+
+$ moshcode herd task t-02        # transitions, and what came back
+$ moshcode herd log api          # the raw state history
+$ moshcode herd stats api
+api          working 3h02 · blocked 6h11 · idle 1h40
+  blocked 6h11 over 2 spell(s) — that one is you
+```
+
+Blocked time is the herd's name for *human latency*: the agent was ready and
+you were asleep. Ledgers live in `~/.moshcode/herd/tasks/<session>.jsonl` at
+`0600`, capped at the last 500 tasks per session. From a script,
+`herdTasks(name)` and `herdTask(id)` return them as values.
+
+### Agents that are not on this box
+
+A deployed agent can be a herd member. Two kinds: `a2a` speaks
+[A2A v0.3.0](https://a2a-protocol.org/v0.3.0/specification/) (card discovery,
+`message/send`, `tasks/get`, `tasks/cancel`), and `run` is a bare endpoint that
+takes `POST {"prompt": …}` — the shape a `gradient agent deploy` prints.
+
+```sh
+moshcode herd remote add research https://agents.do-ai.run/…/production --kind run
+export MOSHCODE_REMOTE_RESEARCH_TOKEN=…       # never written to the manifest, never synced
+```
+
+```
+$ moshcode ps
+  api        claude   blocked   ~/src/coinpay      12m   hook
+  research   remote   idle      agents.do-ai.run   —     remote
+```
+
+`prompt`, `read`, `wait` and `kill` work on it unchanged, which is the point: a
+fan-out script across a local pty and a deployed agent contains no `if
+(remote)`. A remote's state is the *remote's claim* — `ps` says `remote` in the
+last column so it is never mistaken for something this box verified — and
+`kill` on one deregisters it here rather than reaching across the network to
+end somebody else's agent.
+
+### The herd, over A2A
+
+`moshcode herd serve` exposes this machine's herd to any A2A client: the herd's
+card at `/.well-known/agent-card.json`, each member at `/<name>/`,
+`message/send` → prompt, `tasks/get` → the ledger, `tasks/cancel` → interrupt.
+`blocked` is A2A's `input-required`; the states that do not map cleanly round
+down and carry the honest one in task metadata.
+
+```sh
+moshcode login                 # it verifies tokens against app.moshcode.sh
+moshcode herd serve            # 127.0.0.1:7683 by default
+```
+
+It is a shell on a socket and is treated like one: **no unauthenticated mode,
+loopback included**, a loud warning past `127.0.0.1`, and sessions started with
+`--agent` withheld unless you pass `--expose-autonomous` — an engine with its
+approvals bypassed plus a network prompt is the worst pairing on the menu.
+
+### Which engine is best at *this* repo
+
+Not a leaderboard run against engines nobody deploys on repos nobody has — your
+dataset, your engines, your machine:
+
+```sh
+moshcode herd eval --dataset evals/moshcode.jsonl --engines claude,codex --threshold 0.8
+```
+
+A row is `{"prompt": "…", "expect": "pattern"}` or
+`{"prompt": "…", "rubric": "…"}` (jsonl, json or csv). Scoring is either the
+dataset's own patterns or an engine acting as judge (`--judge claude`). Exit
+codes are distinct on purpose — `0` pass, `4` below the threshold, `5` the
+harness could not run — because CI has to tell a worse agent from a broken box.
+
 ### After a reboot
 
 ```sh

evals/moshcode.jsonl

@@ -0,0 +1,8 @@
+{"id": "state-authority", "prompt": "In src/herd-state.mjs, what value does the `authority` field take when a session's state came from its engine's own lifecycle hook rather than from its screen? Answer with the single word and nothing else.", "expect": "\\bhook\\b"}
+{"id": "vocabulary-file", "prompt": "Which file in this repo holds the moshscript command vocabulary — the verbs a .mosh script can call? Answer with the repo-relative path and nothing else.", "expect": "src/commands\\.mjs"}
+{"id": "safe-answer", "prompt": "The herd's state classifier has one state it treats as always safe, never blocking a launch, and returns when no rule matched. Name it in one word.", "expect": "\\bunknown\\b"}
+{"id": "wait-exit-codes", "prompt": "`moshcode wait` exits with distinct codes so scripts can branch on the outcome. Which exit code means the wait timed out? Answer with the number only.", "expect": "(^|\\D)2(\\D|$)"}
+{"id": "substrate-fallback", "prompt": "The herd runs sessions on tmux when it is available. Name the program it uses to allocate a pty when tmux is not installed. One word.", "expect": "script"}
+{"id": "manifest-mode", "prompt": "What file mode does the herd write its session manifest with, and why? Give the octal mode first.", "expect": "0?600"}
+{"id": "blocked-meaning", "prompt": "In the herd's roster, what does the state `blocked` mean? Answer in one short sentence.", "rubric": "The answer must say that the session is waiting on a human decision or input — an approval, a question, or a menu — and not that it is stuck, crashed, or busy working.", "expect": "human|you|decision|input|approval|ask"}
+{"id": "no-second-api", "prompt": "Does moshcode expose a separate socket API for agents to drive the herd, apart from the CLI? Answer yes or no, then one sentence.", "rubric": "The answer must be 'no': the CLI with --json on every verb is the one surface, and `herd serve` is that same surface answering a socket rather than a second API.", "expect": "^\\W*no\\b"}