Skip to content

✨ workflow supervision (herdr/terminal): view, steer, and hijack smithers agents inside herdr/terminal - #1430

Merged
roninjin10 merged 34 commits into
smithersai:mainfrom
N0xMare:nox/feat-herdr
Aug 11, 2026
Merged

✨ workflow supervision (herdr/terminal): view, steer, and hijack smithers agents inside herdr/terminal#1430
roninjin10 merged 34 commits into
smithersai:mainfrom
N0xMare:nox/feat-herdr

Conversation

@N0xMare

@N0xMare N0xMare commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What this does

Smithers 0.33.0 adds a fully degradable terminal supervision plane backed by herdr: a complete cockpit outline plus an adaptive set of node panes, with direct steer, approve/deny, and resumable hijack controls.

Smithers remains authoritative for execution, isolation, and durability. Herdr renders and relays; if it is absent, unreachable, or protocol-incompatible, optional mirror paths degrade safely and explicit mutating commands fail before making changes.

Screenshot 2026-07-27 at 10 41 46 AM

Feature surface

  • Herdr mirror — one workspace per run, complete cockpit overview, adaptive node-tab cap, attention promotion for gates/failures/hijacks, and terminal outcome markers. Commands: smithers herdr status|attach|open|clean.
  • Workflow supervisor (smithers supervisor, alias top) — gateway-first with a --direct local-store path, graph-primary outline, activity strip, and node/gate detail panes.
  • Steer — durable instructions injected at the next agent generate() boundary, with atomic/idempotent SteerQueued publication and deterministic consumption/expiry.
  • Hijack — resumably park a run and hand off the native agent session without racing sibling agents.
  • First-class effort — derive at spawn, persist on attempts, and render through direct and gateway paths.
  • Detached approval recovery — claim-fenced supervisor resume for approved and genuinely denied gates, including scoped runs that would previously be crowded out by a 500-row page.
  • Post-failure autopsy classification — skip human-denied, cancelled, and quota-parked terminals.

Safety hardening in the refreshed head

  • protocol mismatches are inspectable via herdr status but blocked before workspace/tab/pane mutations
  • Herdr tabs are owned by exact agent identity, never by a presentation-label collision with a user's tab
  • same-run workspace creation is coalesced and reconciled; a loser only closes the workspace it created
  • NDJSON frames are byte-bounded and oversized peers are disconnected
  • Claude settings containing folded secrets use private temporary files that are removed on success, failure, and partial construction
  • Claude transcript CWD discovery reads a bounded 64 KB prefix instead of loading an arbitrarily large JSONL file
  • steer row + event publication is one transaction and retries with the same steer ID are no-ops
  • recorded steer migrations validate/repair their owned index and fail closed on incompatible table/name collisions

Migrations

No manual migration is required. The final merge order reserves:

Preview ledger IDs (0035_attempt_effort_column, 0036_add_steers) are preserved and converge safely to the official IDs.

Behavior changes to existing commands

  1. smithers tail is now first-class; -n selects a node. Use smithers logs <run> -n N for event line count.
  2. logs follows live runs only on a TTY unless -f is supplied.
  3. A successful pause exits 0.
  4. supervisor uses the workspace Gateway by default; --direct reads the local store.
  5. Inside Herdr, monitor/UI commands and post-run HTML summaries print URLs instead of opening a browser. Override with SMITHERS_NO_BROWSER=0.

Merge order

This PR remains independently based on main, but should merge after #1463#1449#1461. The combined verification explicitly retained 0035_agent_checkpoints → 0036_attempt_effort_column → 0037_add_steers; generated declaration and llms conflicts should be regenerated during the final refresh.

Current verified head: 1f9a8039f37ee5739cb820f89a622494abaa4bb8.

Verification

  • standalone Herdr package: 113 passed
  • combined CLI/supervisor/protocol/hijack run: 34 passed
  • testing scheduler/artifact suites: 91 passed standalone; 68 passed in the combined stack check
  • DB: checkpoint persistence, migration repair, pre-limit approval query, SQLite atomicity, and PGlite/PostgreSQL atomicity all pass
  • combined real engine steer runtime: 11 passed, including first start, retry, loop, ordering, expiry, rollback, interleaving, idempotency, and child-run addressing
  • combined checkpoint persistence and completed-task fork/resume E2E pass
  • agents, DB, engine, Herdr, CLI, and testing typechecks pass in the combined stack
  • affected package builds, docs, llms, dependency boundaries, changed-file formatting, and diff checks pass

Baseline-only repository issues remain unchanged: root formatting reports 24 pre-existing files, the declaration gate reports pre-existing integrations declaration drift, and the standalone main-based branch lacks #1463's Effect 4 managed-runtime bridge. The combined stack verification passes those affected runtime paths.

Known limitations

  • Mid-turn steer applies at the next generate() boundary, not between tool calls.
  • Cross-process workspace creation can only reconcile after create because Herdr exposes no server-side idempotency key/CAS; a process crash in that narrow window can strand a duplicate.
  • PostgreSQL outline ordering remains best-effort via ctid.
  • supervisor failures exit 4 (still non-zero).

@N0xMare N0xMare changed the title ✨ Live workflow supervision (herdr): steer, hijack, effort, supervisor cockpit ✨ workflow supervision (herdr/terminal): view, steer, and hijack smithers agents inside herdr/terminal Jul 27, 2026
@refcell

refcell commented Aug 10, 2026

Copy link
Copy Markdown

LGTM

@roninjin10
roninjin10 force-pushed the nox/feat-herdr branch 2 times, most recently from bed59b9 to abe3444 Compare August 11, 2026 05:26
N0xMare and others added 24 commits August 10, 2026 22:57
…pervisor cockpit

Mirror a Smithers run into a herdr terminal workspace and supervise it from the
same terminal as your coding agent: a pane per agent node, a cockpit outline of
the whole graph, and a dual-control dock to steer a running agent with one key
or hijack its session and drive it yourself. Fully degradable — Smithers always
owns execution, isolation, and durability; herdr only renders and relays.

- herdr mirror: adaptive layout (workspace per run, cockpit tab, tab/pane per
  agent node with attention promotion), terminal outcome markers, silent no-op
  when no herdr server is running. `smithers herdr status|attach|open|clean`.
- Workflow supervisor (`smithers supervisor`, alias `top`): gateway-sourced by
  default with a `--direct` local-store path, graph-primary outline, activity
  strip, Enter into a node detail tab (or an approve/deny pane for a gate).
- Steer: durable instruction queued against a node, injected as a user turn at
  its next generate boundary (SteerQueued → SteerConsumed, expiry at terminal),
  backed by the new _smithers_steers table (migration 0034_add_steers).
- Hijack: park a run resumably and hand off the agent's native session; the
  pane refuses a run-wide hijack while sibling agents are in flight.
- First-class reasoning effort: derived at spawn, persisted to the new
  _smithers_attempts.effort column (migration 0033_attempt_effort_column), and
  rendered on both the direct-db and gateway display paths. Claude's settings
  sources now fold into a single spec-accurate --settings flag that no longer
  clobbers a user's settings file.
- Approve/deny auto-resume for parked detached runs, claim-fenced; composes with
  the supervisor's bounded unattended-resume retries (attended resumes are
  exempt by claim-owner prefix).
- Cause-classified post-failure autopsies: skip human-denied, cancelled, and
  quota-parked terminals.
- The bundled `smithers` skill is now herdr-aware: in a herdr workspace (or with
  a supervisor open) the agent uses the terminal cockpit instead of opening the
  browser Monitor.

Docs: herdr integration guide, workflow supervisor guide, 0.32.0 changelog.
Rebasing onto main needed the generated artifacts rebuilt rather than merged:
regenerate pnpm-lock.yaml/bun.lock for the new @smithers-orchestrator/herdr
workspace package and the agents declaration bundle, and bump the documented
workflow.run flag count to 42 for the new --herdr flag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sting to docs.json

Fixes orphaned pages flagged by review: docs/integrations/herdr.mdx,
docs/guide/workflow-supervisor.mdx, and
docs/guides/token-free-visibility-testing.mdx were shipped without
navigation entries, so they were invisible on the Mintlify site despite
being referenced elsewhere and included in the llms bundles. Regenerated
llms bundles via pnpm docs:llms.

Co-Authored-By: Codex Sol <noreply@openai.com>
ClaudeCodeAgent folded a user --settings FILE's parsed JSON into the
single merged --settings flag and re-emitted it inline as
JSON.stringify(mergedSettings) on the spawned CLI's argv, even when
Smithers had nothing to add (no effort, no durability socket). Settings
files routinely hold secrets (env.ANTHROPIC_API_KEY, apiKeyHelper
output), and argv is world-readable via `ps` on shared hosts, so this
silently converted a previously-safe file reference into a process-
listing secret leak.

Fix: only read+fold a settings file when Smithers actually has
something to inject; a pure passthrough now emits the file path
untouched. When folding is required, write the merged object to a
private (mode 0600) temp file and pass that path instead of inlining
JSON onto argv.

Co-Authored-By: Codex Sol <noreply@openai.com>
0033_attempt_effort_column and 0034_add_steers collided with
main's 0033_scorer_identity / 0034_run_cancellation_attribution
and 0034_add_steers was array-ordered before 0033_scorer_identity.
Renumber to 0035/0036, fix array order, and give
0036_add_steers a real checksumForStatements checksum instead of
a nonexistent .sql file path.

Co-Authored-By: Codex Sol <noreply@openai.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Codex Sol <noreply@openai.com>
Co-Authored-By: Codex Sol <noreply@openai.com>
Co-Authored-By: Codex Sol <noreply@openai.com>
Co-Authored-By: Codex Sol <noreply@openai.com>
…column

The async external/Cloudflare bootstrap (runSmithersSchemaInitSqliteAsync)
does not run the versioned migration ledger, so CREATE TABLE IF NOT EXISTS
never adds the 0036 effort column to a pre-existing _smithers_attempts
table. Persisting effort then fails against stores created before the
column landed. Apply LEGACY_COLUMN_MIGRATIONS explicitly in the async path,
mirroring the run-cancellation column upgrade already there.

Co-Authored-By: Codex Sol <noreply@openai.com>
…tion

Steer expiry only ran from engine-owned terminal paths, so a steer queued
on a parked run stayed queued forever when the CLI terminalized it via
cancel/down without another engine boot; a crash between the terminal
commit and the separate expiry loop stranded it the same way. Fold expiry
into finalizeCancelledRun's cancel-finalization transaction so every
terminal cancel transition expires (and durably records SteerExpired for)
still-queued steers atomically. Idempotent on replay — markSteerExpired
only touches rows still queued.

Co-Authored-By: Codex Sol <noreply@openai.com>
roninjin10 and others added 7 commits August 10, 2026 22:57
The single generated index.d.ts omitted runtime exports such as
detectHarnessCommand and shouldSplitCockpit, and the "./*" export mapped
every wildcard subpath's types back to that same incomplete file. Emit one
declaration per entry (tsup dts per module) so each "./src/*.js" subpath
resolves to its matching "./src/*.d.ts", and point the wildcard types at
"./src/*.d.ts". ignoreDeprecations is required for the tsup dts build.

Co-Authored-By: Codex Sol <noreply@openai.com>
Co-Authored-By: Codex Sol <noreply@openai.com>
Co-Authored-By: Codex Sol <noreply@openai.com>
William Cory added 3 commits August 11, 2026 00:11
`describe.skipIf(!isHerdrInstalled())` only asked whether `herdr` was on
PATH. The client pins wire protocol 16 (herdr 0.7.3) and deliberately fails
closed on any other protocol, so a developer carrying a NEWER herdr — the
current release, 0.8.0, speaks protocol 19 — ran the real-server suites and
got 10 hard failures asserting against the exact mismatch errors the product
is supposed to return.

Add `installedHerdrProtocol()` (reads `herdr status client`, a local
synchronous introspection needing no running server, so it is safe where
`skipIf` is evaluated at module load) and `isCompatibleHerdrInstalled()`,
and gate the seven real-server suites on it.

Verified with herdr 0.8.0 on PATH: apps/cli herdr suites go 10 fail -> 0
fail / 24 skip, and packages/herdr is 71 pass / 49 skip / 0 fail both with
and without the binary present.
The version-pinning section claimed a non-16 protocol "logs a warning and
keeps going". Only the optional mirror paths do that; explicit `herdr`
commands fail closed — verified against a real herdr 0.8.0 server,
`herdr status` returns HERDR_PROTOCOL_MISMATCH with exit 4 and reports that
no mutating call was made, which is what `probeCompatibleHerdr` is built to
do.

Document both paths, and note that herdr releases after 0.7.3 speak higher
protocol numbers, so herdr features are (safely) inert until this client is
updated — otherwise "nothing happens" reads as a bug.

Bundles regenerated with `pnpm docs:llms`.
The rebase left `@smthrs/herdr` and `@pierre/diffs` appended after
`@smthrs/vcs`, out of order in an otherwise sorted block. Slot both into
their alphabetical positions — which is also the order pnpm-lock.yaml
already records, so no lockfile change is involved.
@roninjin10
roninjin10 merged commit c563ef5 into smithersai:main Aug 11, 2026
19 checks passed
@roninjin10

Copy link
Copy Markdown
Contributor

Merged — with this, your full stack (#1463#1449#1461#1430) is on main. Polish applied on top of your commits with authorship preserved: rebased onto current main, migrations renumbered to 0039_attempt_effort_column / 0040_add_steers (with convergence tests for the preview ids), steer-finalization durability hardening, and herdr-absent degradation verified empirically — every suite passes with no herdr binary, and every failure path returns a structured error instead of crashing. Two known follow-ups we're tracking as issues rather than blocking the merge: the client pins herdr wire protocol 16 (herdr 0.7.3) while current herdr 0.8.0 speaks protocol 19, so supervision fails closed (HERDR_PROTOCOL_MISMATCH) against a fresh herdr install until the 16→19 port lands; and the packages/testing runtime dependency on smthrs creates a cycle with @smthrs/testing we plan to break via an optional peer dependency. Verified: full root gate set, 11 package suites, and the complete 252-batch CLI suite, all green. Thanks for a remarkably disciplined 273-file PR — the degradation contract made it reviewable.

roninjin10 pushed a commit that referenced this pull request Aug 11, 2026
packages/smithers depends on @smthrs/testing, and #1430 made
@smthrs/testing depend on smthrs (scripts/watch-pack.mjs imports the
facade at runtime), closing a publish-time cycle between two published
packages.

Demote smthrs to an optional peerDependency (workspace:^, so pnpm/bun
publish it as a caret range off the lockstep monorepo version) and keep
the workspace:* devDependency so the package's own tests and the
campaign scripts still resolve it. watch-pack.mjs now loads the facade
lazily and converts ERR_MODULE_NOT_FOUND into an actionable message,
matching the optional-peer guards in packages/{daytona,vercel,aws}.

Refs #1505

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
roninjin10 pushed a commit that referenced this pull request Aug 11, 2026
Closes #1504. The client shipped in #1430 pinned herdr 0.7.3 / protocol 16, so
every herdr path failed closed with HERDR_PROTOCOL_MISMATCH against a current
herdr (0.8.0 / protocol 19) — safe, but inert. Read from the herdr v0.7.3..v0.8.0
source diff, then verified against a real pinned 0.8.0 server.

Wire changes that reach this client:

- `agent.start` was reshaped. Protocol 16 took `{name, argv, tab_id, cwd, env,
  split, focus}` and CREATED a pane running that argv. Protocol 19 takes
  `{name, kind, pane_id, args}` and only attaches one of herdr's known
  interactive agent kinds to an existing shell pane (`unsupported_agent_kind`
  otherwise). Panes are now opened with `tab.create` (whose root pane carries the
  caller's `cwd`/`env`) and driven with `pane.send_input` — text plus Enter in one
  atomic call, which is exactly how herdr itself launches into a pane.
- The registered agent `name` `agent.start` used to set is now constrained to
  `[a-z][a-z0-9_-]{0,31}`, too narrow for `smithers:<runId>:<nodeId>`. Pane
  ownership keys on the REPORTED agent (`pane.report_agent {agent}`) instead,
  which carries that exact identity; `name` is still matched so a pane registered
  by an older herdr is adopted, not duplicated.
- `agent_name_taken` no longer fences the claim (it guards `agent.start` /
  `agent.rename` only), so a same-name race is reconciled against `agent.list`:
  the loser releases its claim and closes the tab IT created.
- `custom_status` was removed from `pane.report_agent`, `pane.report_metadata`,
  `PaneInfo`, `AgentInfo`, and `pane.agent_status_changed`, replaced by the
  `tokens` metadata map. Gate questions and terminal outcomes now ride in a
  `status` token.
- Additive: `workspace.report_metadata`, `agent.prompt`/`send_keys`/`wait`,
  `agent.view.*`, `pane.graphics.*`, `popup.close`, `workspace.move_block`, the
  `workspace.metadata_updated`/`workspace.reordered`/`pane.updated` events, and
  `terminal_title`/`launch_pending`/`interactive_ready`/`state_change_seq`.

Two hazards the shell-mediated launch introduces, both fixed here:

- herdr drops any authority report whose `seq` is `<=` the last one recorded for
  the source, so the identity claim takes the caller's next seq rather than a raw
  `Date.now()` — otherwise it out-ranks and silently swallows every status push
  that follows it.
- A pane runs the operator's INTERACTIVE shell, whose rc files rebuild `PATH`, so
  a PATH-dependent `argv[0]` would resolve against the operator's PATH. The
  caller's PATH rides in as `SMITHERS_HERDR_PATH` and `env` restores it for the
  launched process.

Also fixes an on-demand pane name that never matched the surface's own
convention (`smithers:<run>:node:<node>` vs `smithers:<run>:<node>`), which made
`smithers herdr open` duplicate a live mirror's pane instead of adopting it. It
was unreachable while every herdr path failed closed.

Compatibility posture: single-version pin on 19. The safety contract is
unchanged — mismatches stay inspectable via `herdr status` and blocked before any
mutation, NDJSON stays byte-bounded, tab ownership stays exact-identity, and
same-run workspace creation stays coalesced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants