fix(bin): relaunch secondmates at fleet startup only when work is pending - #13
Merged
Conversation
added 7 commits
August 9, 2026 01:00
…istration Fleet startup previously relaunched every registered dead/missing secondmate unconditionally. The captain decided 2026-08-03 (superseding the 2026-07-31 open-at-the-stop rule) that startup should launch only the primary, and relaunch a secondmate only when its own durable records show pending work (a queued/in-flight backlog item, or an undelivered in-flight child task), since an idle secondmate just burns startup tokens with no work to do.
…ing-work rule The diagnostic handling playbook still framed the sweep as guaranteeing every registered secondmate is live, which now reads as a contradiction: a registered secondmate with no pending work is deliberately left down and stays silent. It also did not cover the two new skip reasons that refuse to guess pending work from an unresolvable home.
secondmate_home_has_pending_work bound its home argument to a local it never read, since both derived paths build off "$1" directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fleet startup must launch the first mate and nothing else - no second mate is ever auto-started just because it is registered or happened to be open when the fleet stopped.
Captain decision 2026-08-03, on startup-token-cost grounds: 'fleet startup should never be anything more than first mate. I will manually use firstmate to call any secondmates, thus reduces startup cost.' Clarified: 'if there is ongoing work that was paused due to a crash or limit hit, those can restart. but if theres no work pending. we dont open an idle secondmate just for it to use startup tokens and perhaps not get used that session.'
THE RULE: pending work is the test. A secondmate with work paused mid-flight (crash, quota limit, killed session) IS relaunched so that work resumes. A secondmate with an empty queue is NOT opened, regardless of registration or of having been open at the stop. The captain opens the rest himself.
IMPORTANT HISTORY - this is the THIRD position on this question, and the reviewer will find superseded quotes. On 2026-07-31 the captain explicitly REJECTED a work-in-flight test and set the rule to 'reopen exactly the mates that were OPEN at the stop; busy-versus-idle is NOT the test', because he wanted idle-but-open mates back. On 2026-08-03 he reversed that on cost grounds. Quotes still sitting in data/captain.md history reflect the superseded 2026-07-31 rule and must NOT be used to argue the new behavior is wrong. Deliberately reintroducing relaunch-when-idle would be the bug.
Pending work is judged ONLY from that secondmate's own durable records, never from whether its pane happened to be alive: a non-blank line under '## Queued' or '## In flight' in that home's data/backlog.md, or any *.meta file left under that home's state/ by a child task it dispatched and never tore down. Those exact section headings are the repo-wide canonical shape (bin/fm-backlog-handoff.sh seeds every secondmate home with them; fm-session-start.sh and fm-fleet-snapshot.sh parse the same strings). This is deliberately a cheap presence test, not the structured tasks-axi backlog schema - it only decides whether a launch is worth the startup tokens.
Deliberate design decisions a reviewer should not flag as mistakes:
Scope covered: bin/fm-bootstrap.sh (the secondmate_home_has_pending_work helper plus the liveness sweep), AGENTS.md sections 3 and 5, the secondmate-provisioning skill's Recovery section (the single owner of the pending-work test, which every other surface cross-references rather than restating, per this repo's one-owner rule), the bootstrap-diagnostics skill's SECONDMATE_LIVENESS entry, docs/architecture.md and docs/configuration.md, and tests.
Also in this branch, deliberately: the branch was rebased from an eight-commit-stale base onto current main, resolving an AGENTS.md conflict where main had grown the bootstrap sweep list from five to six entries. Two small follow-up commits: one aligning the bootstrap-diagnostics playbook, which still told the agent to investigate any non-live registered secondmate (now wrong - an idle mate being down is correct), and one dropping an unused 'local home=$1' in the new helper.
Known and intentionally out of scope: a separate queued backlog item 'secondmate-no-autoload' (a per-mate opt-out list) is made moot by this pending-work test, but is NOT touched here and will be reconciled separately. Also noted but deliberately not addressed: a captain-held item sitting under '## Queued' still counts as pending work and will relaunch its mate.
Verification already run locally, all green: bin/fm-lint.sh exit 0 (shellcheck 0.11.0 pinned), tests/fm-secondmate-liveness.test.sh (including four new pending-work cases that drive real bootstrap runs against a fake tmux and assert no new-window for an idle mate), tests/fm-secondmate-sync.test.sh, and bin/fm-doc-audience-check.sh.
What Changed
bin/fm-bootstrap.shgainssecondmate_home_has_pending_workand rewires the session-start liveness sweep around it: a confirmed-dead husk is still killed, but the respawn now runs only when that secondmate's own durable records show pending work — a non-blank line under## Queuedor## In flightin its homedata/backlog.md, or a leftover*.metaunder its homestate/. A registered-but-idle mate is left down silently (visible only as aBOOTSTRAP_INFOfact underFM_BOOTSTRAP_VERBOSE_FACTS=1), and an unresolvable, unvalidated, or unreadable home/backlog emits aSECONDMATE_LIVENESS:skip instead of guessing.SECONDMATE_LEFT_DOWN_IDShand-off so the sweeps that follow still fast-forward the home and propagate inherited config, but never send into the dead-by-design endpoint: noNUDGE_SECONDMATES:orCONFIG_REREAD:line, no queued reread generation, and any pre-existing nudge marker is retired rather than carried into the reopening session.AGENTS.md(sections 3 and 5),README.md,docs/architecture.md,docs/configuration.md, thesecondmate-provisioningskill's Recovery section (sole owner of the pending-work test) and thebootstrap-diagnosticsSECONDMATE_LIVENESSentry;tests/fm-secondmate-liveness.test.sh(21 cases) andtests/fm-secondmate-sync.test.sh(22 cases) cover the idle-left-down, in-flight-relaunch, refuse-to-guess, and left-down-not-nudged paths and pass in full.Risk Assessment
✅ Low: Every finding from the prior rounds is now resolved with minimal, correctly-placed changes - the marker retirement sits after the id-safety and filename guards so it can only delete the canonical path, it rests on a premise the repo's own owner skill states explicitly (a fresh launch reads its instruction surface from disk), and the three-session path from the finding is covered end to end by a new test - leaving a well-bounded change with no outstanding concerns.
Testing
Ran the two suites the intent names plus the direct bootstrap and documentation suites, then drove a real end-to-end fleet startup against a fake fleet of four dead-but-registered secondmates to show the policy as a captain would experience it: the idle mate is left down with its dead husk still cleaned up and no actionable diagnostic, while the mates holding a queued item, an in-flight item, or leftover child task metadata are each relaunched, and the verbose run surfaces the left-down outcome only as a BOOTSTRAP_INFO fact. This is a shell CLI startup sweep with no rendered UI surface, so the reviewer-visible artifact is the annotated startup transcript rather than a screenshot. Everything relevant to the change is green; the single failure seen is a pre-existing orca assertion in tests/fm-bootstrap.test.sh that collides with this host's GNOME screen reader and reproduces unchanged at the base commit.
Evidence: Fleet startup transcript — idle secondmate left down, pending-work mates relaunched (both verbosity modes)
Evidence: Reproducer: end-to-end fleet startup demo script
Evidence: Key excerpt — what fleet startup actually launched
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 3 issues found → auto-fixed (2) ✅
bin/fm-bootstrap.sh:555- A secondmate deliberately left down for having no pending work keeps its kind=secondmate meta, and the sweeps that run after it treat meta presence as liveness (bin/fm-ff-lib.sh:247 states this explicitly; process_secondmate at bin/fm-ff-lib.sh:390 never probes). Concrete path: secondmate_liveness_sweep (bin/fm-bootstrap.sh:994) leaves idle sm1 down silently; secondmate_sync (:995) fast-forwards its home, gets FF_STATUS=updated with a non-empty FF_INSTR, and calls fm_ff_after_instruction_update -> secondmate_send_nudge, which runs fm-send.sh fm-sm1 against the dead window. The send fails, printing an actionable 'NUDGE_SECONDMATES: secondmate sm1: send failed: ...' line and leaving a persistent state/.secondmate-nudge-pending/sm1.pending marker that secondmate_retry_pending_nudges re-fails every session. The config-inheritance loop at bin/fm-bootstrap.sh:407-455 does the same via fm_config_send_reread_nudge, and because sm1 is absent from SECONDMATE_RESPAWNED_IDS, reread_skip_pending stays 0 so pending reread generations accumulate until 'CONFIG_REREAD: secondmate sm1: send failed: retry instruction queue is full'. Before this change this was unreachable: the liveness sweep respawned every dead mate at :994, so the endpoint was live by :995. The intent states the correctly-left-down-idle case must not produce an actionable diagnostic, and bootstrap-diagnostics tells the agent to fix the endpoint and rerun session start - now unfixable by design. Earliest shared boundary: have the liveness sweep record the ids it deliberately left down (mirroring SECONDMATE_RESPAWNED_IDS) and have the sync and inherit sweeps skip only the send for those ids, still fast-forwarding the home and propagating files.bin/fm-bootstrap.sh:547- The new pending-work gate uses the recorded home= path directly, unlike every other consumer of that field (process_secondmate at bin/fm-ff-lib.sh:397 and secondmate_retry_pending_nudges at bin/fm-bootstrap.sh:360-364 both run validate_secondmate_home, which requires the .fm-secondmate-home marker). A stale home= pointing at the primary checkout passes [ -d ] and makes the state/*.meta glob match the primary's own secondmate metas, so the mate is relaunched every session as 'pending work' - the idle relaunch the policy forbids; a home= pointing at any other existing directory reads as idle and strands a busy mate. Routing the resolution through validate_secondmate_home would turn both into the 'cannot judge pending work' skip this change deliberately added.bin/fm-bootstrap.sh:484- secondmate_home_has_pending_work cannot distinguish awk's 'no items' exit 1 from an awk error (exit 2) on an existing but unreadable or malformed data/backlog.md; both fall through to the state/ glob and then to 'no pending work'. A home with real '## In flight' items but an unreadable backlog and no dispatched children is left down silently, with no SECONDMATE_LIVENESS line - inconsistent with the refuse-to-guess skips this same change added for an unresolvable home. awk's stderr is also unredirected, so the raw error lands in the bootstrap digest.🔧 Fix: stop nudging left-down mates; validate home and backlog reads
1 warning still open:
bin/fm-bootstrap.sh:356- Skipping the retry for a left-down mate preserves its nudge marker, but the home keeps fast-forwarding while the mate is down and the marker's commit is never refreshed, so the reopening session fails the commit guard permanently. Concrete path: session A, sm1 live, instructions advance, home ff's to c1, fm-send.sh fails transiently, marker written with commit=c1. Session B, sm1's endpoint is dead and its backlog empty, so the liveness sweep leaves it down and adds it to SECONDMATE_LEFT_DOWN_IDS; secondmate_retry_pending_nudges skips it at :356 keeping commit=c1, while process_secondmate still fast-forwards the home to c2 and fm_ff_after_instruction_update returns early at :340, so the marker is never rewritten. Session C, after the captain reopens sm1: it reads alive, is no longer left down, the retry runs, head (c2) != commit (c1) hits the guard at :393, and 'NUDGE_SECONDMATES: secondmate sm1: send failed: retry target is not at recorded instruction commit' prints; the marker is only removed on a successful send, so the actionable line recurs every session forever. This is the same unfixable-diagnostic class this commit set out to remove, and bootstrap-diagnostics still instructs the operator to keep the marker and rerun session start. Minimal fix, consistent with the reasoning this same commit already applies to reread_skip_pending: drop the marker when the mate is left down, because the launch that eventually reopens it re-reads its instructions at startup anyway, and correct the comment at :353-355 which currently promises the marker is kept 'for the session that reopens it'.🔧 Fix: retire unsatisfiable nudge markers for left-down secondmates
✅ Re-checked - no issues remain.
tests/fm-bootstrap.test.sh:403- Pre-existing, environment-dependent failure in tests/fm-bootstrap.test.sh:test_orca_backend_gates_orca_tool_only_when_selectedasserts bootstrap reportsMISSING: orca, but this host ships GNOME's screen reader at /usr/bin/orca, which is inside the test's BASE_PATH (/usr/bin:/bin:/usr/sbin:/sbin), so the tool resolves as present and the expected line is never emitted. Reproduced identically from a cleangit archiveof base commit c2e506b, so it predates this branch and is unrelated to the secondmate launch-policy change. Not fixed here because the repair belongs to an unrelated test and would add noise to this diff.bash tests/fm-secondmate-liveness.test.sh— all 21 cases pass, including the six new ones: idle mate left down, in-flight child metadata relaunches, and the no-home / missing-home / unsafe-home / unreadable-backlog refuse-to-guess skipsbash tests/fm-secondmate-sync.test.sh— all 22 cases pass, includingT8g(left-down home still fast-forwards but is never sent into) andT8h(unsatisfiable nudge marker retired rather than carried into the reopening session)Manual end-to-end fleet startup:/tmp/no-mistakes-evidence/01KZKBZ27D9GHDDYP9QP15RQE9/fleet-startup-demo.shbuilds a firstmate home with four registered secondmates (sm-idleempty backlog,sm-queueditem under## Queued,sm-inflightitem under## In flight,sm-crashedempty backlog + a leftoverstate/child1.meta), kills every agent pane to bare shells, then runs the realbin/fm-bootstrap.shagainst a fake tmux carrying a live window inventorySame demo re-run withFM_BOOTSTRAP_VERBOSE_FACTS=1to confirm the silent left-down case surfaces as aBOOTSTRAP_INFOfact and nothing moreDefault-verbosity digest grepped forSECONDMATE_LIVENESS|NUDGE_SECONDMATES|CONFIG_REREAD|BOOTSTRAP_INFO: nudged— zero matches, confirming the correct left-down outcome emits no actionable diagnosticgrep -rn '## In flight|## Queued' bin/fm-backlog-handoff.sh bin/fm-session-start.sh bin/fm-fleet-snapshot.sh— confirms the headingssecondmate_home_has_pending_workparses are byte-identical to the ones seeded at bin/fm-backlog-handoff.sh:318bash tests/fm-bootstrap.test.sh— direct suite for the changed script; one pre-existing orca failure (see findings), all other cases passbash tests/fm-documentation-audiences.test.sh— all 4 cases pass, covering the AGENTS.md / docs/ / skill prose surfaces this branch editsBase-commit control:git archive c2e506b | tar -x -C /tmp/fm-base-check && bash tests/fm-bootstrap.test.sh— reproduces the identical orca failure, proving it predates the branchdocs/configuration.md:316- docs/configuration.md:316 and .agents/skills/secondmate-provisioning/SKILL.md:114-115 still describe bin/fm-config-push.sh as using "live secondmate discovery", but that discovery is the same meta-record-based helper the bootstrap sweep uses, so a mid-session config push would target a secondmate the liveness sweep deliberately left down (whose endpoint is dead by design) and report a send error. Left unchanged deliberately: the mid-session push path is not touched by this branch, and whether it should skip left-down mates is a behavior decision, not a doc fix. Worth reconciling in the follow-up that also resolves the 'secondmate-no-autoload' backlog item.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.