diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8ea6bb4e9..e8a195cda 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -444,6 +444,27 @@ jobs: make install-lint make lint + launchd-drivers: + name: launchd drivers (bash 3.2) + # macOS DELIBERATELY, and it is the whole point of this job: the rig runs bash 3.2.57 and + # GitHub's macOS runners still ship 3.2 as /bin/bash, while ubuntu ships bash 5. A suite + # green on bash 5 proves nothing about the constraint that actually bites here — every + # `declare -A` or ${v,,} that reaches tools/launchd/ breaks the rig and passes on ubuntu. + # No Go, no cache: these are shell + git tests and nothing else. + runs-on: macos-latest + + steps: + - uses: actions/checkout@v7 + + - name: Confirm /bin/bash really is 3.2 (guard against a silent runner upgrade) + run: | + /bin/bash --version | head -1 + /bin/bash -c '[ "${BASH_VERSINFO[0]}" -eq 3 ]' \ + || { echo "::error::/bin/bash is no longer 3.x — this job no longer tests the rig's shell"; exit 1; } + + - name: Run launchd driver tests + run: make test-launchd-drivers + govulncheck: name: govulncheck (vuln gate) runs-on: ubuntu-latest diff --git a/changelogs/v0.18-current.md b/changelogs/v0.18-current.md index cdaaa1700..36e3e6651 100644 --- a/changelogs/v0.18-current.md +++ b/changelogs/v0.18-current.md @@ -545,6 +545,34 @@ ### Mission infrastructure (no user-facing language or CLI change) +- **Launchd drivers now run committed code, and failing to costs you a notification instead + of nothing** ([#558](https://github.com/sunholo-data/ailang/issues/558)). launchd invokes + each driver by absolute path into the shared clone, and *everything* a fire reads hangs off + that one root (`mission-control.sh:40`): the driver, the skill under `.claude/skills/`, and + the charter under `design_docs/`. Nothing keeps that clone current and nothing reported the + drift, so three artefact classes went stale together — measured as `#556`'s retired qwen3.5 + running 24/24 two days after retirement (iter-131), a stale skill (iter-128), a stale charter + (iter-129), and again on 2026-08-12 when `564cc4640`'s lane-degradation fix sat inert on V1, + 12 commits behind, while both sibling missions had it. + + Previous repairs patched individual artefacts, which fixes one third of the bug. New + `tools/launchd/lib/pin-root.sh` pins the **root**: it re-execs the driver out of a worktree + pinned to committed `origin/dev`, moving script, skill and charter together. It re-points + `MISSION_WORKDIR` as well — `mission-motoko.env:8` and `mission-world.env:5` pin it, so + pinning only the script would leave those two running a fresh driver against a stale charter, + a half-fix that reports green. Sprint worktrees are unaffected (created by absolute path). + + A failed pin is **loud, not fatal**: aborting would make network availability a hard + dependency of every fire, so the fire proceeds and posts *"driver ran UNPINNED"* to both human + channels. Silently continuing would have rebuilt the very defect `564cc4640` removed one layer + up. Escape hatch `AILANG_DRIVER_PIN=0`; `MISSION_DRY_RUN=1` reports `pin=( behind)`. + The two emit blocks share one `_mc_notify`, because two near-identical notifiers is how one of + them silently rots. First-ever automated coverage for `tools/launchd` (37 tests, `make + test-launchd-drivers`, bash 3.2.57): both pin arms, both notify arms, the fetch-failure and + driver-missing-from-ref refusals, and a control proving the stale clone really was stale. + Stated plainly: this cannot help until the shared clone receives it once, and that first + reconcile is human (Principle 0). + - **`dev.ailang.mission-recovery` — a probe-stall costs ~20 min of loop time instead of 90.** When every controller probe times out the driver refuses, and with `StartInterval` at 5400s that slot is gone for a full 90 minutes even if the stall diff --git a/design_docs/v1-mission.md b/design_docs/v1-mission.md index 488ab7906..d145be255 100644 --- a/design_docs/v1-mission.md +++ b/design_docs/v1-mission.md @@ -1654,6 +1654,43 @@ frozen; contracts projection live). evaluator when the gemini lane is unavailable). Rationale: the opus bucket dried Thursday this week at ~55% duty cycle; these two move the remaining Anthropic-heavy sub-agent roles off-bucket so the week stretches toward Monday-to-Monday. +- **[BLOCKED-ON-EVIDENCE — do NOT pick before the gate below is measured] m-driver-pin-rollout** + (`#558`; mission infra, not a language change; does not gate v1.0). **Prerequisite already + shipped**: PR `#666` (2026-08-12, attended) adds `tools/launchd/lib/pin-root.sh` and wires it + into `mission-control.sh` ONLY — the driver re-execs from a worktree pinned to committed + `origin/dev`, moving the driver, the skill and the charter together, and posting *"driver ran + UNPINNED"* on both human channels when the pin fails. This item is the rollout to the + remaining **five** entry points: `nightly-eval.sh`, `nightly-lang-eval.sh`, + `mission-recovery.sh`, `os-rotation-filler.sh`, `rig-watchdog.sh`. + + **Why one root, not five patches** (Principle 3): everything a fire reads hangs off `REPO` at + `mission-control.sh:40`, so the driver (`$0`), the skill (`cwd`) and the charter (`cwd`) go + stale *together*. Measured four times — `#556`'s retired qwen3.5 running 24/24 (iter-131), a + stale skill (iter-128), a stale charter (iter-129), and `564cc4640` inert on V1 at 12 commits + behind (2026-08-12) while both sibling missions had it. Two one-time human reconciles, zero + durable fixes, until `#666`. + + **INERT UNTIL RECONCILED, and say so rather than reporting the capability as gained** — the + same trap that deferred `m-planner-codex-lane` at iter-131. `#666` cannot take effect until + the shared clone receives it once, and that is a human branch op (the standing fast-forward + authorisation above does NOT cover it: local dev is 1 ahead, not 0). **Order matters:** merge + `#666` FIRST, reconcile SECOND — reconciling first brings the lane fix but not the pin, and + the clone simply starts drifting again. + + **THE GATE (evidence, not a delay):** ≥3 consecutive V1 fires logging `driver pin: running + committed origin/dev @ ` with a normal iteration completing. Read it from + `/tmp/ailang-mission-control.log`, not from the file's presence on disk — measuring the wrong + copy is the same class of error as the bug. **Known limit of that gate, stated up front:** three + green fires exercise the re-exec, the root move and the fetch, but NOT the failure path, which + production will not produce on demand. That arm is covered only by `make test-launchd-drivers` + (37 assertions, bash 3.2 CI job) — so a passing gate is evidence about the happy path alone and + must not be reported as whole-fix confidence. + + **Not in scope, and not ours to schedule:** `ailang-world` is a **different repo** + (`sunholo-data/ailang-world`) whose driver is a hand-synced *fork* — 513 lines against this + repo's 671, with a differently-shaped fallback site. It can never take this by copy, only by + port. Handed over on the cross-mission channel as `msg_20260812_085746`; World's loop decides + when. Do not "fix" it from here. - **[QUORUM-BLOCKED 2026-08-11 (iter-179) — doc LANDED, PR `#657` → squash `0a84f5377`, Gate 3b GREEN (4/4 required, `checks=20`, zero not-green). **Reclassified P0**: `#616` is a STATIC EFFECT-SOUNDNESS hole, not the DX/error-message item it was queued as — a function with NO effect annotation calling a diff --git a/make/test.mk b/make/test.mk index 90979fc0d..222c8923a 100644 --- a/make/test.mk +++ b/make/test.mk @@ -7,7 +7,7 @@ .PHONY: test-operator-assertions test-regression-guards test-builtin-consistency .PHONY: test-stdlib-canaries test-row-properties test-golden-types test-repl-smoke .PHONY: test-sim-stub test-stdlib-freeze verify-no-shim verify-lowering -.PHONY: test-nightly-classifier +.PHONY: test-nightly-classifier test-launchd-drivers # Core tests. Depends on build so integration tests that shell out to the # ailang binary never see a stale bin/ailang — a stale binary caused phantom @@ -32,6 +32,15 @@ test: build ## Run all Go unit tests (builds bin/ailang first) test-nightly-classifier: ## Run nightly variance-guard contract and replay tests @python3 tools/test_nightly_classify.py -v +# The launchd drivers carried ZERO automated coverage until #558's second recurrence — a large +# part of why two silent-staleness bugs shipped unnoticed. /bin/bash explicitly, not $$SHELL: +# the rig runs 3.2.57, so a suite that only passes under a newer bash proves nothing about it. +test-launchd-drivers: ## Run launchd driver tests (pin-root + degradation notices, bash 3.2) + @/bin/bash tools/launchd/test_pin_root.sh + @/bin/bash tools/launchd/test_driver_notify.sh + @for f in tools/launchd/*.sh tools/launchd/lib/*.sh; do /bin/bash -n "$$f" || exit 1; done + @echo "launchd drivers: tests + bash 3.2 syntax OK" + test-parser: ## Run parser tests only @echo "Testing parser..." @$(GOTEST) ./internal/parser diff --git a/tools/launchd/lib/pin-root.sh b/tools/launchd/lib/pin-root.sh new file mode 100644 index 000000000..950284a9b --- /dev/null +++ b/tools/launchd/lib/pin-root.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# pin-root.sh — make a launchd driver run COMMITTED code, and make failing to do so LOUD. +# +# THE CLASS THIS CLOSES (#558; measured twice — 2026-08-03 and 2026-08-12). +# launchd invokes each driver by absolute path into a shared, mutable clone that nothing +# keeps current, and everything the fire reads hangs off that one root: +# +# REPO="${MISSION_WORKDIR:-$(cd "$(dirname "$0")/../.." && pwd)}"; cd "$REPO" +# +# * the DRIVER itself -> #556 retired qwen3.5 and the nightly ran it 24/24 two days +# later; 564cc4640's lane fix was inert on V1 while both +# sibling missions had it +# * the SKILL (.claude/skills) -> stale skill, iteration 128 +# * the CHARTER (design_docs) -> stale charter, iteration 129 +# +# Three symptoms, one root. Patching any single artefact fixes one third of a bug, so this +# pins the ROOT: re-exec the driver out of a worktree pinned to a committed ref, which moves +# the script, the skill and the charter together in one move. +# +# WHY THE FAILURE PATH IS LOUD RATHER THAN SILENT. If the fetch fails and we quietly carry on +# from the working tree, we have rebuilt the exact defect 564cc4640 removed one layer up — a +# fallback whose only witness is a log nobody reads (Critical Principle 2). So a failed pin is +# NOT fatal (making network availability a hard dependency of every fire trades a rare silent +# staleness for a common loud outage) but it MUST be reported on the human channel by the +# caller. This file sets PIN_STATUS=STALE and returns 1; emitting is the caller's job, because +# only the caller knows its own early-exit points — a fire that never runs must never post. +# +# CONTRACT +# source, never execute. $0 stays the CALLER's path, which is what makes $0-relative +# resolution below refer to the driver rather than to this file. +# +# in $0 the driver's path (implicit) +# AILANG_DRIVER_REF ref to pin to (default origin/dev) +# AILANG_DRIVER_PIN=0 opt out entirely (default on) +# AILANG_DRIVER_FETCH_TIMEOUT bounded fetch, seconds (default 120) +# AILANG_DRIVER_PIN_DIR override worktree path +# MISSION_NAME namespaces the worktree so missions never collide +# out PIN_STATUS pinned | disabled | STALE +# PIN_NOTE one-line human summary, safe to log or post +# PIN_DRIFT how many commits the source clone is behind the ref ("?" if unknown) +# exit re-execs on success and NEVER RETURNS; returns 0 (already pinned / opted out) or +# 1 (STALE — caller continues on the working tree, loudly) +# +# CHICKEN AND EGG, stated rather than hidden: this block lives in the driver it protects, so it +# does nothing until the shared clone receives it once. That first reconcile is human (Principle +# 0 forbids unattended branch ops on a shared dirty tree). Afterwards it is self-maintaining. +# +# Portable to macOS bash 3.2.57 — no associative arrays, no ${v,,}, no GNU timeout. + +PIN_STATUS="unknown" +PIN_NOTE="" +PIN_DRIFT="?" + +# _pin_bounded SECONDS CMD... — hard wall-clock cap; rc = CMD's rc, or 124 on expiry. +# Deliberately duplicates mission-control.sh's _mc_bounded rather than depending on it: this +# file has to be sourceable by drivers that define no such helper, and the rig has no GNU +# timeout (mission-control.sh:37). An unbounded fetch here would hang the fire before the +# driver's own watchdog exists — the failure mode _mc_bounded was written for. +_pin_bounded() { + local secs="$1"; shift + local out_f rc deadline pid + out_f=$(mktemp -t pin_bounded) || { PIN_BOUNDED_OUT=""; return 125; } + ( exec "$@" ) >"$out_f" 2>&1 & + pid=$! + deadline=$(( $(date +%s) + secs )) + while kill -0 "$pid" 2>/dev/null; do + if [ "$(date +%s)" -ge "$deadline" ]; then + kill "$pid" 2>/dev/null; sleep 1; kill -9 "$pid" 2>/dev/null + PIN_BOUNDED_OUT="$(cat "$out_f" 2>/dev/null)"; rm -f "$out_f" + return 124 + fi + sleep 1 + done + wait "$pid"; rc=$? + PIN_BOUNDED_OUT="$(cat "$out_f" 2>/dev/null)"; rm -f "$out_f" + return "$rc" +} + +_pin_stale() { # $1 = reason -> STALE, caller reports and continues on the working tree + PIN_STATUS="STALE" + PIN_NOTE="$1" + return 1 +} + +# pin_root_to_committed_ref "$@" — pass the driver's own args so the re-exec forwards them. +pin_root_to_committed_ref() { + if [ -n "${AILANG_DRIVER_PINNED:-}" ]; then + # we ARE the re-exec'd copy; the values below crossed the exec in the environment + PIN_STATUS="pinned" + PIN_DRIFT="${AILANG_DRIVER_DRIFT:-?}" + PIN_NOTE="running committed ${AILANG_DRIVER_REF:-origin/dev} @ ${AILANG_DRIVER_PINNED} from ${MISSION_WORKDIR:-?} (source clone ${AILANG_DRIVER_SRC:-?} was ${PIN_DRIFT} behind)" + return 0 + fi + + if [ "${AILANG_DRIVER_PIN:-1}" = "0" ]; then + PIN_STATUS="disabled" + PIN_NOTE="pin disabled via AILANG_DRIVER_PIN=0 — running the working tree as-is" + return 0 + fi + + local ref src script wt target short drift fetch_s rc + ref="${AILANG_DRIVER_REF:-origin/dev}" + fetch_s="${AILANG_DRIVER_FETCH_TIMEOUT:-120}" + script=$(basename "$0") + + src=$(cd "$(dirname "$0")/../.." 2>/dev/null && pwd) + if [ -z "$src" ]; then + _pin_stale "cannot resolve the source clone from \$0=$0"; return 1 + fi + if ! git -C "$src" rev-parse --git-dir >/dev/null 2>&1; then + _pin_stale "source clone $src is not a git repository"; return 1 + fi + + _pin_bounded "$fetch_s" git -C "$src" fetch --quiet origin; rc=$? + if [ "$rc" -eq 124 ]; then + _pin_stale "git fetch origin exceeded ${fetch_s}s in $src"; return 1 + elif [ "$rc" -ne 0 ]; then + _pin_stale "git fetch origin failed (rc=$rc) in $src: $(printf '%s' "${PIN_BOUNDED_OUT:-}" | tail -c 200 | tr '\n' ' ')"; return 1 + fi + + target=$(git -C "$src" rev-parse "$ref" 2>/dev/null) + if [ -z "$target" ]; then + _pin_stale "cannot resolve $ref in $src"; return 1 + fi + short=$(git -C "$src" rev-parse --short "$target" 2>/dev/null) + drift=$(git -C "$src" rev-list --count "HEAD..$ref" 2>/dev/null) + [ -n "$drift" ] || drift="?" + PIN_DRIFT="$drift" + + wt="${AILANG_DRIVER_PIN_DIR:-$HOME/.ailang-driver-pin/${MISSION_NAME:-$(basename "$script" .sh)}}" + + # Refresh (or create) the pin worktree. It is a throwaway checkout with no user work in it, + # kept outside every dev clone, so --force can never overwrite in-progress changes. + git -C "$src" worktree prune >/dev/null 2>&1 + if git -C "$src" worktree list --porcelain 2>/dev/null | grep -qx "worktree $wt"; then + if ! git -C "$wt" checkout --quiet --detach --force "$target" 2>/dev/null; then + _pin_stale "pin worktree checkout $short failed at $wt"; return 1 + fi + else + mkdir -p "$(dirname "$wt")" + if ! git -C "$src" worktree add --quiet --detach --force "$wt" "$target" 2>/dev/null; then + _pin_stale "pin worktree add $short failed at $wt"; return 1 + fi + fi + + # Refuse to exec into a driver the ref does not have — a rename on dev would otherwise turn + # a pin into a silent no-run. Control for the whole worktree step in one assertion. + if [ ! -f "$wt/tools/launchd/$script" ]; then + _pin_stale "$ref has no tools/launchd/$script — refusing to re-exec into a missing driver"; return 1 + fi + + # MISSION_WORKDIR moves too, and that is the point: mission-control.sh:40 reads it AHEAD of + # $0-relative resolution, so pinning only the script would leave motoko and World (whose env + # files pin MISSION_WORKDIR — mission-motoko.env:8, mission-world.env:5) running a pinned + # driver against a stale charter and skill. That half-fix reports green, which is worse than + # no fix. Sprint worktrees are unaffected: the skill creates them by absolute path + # (mission-control SKILL.md:1662), not relative to cwd. + AILANG_DRIVER_PINNED="$short" + AILANG_DRIVER_SRC="$src" + AILANG_DRIVER_DRIFT="$drift" + AILANG_DRIVER_REF="$ref" + MISSION_WORKDIR="$wt" + export AILANG_DRIVER_PINNED AILANG_DRIVER_SRC AILANG_DRIVER_DRIFT AILANG_DRIVER_REF MISSION_WORKDIR + + exec /bin/bash "$wt/tools/launchd/$script" "$@" +} diff --git a/tools/launchd/mission-control.sh b/tools/launchd/mission-control.sh index e2bbe180f..9f91ebd83 100755 --- a/tools/launchd/mission-control.sh +++ b/tools/launchd/mission-control.sh @@ -94,6 +94,25 @@ unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN OPENAI_API_KEY log() { echo "[$(date '+%F %H:%M:%S')] $*" | tee -a "$LOG"; } +# _mc_notify TITLE BODY LABEL — report a degradation on BOTH human channels. +# Extracted from the lane-degradation block (564cc4640) when the driver-pin notice (#558) needed +# the identical shape: two near-identical emit blocks is how one of them silently rots. +# NOT fail-closed: aborting on a failed post would make GitHub/controlplane availability a hard +# dependency of every fire. A failed post is LOUD in the driver log instead — the one thing the +# silent-fallback class never was (Critical Principle 2). +_mc_notify() { + local title="$1" body="$2" label="$3" + ailang messages send controlplane "$body" \ + --title "$title" --from "$MSG_FROM" 2>/dev/null \ + || log "WARNING: ${label} notice FAILED to send via ailang messages" + if [ -n "${MISSION_GH_ISSUE:-}" ]; then + gh issue comment "$MISSION_GH_ISSUE" --repo "$MISSION_REPO" --body "$body" >/dev/null 2>&1 \ + || log "WARNING: ${label} notice FAILED to post to issue #${MISSION_GH_ISSUE}" + else + log "WARNING: ${label} notice needed but MISSION_GH_ISSUE is unset — no issue notice possible" + fi +} + # --- stall detection (see the stall watchdog below) ------------------------- # _mc_descendants PID → echoes PID and every descendant PID (one per line). _mc_descendants() { @@ -279,6 +298,43 @@ MISSION_GH_ISSUE="${MISSION_GH_ISSUE:-$(head -1 "$GH_ISSUE_FILE" 2>/dev/null)}" [ -z "${MISSION_GH_ISSUE:-}" ] && [ "$MISSION_NAME" = "v1" ] && MISSION_GH_ISSUE=329 export MISSION_GH_ISSUE +# --- DRIVER PIN (#558) ------------------------------------------------------- +# Re-exec this driver out of a worktree pinned to committed origin/dev, so the script, the +# skill and the charter all come from the same reviewed commit rather than from whatever the +# shared clone happens to hold. See tools/launchd/lib/pin-root.sh for the full rationale. +# +# PLACED HERE, and the position is load-bearing in both directions: +# * BEFORE the model probes below — a re-exec restarts the script from the top, so pinning +# any later would probe every lane twice and bill it twice; +# * BEFORE the pidfile is written (line ~583) — the re-exec'd copy would otherwise read its +# own parent's pid and yield to itself as an overlap, turning every fire into a no-op; +# * AFTER the state block, so a failed pin has LOG, log(), MSG_FROM and MISSION_GH_ISSUE +# available to report with. Reporting a stale driver on a channel that needs the stale +# driver's own config to be resolved is not reporting. +# Sourced from $REPO, which on the first pass is the UNPINNED clone: the stale helper re-execs +# into the pinned driver, which then sources the pinned helper. Two passes by construction. +if [ -f "$REPO/tools/launchd/lib/pin-root.sh" ]; then + . "$REPO/tools/launchd/lib/pin-root.sh" + pin_root_to_committed_ref "$@" +else + # PIN_DRIFT is normally initialised by the helper. It must be set HERE too: this branch is the + # pre-helper clone, i.e. exactly the case it exists to survive, and `set -u` would otherwise + # abort at the DRY RUN line below — the fallback crashing only on the fallback path. + PIN_STATUS="STALE" + PIN_DRIFT="?" + PIN_NOTE="$REPO/tools/launchd/lib/pin-root.sh is absent — this clone predates the driver pin (#558)" +fi +_pin_degraded="" +if [ "$PIN_STATUS" = "STALE" ]; then + # Deliberately NOT fatal: aborting would make network/git availability a hard dependency of + # every fire, trading rare silent staleness for common loud outage. Loud instead of fatal. + log "DRIVER PIN FAILED — this fire runs the WORKING TREE at $(git -C "$REPO" rev-parse --short HEAD 2>/dev/null), not committed code: $PIN_NOTE" + _pin_degraded=" +- driver pin: **FAILED** — \`$PIN_NOTE\`. This fire ran \`$REPO\` at \`$(git -C "$REPO" rev-parse --short HEAD 2>/dev/null)\` ($(git -C "$REPO" rev-list --count HEAD..origin/dev 2>/dev/null || echo '?') behind \`origin/dev\`), so any landed driver/skill/charter fix newer than that commit was NOT in effect." +else + log "driver pin: $PIN_NOTE" +fi + # designer default is the claude-CLI lane (claude:), NOT the bare "fable" alias: the # Agent tool pins only sonnet|opus|haiku (F1, iteration 31), so under an opus-first controller a # bare "fable" would silently fall back to opus. claude:claude-fable-5 = a REAL bounded Fable run. @@ -471,7 +527,7 @@ if [ "${MISSION_DRY_RUN:-0}" = "1" ]; then else _dry_lanes="ok" fi - log "DRY RUN ok: mission=$MISSION_NAME repo-slug=$MISSION_REPO doc=$MISSION_DOC workdir=$REPO pidfile=$PIDFILE prefs=$PREFS timeout=${HARD_TIMEOUT}s | roles: designer=$MISSION_DESIGNER_MODEL planner=$MISSION_PLANNER_MODEL executor=$MISSION_EXECUTOR_MODEL evaluator=$MISSION_EVALUATOR_MODEL | lanes=$_dry_lanes"; exit 0 + log "DRY RUN ok: mission=$MISSION_NAME repo-slug=$MISSION_REPO doc=$MISSION_DOC workdir=$REPO pidfile=$PIDFILE prefs=$PREFS timeout=${HARD_TIMEOUT}s | roles: designer=$MISSION_DESIGNER_MODEL planner=$MISSION_PLANNER_MODEL executor=$MISSION_EXECUTOR_MODEL evaluator=$MISSION_EVALUATOR_MODEL | lanes=$_dry_lanes | pin=$PIN_STATUS($PIN_DRIFT behind)"; exit 0 fi # 4. Select the model (probe doubles as the subscription-auth check: API keys @@ -554,15 +610,23 @@ ${_lane_degraded} Controller: \`${MODEL}\` (${MODEL_WHY}). Effective roles now: designer=\`${MISSION_DESIGNER_MODEL}\` planner=\`${MISSION_PLANNER_MODEL}\` executor=\`${MISSION_EXECUTOR_MODEL}\` evaluator=\`${MISSION_EVALUATOR_MODEL}\`. Driver log: \`${LOG}\`. If this repeats across fires, the lane is down — check the bucket, and check that this mission's plist carries a PATH that reaches the CLI (the World mission lost five iterations to exactly that)." log "LANE DEGRADED this fire:$(printf '%s' "$_lane_degraded" | tr '\n' ' ')" - ailang messages send controlplane "$_deg_body" \ - --title "Mission ${MISSION_NAME}: executor/planner lane degraded" --from "$MSG_FROM" 2>/dev/null \ - || log "WARNING: lane-degradation notice FAILED to send via ailang messages" - if [ -n "${MISSION_GH_ISSUE:-}" ]; then - gh issue comment "$MISSION_GH_ISSUE" --repo "$MISSION_REPO" --body "$_deg_body" >/dev/null 2>&1 \ - || log "WARNING: lane-degradation notice FAILED to post to issue #${MISSION_GH_ISSUE}" - else - log "WARNING: lane degraded but MISSION_GH_ISSUE is unset — no issue notice possible" - fi + _mc_notify "Mission ${MISSION_NAME}: executor/planner lane degraded" "$_deg_body" "lane-degradation" +fi + +# DRIVER-PIN NOTICE — same site and same reasoning as the lane notice above: after every early +# exit, so a fire that does not run cannot post. Emitted only when the pin actually FAILED, i.e. +# only when stale code really did run. The shared clone being behind is not itself reportable — +# once drivers pin, that drift is harmless, and posting it every 90 minutes would train the +# channel to be ignored, which is how the original silent fallback survived twelve commits. +if [ -n "$_pin_degraded" ]; then + _pin_body="**Driver ran UNPINNED on this fire** — recorded before the iteration ran. +${_pin_degraded} + +Mission \`${MISSION_NAME}\`. Driver log: \`${LOG}\`. The fire still ran; only its code provenance is +unknown. Fix: reconcile that clone with \`origin/dev\`, or find why the fetch failed. Until then +every fire silently runs whatever that working tree happens to hold — the class \`#558\` tracks, +measured twice (2026-08-03 \`#556\`, 2026-08-12 \`564cc4640\`)." + _mc_notify "Mission ${MISSION_NAME}: driver ran UNPINNED (code provenance unknown)" "$_pin_body" "driver-pin" fi log "=== mission iteration starting (controller=$MODEL via ${MODEL_WHY}, timeout=${HARD_TIMEOUT}s | bg-wait-ceiling=${CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS}ms | roles: designer=$MISSION_DESIGNER_MODEL planner=$MISSION_PLANNER_MODEL executor=$MISSION_EXECUTOR_MODEL evaluator=$MISSION_EVALUATOR_MODEL) ===" diff --git a/tools/launchd/test_driver_notify.sh b/tools/launchd/test_driver_notify.sh new file mode 100755 index 000000000..836895738 --- /dev/null +++ b/tools/launchd/test_driver_notify.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# Stubbed-channel tests over the REAL emit blocks in mission-control.sh. +# The blocks are awk-extracted from the file, never retyped: a retyped copy tests the copy. +set -uo pipefail + +SP="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +DRV="$REPO_ROOT/tools/launchd/mission-control.sh" +LAB="${TMPDIR:-/tmp}/emitlab.$$"; rm -rf "$LAB"; mkdir -p "$LAB" + +awk '/^_mc_notify\(\) \{/,/^\}/' "$DRV" > "$LAB/notify.sh" +awk '/^if \[ -n "\$_pin_degraded" \]; then/,/^fi$/' "$DRV" > "$LAB/pin_block.sh" +awk '/^if \[ -n "\$_lane_degraded" \]; then/,/^fi$/' "$DRV" > "$LAB/lane_block.sh" + +for f in notify pin_block lane_block; do + if [ ! -s "$LAB/$f.sh" ]; then echo "FATAL: extraction of $f produced nothing"; exit 1; fi +done +echo "extracted: notify=$(wc -l < "$LAB/notify.sh") pin=$(wc -l < "$LAB/pin_block.sh") lane=$(wc -l < "$LAB/lane_block.sh") lines" + +PASS=0; FAIL=0 +ok(){ PASS=$((PASS+1)); echo " PASS: $1"; } +bad(){ FAIL=$((FAIL+1)); echo " FAIL: $1"; echo " trace: $2"; } +check(){ case "$2" in *"$3"*) ok "$1";; *) bad "$1" "$(printf '%s' "$2"|tr '\n' '|')";; esac; } +checkno(){ case "$2" in *"$3"*) bad "$1" "$(printf '%s' "$2"|tr '\n' '|')";; *) ok "$1";; esac; } + +run() { # $1=block $2=degraded-value -> prints trace; env AILANG_RC/GH_RC/ISSUE tweak it + local block="$1" val="$2" + /bin/bash -c ' + set -uo pipefail + TRACE="" + log() { TRACE="$TRACE +LOG:$*"; } + ailang() { TRACE="$TRACE +AILANG:$*"; return ${AILANG_RC:-0}; } + gh() { TRACE="$TRACE +GH:$*"; return ${GH_RC:-0}; } + MISSION_NAME=v1; MISSION_REPO=sunholo-data/ailang; MSG_FROM=mission-control + MISSION_GH_ISSUE="${ISSUE-635}"; LOG=/tmp/x.log; REPO=/tmp/repo + MODEL=claude-opus-5; MODEL_WHY="probe ok" + MISSION_DESIGNER_MODEL=d; MISSION_PLANNER_MODEL=p; MISSION_EXECUTOR_MODEL=e; MISSION_EVALUATOR_MODEL=v + _pin_degraded=""; _lane_degraded="" + . "$1" # _mc_notify + eval "$3=\"$4\"" # set the ledger under test + . "$2" # the block + printf "%s" "$TRACE" + echo " +RC:$?" + ' _ "$LAB/notify.sh" "$LAB/$block.sh" \ + "$( [ "$block" = pin_block ] && echo _pin_degraded || echo _lane_degraded )" "$val" 2>&1 +} + +echo "== pin notice ==" +T=$(run pin_block "- driver pin: FAILED") +check "fires on both channels (ailang)" "$T" "AILANG:messages send controlplane" +check "posts to the bookkeeping issue" "$T" "GH:issue comment 635" +check "titled as UNPINNED" "$T" "driver ran UNPINNED" +check "names the tracking issue" "$T" "#558" + +echo "== pin notice: SILENT when healthy (the control) ==" +T=$(run pin_block "") +checkno "no ailang call" "$T" "AILANG:" +checkno "no gh call" "$T" "GH:" + +echo "== failed post is LOUD, and never aborts ==" +T=$(AILANG_RC=1 run pin_block "- x") +check "warns on send failure" "$T" "LOG:WARNING: driver-pin notice FAILED to send" +check "block still exits 0" "$T" "RC:0" +T=$(GH_RC=1 run pin_block "- x") +check "warns on issue failure" "$T" "LOG:WARNING: driver-pin notice FAILED to post" +check "block still exits 0" "$T" "RC:0" + +echo "== unset issue warns rather than silently skipping ==" +T=$(ISSUE= run pin_block "- x") +check "warns on unset issue" "$T" "MISSION_GH_ISSUE is unset" +checkno "no gh call attempted" "$T" "GH:issue" + +echo "== REGRESSION: lane block still works through the extracted _mc_notify ==" +T=$(run lane_block "- codex lane down") +check "lane fires on ailang" "$T" "AILANG:messages send controlplane" +check "lane posts to the issue" "$T" "GH:issue comment 635" +check "lane keeps its own title" "$T" "executor/planner lane degraded" +check "lane logs its summary" "$T" "LOG:LANE DEGRADED this fire" +T=$(run lane_block "") +checkno "lane SILENT when healthy" "$T" "AILANG:" + +echo "" +echo "==== $PASS passed, $FAIL failed ====" +[ "$FAIL" -eq 0 ] diff --git a/tools/launchd/test_pin_root.sh b/tools/launchd/test_pin_root.sh new file mode 100755 index 000000000..2a78aa082 --- /dev/null +++ b/tools/launchd/test_pin_root.sh @@ -0,0 +1,124 @@ +#!/bin/bash +# End-to-end test for tools/launchd/lib/pin-root.sh under bash 3.2. +# Builds a real origin + a deliberately-stale clone, then asserts on OBSERVED behaviour. +set -uo pipefail + +SP="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +SRC_HELPER="$REPO_ROOT/tools/launchd/lib/pin-root.sh" +T="${TMPDIR:-/tmp}/pinlab.$$" +rm -rf "$T"; mkdir -p "$T" +# Normalise: macOS $TMPDIR carries a trailing slash, so the raw string would not match the +# realpath the driver reports, and the path assertions below would fail on punctuation. +T="$(cd "$T" && pwd)" +trap 'rm -rf "$T"' EXIT + +# Hermetic identity + branch name. A CI runner has no user.name/user.email and may default +# `init.defaultBranch` to anything, so without these the lab commits fail there and the suite +# reports a git-config problem as a pin failure — a test that only passes on the author's box. +export GIT_AUTHOR_NAME=pinlab GIT_AUTHOR_EMAIL=pinlab@invalid +export GIT_COMMITTER_NAME=pinlab GIT_COMMITTER_EMAIL=pinlab@invalid +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " PASS: $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL: $1"; echo " got: $2"; } +check(){ # name haystack needle + case "$2" in *"$3"*) ok "$1";; *) bad "$1" "$(printf '%s' "$2" | tr '\n' '|' | tail -c 300)";; esac +} +checkno(){ case "$2" in *"$3"*) bad "$1" "$(printf '%s' "$2" | tr '\n' '|' | tail -c 300)";; *) ok "$1";; esac; } + +# ---- build origin ---------------------------------------------------------- +git init --quiet --bare "$T/origin.git" +git clone --quiet "$T/origin.git" "$T/seed" 2>/dev/null +cd "$T/seed" +git checkout --quiet -b dev +mkdir -p tools/launchd/lib +cp "$SRC_HELPER" tools/launchd/lib/pin-root.sh +cat > tools/launchd/fake-driver.sh <<'DRIVER' +#!/usr/bin/env bash +set -uo pipefail +REPO="${MISSION_WORKDIR:-$(cd "$(dirname "$0")/../.." && pwd)}" +cd "$REPO" || exit 1 +LOG=/dev/null +log() { echo "[drv] $*"; } +if [ -f "$REPO/tools/launchd/lib/pin-root.sh" ]; then + . "$REPO/tools/launchd/lib/pin-root.sh" + pin_root_to_committed_ref "$@" +else + PIN_STATUS="STALE"; PIN_NOTE="helper absent" +fi +echo "STATUS=$PIN_STATUS" +echo "DRIFT=$PIN_DRIFT" +echo "NOTE=$PIN_NOTE" +echo "REPO=$REPO" +echo "MARKER=$(cat "$REPO/MARKER" 2>/dev/null)" +echo "ARGS=$*" +DRIVER +chmod +x tools/launchd/fake-driver.sh +echo "STALE-CONTENT" > MARKER +git add -A >/dev/null; git commit --quiet -m "base" +git push --quiet origin dev 2>/dev/null + +# clone that will go stale +git clone --quiet --branch dev "$T/origin.git" "$T/clone" 2>/dev/null + +# advance origin ONE commit: the clone is now genuinely behind +echo "FRESH-CONTENT" > MARKER +git commit --quiet -am "advance" +git push --quiet origin dev 2>/dev/null +cd "$T" + +DRV="$T/clone/tools/launchd/fake-driver.sh" +export AILANG_DRIVER_PIN_DIR="$T/pinwt" + +echo "== 1. happy path: stale clone re-execs into committed origin/dev ==" +OUT=$(/bin/bash "$DRV" alpha beta 2>&1) +check "status is pinned" "$OUT" "STATUS=pinned" +check "drift measured as 1" "$OUT" "DRIFT=1" +check "ROOT moved to the pin worktree" "$OUT" "REPO=$T/pinwt" +check "reads FRESH content, not stale" "$OUT" "MARKER=FRESH-CONTENT" +checkno "stale content is NOT read" "$OUT" "MARKER=STALE-CONTENT" +check "args survive the re-exec" "$OUT" "ARGS=alpha beta" + +echo "== 2. control: the clone really was stale (instrument check) ==" +CTL=$(AILANG_DRIVER_PIN=0 /bin/bash "$DRV" 2>&1) +check "opt-out reports disabled" "$CTL" "STATUS=disabled" +check "opt-out reads STALE content" "$CTL" "MARKER=STALE-CONTENT" +check "opt-out root stays in the clone" "$CTL" "REPO=$T/clone" + +echo "== 3. no recursion: second pass returns without re-exec ==" +REC=$(AILANG_DRIVER_PINNED=deadbee AILANG_DRIVER_DRIFT=7 /bin/bash "$DRV" 2>&1) +check "already-pinned short-circuits" "$REC" "STATUS=pinned" +check "carries drift across the exec" "$REC" "DRIFT=7" + +echo "== 4. fetch failure is STALE + loud, never silent-ok ==" +git -C "$T/clone" remote set-url origin "$T/does-not-exist.git" +FF=$(AILANG_DRIVER_FETCH_TIMEOUT=20 /bin/bash "$DRV" 2>&1) +check "status STALE" "$FF" "STATUS=STALE" +check "reason names the fetch" "$FF" "git fetch origin failed" +check "driver still ran (fail-open)" "$FF" "MARKER=STALE-CONTENT" +checkno "never reports pinned" "$FF" "STATUS=pinned" +git -C "$T/clone" remote set-url origin "$T/origin.git" + +echo "== 5. driver absent FROM THE REF => refuse to exec into nothing ==" +# The driver must EXIST locally (so $0 resolves and the guard is what fires) but be absent from +# origin/dev — the shape a rename-on-dev would produce. A test asserting bash's own +# "No such file" would pass without the guard ever running. +cp "$T/clone/tools/launchd/fake-driver.sh" "$T/clone/tools/launchd/local-only.sh" +MD=$(AILANG_DRIVER_PIN_DIR="$T/pinwt2" /bin/bash "$T/clone/tools/launchd/local-only.sh" 2>&1) +check "guard fires, not bash" "$MD" "has no tools/launchd/local-only.sh" +check "status STALE" "$MD" "STATUS=STALE" +checkno "did NOT exec into nothing" "$MD" "No such file" + +echo "== 6. non-repo source => STALE, not a crash ==" +mkdir -p "$T/plain/tools/launchd/lib" +cp "$SRC_HELPER" "$T/plain/tools/launchd/lib/pin-root.sh" +cp "$T/clone/tools/launchd/fake-driver.sh" "$T/plain/tools/launchd/" +NR=$(/bin/bash "$T/plain/tools/launchd/fake-driver.sh" 2>&1) +check "non-repo is STALE" "$NR" "STATUS=STALE" +check "reason names the repo problem" "$NR" "not a git repository" + +echo "" +echo "==== $PASS passed, $FAIL failed ====" +[ "$FAIL" -eq 0 ]