From 1f1c45471f6d367d91893ef23f5a9351d93e83df Mon Sep 17 00:00:00 2001 From: "Voight-Kampff (bot)" <151556158+sunholo-voight-kampff@users.noreply.github.com> Date: Wed, 12 Aug 2026 09:45:27 +0200 Subject: [PATCH 1/3] fix(driver): pin launchd drivers to committed code, and make an unpinned fire LOUD (#558) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit THE DEFECT — one root, three symptoms. launchd invokes each driver by absolute path into the shared clone, and everything a fire reads hangs off that single root (mission-control.sh:40 `REPO="${MISSION_WORKDIR:-$(dirname $0/../..)}"; cd "$REPO"`): the driver, the skill under .claude/skills/, and the charter under design_docs/. Nothing keeps that clone current and nothing reports the drift, so all three go stale together and every previous repair patched one artefact — a third of a bug. Measured four times now: #556's retired qwen3.5 ran 24/24 two days after retirement (iter-131); a stale skill (iter-128); a stale charter (iter-129); and 2026-08-12, 564cc4640's lane-degradation fix inert on V1 at 12 commits behind while ailang-motoko and ailang-world both had it. Two one-time human reconciles, zero durable fixes. THE FIX. tools/launchd/lib/pin-root.sh re-execs the driver out of a worktree pinned to committed origin/dev, which moves script, skill and charter in one step. Three details carry the correctness: * It re-points MISSION_WORKDIR too. mission-motoko.env:8 and mission-world.env:5 pin it, and mission-control.sh:40 reads it AHEAD of $0-relative resolution — so pinning only the script leaves those two on a fresh driver against a stale charter and skill. That half-fix reports green, which is worse than no fix. Sprint worktrees are unaffected: the skill creates them by absolute path (SKILL.md:1662). * A failed pin is LOUD, not fatal. Aborting would make network availability a hard dependency of every fire — trading rare silent staleness for common loud outage. So the fire proceeds and posts "driver ran UNPINNED" on both human channels. Continuing SILENTLY would have rebuilt the exact defect 564cc4640 removed one layer up. * Position is load-bearing: after the state block (so a failure has LOG/MSG_FROM/ MISSION_GH_ISSUE to report with), before the probes (a re-exec restarts the script, so pinning later bills every lane twice), before the pidfile write (the re-exec'd copy would read its parent's pid and yield to itself, no-opping every fire). Both notices now share one _mc_notify, extracted from 564cc4640's block: two near-identical notifiers is how one of them silently rots. TESTED — first automated coverage tools/launchd has ever had (37 assertions, `make test-launchd-drivers`, /bin/bash 3.2.57 explicitly, since the rig has no newer one): - real origin + deliberately-stale clone: re-exec observed, ROOT moved, FRESH content read, args survive; plus a control proving the clone really was stale (AILANG_DRIVER_PIN=0 reads STALE-CONTENT from the clone path). - no recursion on the second pass; drift carried across the exec. - fetch failure -> STALE + reason + fire still runs, never reports pinned. - driver absent FROM THE REF -> our guard fires, not bash's "No such file" (the first version of that test passed for the wrong reason and was rewritten). - notices awk-extracted from the real file, never retyped: fires when degraded, SILENT when healthy, rc=1 on either channel -> WARNING and still exits 0, unset issue -> WARNING not a silent skip. Lane block regression-tested through the extraction. - `set -u` bug caught by inspection and fixed: PIN_DRIFT was initialised only inside the helper, so the helper-absent branch — the pre-helper clone this branch exists to survive — would have aborted at the DRY RUN line. The fallback crashing only on the fallback path. CHICKEN AND EGG, stated rather than hidden: this cannot take effect until the shared clone receives it once, and that first reconcile is human — Principle 0 forbids unattended branch ops on a shared dirty tree. Wired into mission-control.sh only; the other five entry points follow once this has survived live fires. Co-Authored-By: Claude Opus 5 --- changelogs/v0.18-current.md | 28 +++++ make/test.mk | 11 +- tools/launchd/lib/pin-root.sh | 166 ++++++++++++++++++++++++++++ tools/launchd/mission-control.sh | 84 ++++++++++++-- tools/launchd/test_driver_notify.sh | 88 +++++++++++++++ tools/launchd/test_pin_root.sh | 117 ++++++++++++++++++++ 6 files changed, 483 insertions(+), 11 deletions(-) create mode 100644 tools/launchd/lib/pin-root.sh create mode 100755 tools/launchd/test_driver_notify.sh create mode 100755 tools/launchd/test_pin_root.sh diff --git a/changelogs/v0.18-current.md b/changelogs/v0.18-current.md index cdaaa1700..36e3e6651 100644 --- a/changelogs/v0.18-current.md +++ b/changelogs/v0.18-current.md @@ -545,6 +545,34 @@ ### Mission infrastructure (no user-facing language or CLI change) +- **Launchd drivers now run committed code, and failing to costs you a notification instead + of nothing** ([#558](https://github.com/sunholo-data/ailang/issues/558)). launchd invokes + each driver by absolute path into the shared clone, and *everything* a fire reads hangs off + that one root (`mission-control.sh:40`): the driver, the skill under `.claude/skills/`, and + the charter under `design_docs/`. Nothing keeps that clone current and nothing reported the + drift, so three artefact classes went stale together — measured as `#556`'s retired qwen3.5 + running 24/24 two days after retirement (iter-131), a stale skill (iter-128), a stale charter + (iter-129), and again on 2026-08-12 when `564cc4640`'s lane-degradation fix sat inert on V1, + 12 commits behind, while both sibling missions had it. + + Previous repairs patched individual artefacts, which fixes one third of the bug. New + `tools/launchd/lib/pin-root.sh` pins the **root**: it re-execs the driver out of a worktree + pinned to committed `origin/dev`, moving script, skill and charter together. It re-points + `MISSION_WORKDIR` as well — `mission-motoko.env:8` and `mission-world.env:5` pin it, so + pinning only the script would leave those two running a fresh driver against a stale charter, + a half-fix that reports green. Sprint worktrees are unaffected (created by absolute path). + + A failed pin is **loud, not fatal**: aborting would make network availability a hard + dependency of every fire, so the fire proceeds and posts *"driver ran UNPINNED"* to both human + channels. Silently continuing would have rebuilt the very defect `564cc4640` removed one layer + up. Escape hatch `AILANG_DRIVER_PIN=0`; `MISSION_DRY_RUN=1` reports `pin=( behind)`. + The two emit blocks share one `_mc_notify`, because two near-identical notifiers is how one of + them silently rots. First-ever automated coverage for `tools/launchd` (37 tests, `make + test-launchd-drivers`, bash 3.2.57): both pin arms, both notify arms, the fetch-failure and + driver-missing-from-ref refusals, and a control proving the stale clone really was stale. + Stated plainly: this cannot help until the shared clone receives it once, and that first + reconcile is human (Principle 0). + - **`dev.ailang.mission-recovery` — a probe-stall costs ~20 min of loop time instead of 90.** When every controller probe times out the driver refuses, and with `StartInterval` at 5400s that slot is gone for a full 90 minutes even if the stall diff --git a/make/test.mk b/make/test.mk index 90979fc0d..222c8923a 100644 --- a/make/test.mk +++ b/make/test.mk @@ -7,7 +7,7 @@ .PHONY: test-operator-assertions test-regression-guards test-builtin-consistency .PHONY: test-stdlib-canaries test-row-properties test-golden-types test-repl-smoke .PHONY: test-sim-stub test-stdlib-freeze verify-no-shim verify-lowering -.PHONY: test-nightly-classifier +.PHONY: test-nightly-classifier test-launchd-drivers # Core tests. Depends on build so integration tests that shell out to the # ailang binary never see a stale bin/ailang — a stale binary caused phantom @@ -32,6 +32,15 @@ test: build ## Run all Go unit tests (builds bin/ailang first) test-nightly-classifier: ## Run nightly variance-guard contract and replay tests @python3 tools/test_nightly_classify.py -v +# The launchd drivers carried ZERO automated coverage until #558's second recurrence — a large +# part of why two silent-staleness bugs shipped unnoticed. /bin/bash explicitly, not $$SHELL: +# the rig runs 3.2.57, so a suite that only passes under a newer bash proves nothing about it. +test-launchd-drivers: ## Run launchd driver tests (pin-root + degradation notices, bash 3.2) + @/bin/bash tools/launchd/test_pin_root.sh + @/bin/bash tools/launchd/test_driver_notify.sh + @for f in tools/launchd/*.sh tools/launchd/lib/*.sh; do /bin/bash -n "$$f" || exit 1; done + @echo "launchd drivers: tests + bash 3.2 syntax OK" + test-parser: ## Run parser tests only @echo "Testing parser..." @$(GOTEST) ./internal/parser diff --git a/tools/launchd/lib/pin-root.sh b/tools/launchd/lib/pin-root.sh new file mode 100644 index 000000000..950284a9b --- /dev/null +++ b/tools/launchd/lib/pin-root.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# pin-root.sh — make a launchd driver run COMMITTED code, and make failing to do so LOUD. +# +# THE CLASS THIS CLOSES (#558; measured twice — 2026-08-03 and 2026-08-12). +# launchd invokes each driver by absolute path into a shared, mutable clone that nothing +# keeps current, and everything the fire reads hangs off that one root: +# +# REPO="${MISSION_WORKDIR:-$(cd "$(dirname "$0")/../.." && pwd)}"; cd "$REPO" +# +# * the DRIVER itself -> #556 retired qwen3.5 and the nightly ran it 24/24 two days +# later; 564cc4640's lane fix was inert on V1 while both +# sibling missions had it +# * the SKILL (.claude/skills) -> stale skill, iteration 128 +# * the CHARTER (design_docs) -> stale charter, iteration 129 +# +# Three symptoms, one root. Patching any single artefact fixes one third of a bug, so this +# pins the ROOT: re-exec the driver out of a worktree pinned to a committed ref, which moves +# the script, the skill and the charter together in one move. +# +# WHY THE FAILURE PATH IS LOUD RATHER THAN SILENT. If the fetch fails and we quietly carry on +# from the working tree, we have rebuilt the exact defect 564cc4640 removed one layer up — a +# fallback whose only witness is a log nobody reads (Critical Principle 2). So a failed pin is +# NOT fatal (making network availability a hard dependency of every fire trades a rare silent +# staleness for a common loud outage) but it MUST be reported on the human channel by the +# caller. This file sets PIN_STATUS=STALE and returns 1; emitting is the caller's job, because +# only the caller knows its own early-exit points — a fire that never runs must never post. +# +# CONTRACT +# source, never execute. $0 stays the CALLER's path, which is what makes $0-relative +# resolution below refer to the driver rather than to this file. +# +# in $0 the driver's path (implicit) +# AILANG_DRIVER_REF ref to pin to (default origin/dev) +# AILANG_DRIVER_PIN=0 opt out entirely (default on) +# AILANG_DRIVER_FETCH_TIMEOUT bounded fetch, seconds (default 120) +# AILANG_DRIVER_PIN_DIR override worktree path +# MISSION_NAME namespaces the worktree so missions never collide +# out PIN_STATUS pinned | disabled | STALE +# PIN_NOTE one-line human summary, safe to log or post +# PIN_DRIFT how many commits the source clone is behind the ref ("?" if unknown) +# exit re-execs on success and NEVER RETURNS; returns 0 (already pinned / opted out) or +# 1 (STALE — caller continues on the working tree, loudly) +# +# CHICKEN AND EGG, stated rather than hidden: this block lives in the driver it protects, so it +# does nothing until the shared clone receives it once. That first reconcile is human (Principle +# 0 forbids unattended branch ops on a shared dirty tree). Afterwards it is self-maintaining. +# +# Portable to macOS bash 3.2.57 — no associative arrays, no ${v,,}, no GNU timeout. + +PIN_STATUS="unknown" +PIN_NOTE="" +PIN_DRIFT="?" + +# _pin_bounded SECONDS CMD... — hard wall-clock cap; rc = CMD's rc, or 124 on expiry. +# Deliberately duplicates mission-control.sh's _mc_bounded rather than depending on it: this +# file has to be sourceable by drivers that define no such helper, and the rig has no GNU +# timeout (mission-control.sh:37). An unbounded fetch here would hang the fire before the +# driver's own watchdog exists — the failure mode _mc_bounded was written for. +_pin_bounded() { + local secs="$1"; shift + local out_f rc deadline pid + out_f=$(mktemp -t pin_bounded) || { PIN_BOUNDED_OUT=""; return 125; } + ( exec "$@" ) >"$out_f" 2>&1 & + pid=$! + deadline=$(( $(date +%s) + secs )) + while kill -0 "$pid" 2>/dev/null; do + if [ "$(date +%s)" -ge "$deadline" ]; then + kill "$pid" 2>/dev/null; sleep 1; kill -9 "$pid" 2>/dev/null + PIN_BOUNDED_OUT="$(cat "$out_f" 2>/dev/null)"; rm -f "$out_f" + return 124 + fi + sleep 1 + done + wait "$pid"; rc=$? + PIN_BOUNDED_OUT="$(cat "$out_f" 2>/dev/null)"; rm -f "$out_f" + return "$rc" +} + +_pin_stale() { # $1 = reason -> STALE, caller reports and continues on the working tree + PIN_STATUS="STALE" + PIN_NOTE="$1" + return 1 +} + +# pin_root_to_committed_ref "$@" — pass the driver's own args so the re-exec forwards them. +pin_root_to_committed_ref() { + if [ -n "${AILANG_DRIVER_PINNED:-}" ]; then + # we ARE the re-exec'd copy; the values below crossed the exec in the environment + PIN_STATUS="pinned" + PIN_DRIFT="${AILANG_DRIVER_DRIFT:-?}" + PIN_NOTE="running committed ${AILANG_DRIVER_REF:-origin/dev} @ ${AILANG_DRIVER_PINNED} from ${MISSION_WORKDIR:-?} (source clone ${AILANG_DRIVER_SRC:-?} was ${PIN_DRIFT} behind)" + return 0 + fi + + if [ "${AILANG_DRIVER_PIN:-1}" = "0" ]; then + PIN_STATUS="disabled" + PIN_NOTE="pin disabled via AILANG_DRIVER_PIN=0 — running the working tree as-is" + return 0 + fi + + local ref src script wt target short drift fetch_s rc + ref="${AILANG_DRIVER_REF:-origin/dev}" + fetch_s="${AILANG_DRIVER_FETCH_TIMEOUT:-120}" + script=$(basename "$0") + + src=$(cd "$(dirname "$0")/../.." 2>/dev/null && pwd) + if [ -z "$src" ]; then + _pin_stale "cannot resolve the source clone from \$0=$0"; return 1 + fi + if ! git -C "$src" rev-parse --git-dir >/dev/null 2>&1; then + _pin_stale "source clone $src is not a git repository"; return 1 + fi + + _pin_bounded "$fetch_s" git -C "$src" fetch --quiet origin; rc=$? + if [ "$rc" -eq 124 ]; then + _pin_stale "git fetch origin exceeded ${fetch_s}s in $src"; return 1 + elif [ "$rc" -ne 0 ]; then + _pin_stale "git fetch origin failed (rc=$rc) in $src: $(printf '%s' "${PIN_BOUNDED_OUT:-}" | tail -c 200 | tr '\n' ' ')"; return 1 + fi + + target=$(git -C "$src" rev-parse "$ref" 2>/dev/null) + if [ -z "$target" ]; then + _pin_stale "cannot resolve $ref in $src"; return 1 + fi + short=$(git -C "$src" rev-parse --short "$target" 2>/dev/null) + drift=$(git -C "$src" rev-list --count "HEAD..$ref" 2>/dev/null) + [ -n "$drift" ] || drift="?" + PIN_DRIFT="$drift" + + wt="${AILANG_DRIVER_PIN_DIR:-$HOME/.ailang-driver-pin/${MISSION_NAME:-$(basename "$script" .sh)}}" + + # Refresh (or create) the pin worktree. It is a throwaway checkout with no user work in it, + # kept outside every dev clone, so --force can never overwrite in-progress changes. + git -C "$src" worktree prune >/dev/null 2>&1 + if git -C "$src" worktree list --porcelain 2>/dev/null | grep -qx "worktree $wt"; then + if ! git -C "$wt" checkout --quiet --detach --force "$target" 2>/dev/null; then + _pin_stale "pin worktree checkout $short failed at $wt"; return 1 + fi + else + mkdir -p "$(dirname "$wt")" + if ! git -C "$src" worktree add --quiet --detach --force "$wt" "$target" 2>/dev/null; then + _pin_stale "pin worktree add $short failed at $wt"; return 1 + fi + fi + + # Refuse to exec into a driver the ref does not have — a rename on dev would otherwise turn + # a pin into a silent no-run. Control for the whole worktree step in one assertion. + if [ ! -f "$wt/tools/launchd/$script" ]; then + _pin_stale "$ref has no tools/launchd/$script — refusing to re-exec into a missing driver"; return 1 + fi + + # MISSION_WORKDIR moves too, and that is the point: mission-control.sh:40 reads it AHEAD of + # $0-relative resolution, so pinning only the script would leave motoko and World (whose env + # files pin MISSION_WORKDIR — mission-motoko.env:8, mission-world.env:5) running a pinned + # driver against a stale charter and skill. That half-fix reports green, which is worse than + # no fix. Sprint worktrees are unaffected: the skill creates them by absolute path + # (mission-control SKILL.md:1662), not relative to cwd. + AILANG_DRIVER_PINNED="$short" + AILANG_DRIVER_SRC="$src" + AILANG_DRIVER_DRIFT="$drift" + AILANG_DRIVER_REF="$ref" + MISSION_WORKDIR="$wt" + export AILANG_DRIVER_PINNED AILANG_DRIVER_SRC AILANG_DRIVER_DRIFT AILANG_DRIVER_REF MISSION_WORKDIR + + exec /bin/bash "$wt/tools/launchd/$script" "$@" +} diff --git a/tools/launchd/mission-control.sh b/tools/launchd/mission-control.sh index e2bbe180f..9f91ebd83 100755 --- a/tools/launchd/mission-control.sh +++ b/tools/launchd/mission-control.sh @@ -94,6 +94,25 @@ unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN OPENAI_API_KEY log() { echo "[$(date '+%F %H:%M:%S')] $*" | tee -a "$LOG"; } +# _mc_notify TITLE BODY LABEL — report a degradation on BOTH human channels. +# Extracted from the lane-degradation block (564cc4640) when the driver-pin notice (#558) needed +# the identical shape: two near-identical emit blocks is how one of them silently rots. +# NOT fail-closed: aborting on a failed post would make GitHub/controlplane availability a hard +# dependency of every fire. A failed post is LOUD in the driver log instead — the one thing the +# silent-fallback class never was (Critical Principle 2). +_mc_notify() { + local title="$1" body="$2" label="$3" + ailang messages send controlplane "$body" \ + --title "$title" --from "$MSG_FROM" 2>/dev/null \ + || log "WARNING: ${label} notice FAILED to send via ailang messages" + if [ -n "${MISSION_GH_ISSUE:-}" ]; then + gh issue comment "$MISSION_GH_ISSUE" --repo "$MISSION_REPO" --body "$body" >/dev/null 2>&1 \ + || log "WARNING: ${label} notice FAILED to post to issue #${MISSION_GH_ISSUE}" + else + log "WARNING: ${label} notice needed but MISSION_GH_ISSUE is unset — no issue notice possible" + fi +} + # --- stall detection (see the stall watchdog below) ------------------------- # _mc_descendants PID → echoes PID and every descendant PID (one per line). _mc_descendants() { @@ -279,6 +298,43 @@ MISSION_GH_ISSUE="${MISSION_GH_ISSUE:-$(head -1 "$GH_ISSUE_FILE" 2>/dev/null)}" [ -z "${MISSION_GH_ISSUE:-}" ] && [ "$MISSION_NAME" = "v1" ] && MISSION_GH_ISSUE=329 export MISSION_GH_ISSUE +# --- DRIVER PIN (#558) ------------------------------------------------------- +# Re-exec this driver out of a worktree pinned to committed origin/dev, so the script, the +# skill and the charter all come from the same reviewed commit rather than from whatever the +# shared clone happens to hold. See tools/launchd/lib/pin-root.sh for the full rationale. +# +# PLACED HERE, and the position is load-bearing in both directions: +# * BEFORE the model probes below — a re-exec restarts the script from the top, so pinning +# any later would probe every lane twice and bill it twice; +# * BEFORE the pidfile is written (line ~583) — the re-exec'd copy would otherwise read its +# own parent's pid and yield to itself as an overlap, turning every fire into a no-op; +# * AFTER the state block, so a failed pin has LOG, log(), MSG_FROM and MISSION_GH_ISSUE +# available to report with. Reporting a stale driver on a channel that needs the stale +# driver's own config to be resolved is not reporting. +# Sourced from $REPO, which on the first pass is the UNPINNED clone: the stale helper re-execs +# into the pinned driver, which then sources the pinned helper. Two passes by construction. +if [ -f "$REPO/tools/launchd/lib/pin-root.sh" ]; then + . "$REPO/tools/launchd/lib/pin-root.sh" + pin_root_to_committed_ref "$@" +else + # PIN_DRIFT is normally initialised by the helper. It must be set HERE too: this branch is the + # pre-helper clone, i.e. exactly the case it exists to survive, and `set -u` would otherwise + # abort at the DRY RUN line below — the fallback crashing only on the fallback path. + PIN_STATUS="STALE" + PIN_DRIFT="?" + PIN_NOTE="$REPO/tools/launchd/lib/pin-root.sh is absent — this clone predates the driver pin (#558)" +fi +_pin_degraded="" +if [ "$PIN_STATUS" = "STALE" ]; then + # Deliberately NOT fatal: aborting would make network/git availability a hard dependency of + # every fire, trading rare silent staleness for common loud outage. Loud instead of fatal. + log "DRIVER PIN FAILED — this fire runs the WORKING TREE at $(git -C "$REPO" rev-parse --short HEAD 2>/dev/null), not committed code: $PIN_NOTE" + _pin_degraded=" +- driver pin: **FAILED** — \`$PIN_NOTE\`. This fire ran \`$REPO\` at \`$(git -C "$REPO" rev-parse --short HEAD 2>/dev/null)\` ($(git -C "$REPO" rev-list --count HEAD..origin/dev 2>/dev/null || echo '?') behind \`origin/dev\`), so any landed driver/skill/charter fix newer than that commit was NOT in effect." +else + log "driver pin: $PIN_NOTE" +fi + # designer default is the claude-CLI lane (claude:), NOT the bare "fable" alias: the # Agent tool pins only sonnet|opus|haiku (F1, iteration 31), so under an opus-first controller a # bare "fable" would silently fall back to opus. claude:claude-fable-5 = a REAL bounded Fable run. @@ -471,7 +527,7 @@ if [ "${MISSION_DRY_RUN:-0}" = "1" ]; then else _dry_lanes="ok" fi - log "DRY RUN ok: mission=$MISSION_NAME repo-slug=$MISSION_REPO doc=$MISSION_DOC workdir=$REPO pidfile=$PIDFILE prefs=$PREFS timeout=${HARD_TIMEOUT}s | roles: designer=$MISSION_DESIGNER_MODEL planner=$MISSION_PLANNER_MODEL executor=$MISSION_EXECUTOR_MODEL evaluator=$MISSION_EVALUATOR_MODEL | lanes=$_dry_lanes"; exit 0 + log "DRY RUN ok: mission=$MISSION_NAME repo-slug=$MISSION_REPO doc=$MISSION_DOC workdir=$REPO pidfile=$PIDFILE prefs=$PREFS timeout=${HARD_TIMEOUT}s | roles: designer=$MISSION_DESIGNER_MODEL planner=$MISSION_PLANNER_MODEL executor=$MISSION_EXECUTOR_MODEL evaluator=$MISSION_EVALUATOR_MODEL | lanes=$_dry_lanes | pin=$PIN_STATUS($PIN_DRIFT behind)"; exit 0 fi # 4. Select the model (probe doubles as the subscription-auth check: API keys @@ -554,15 +610,23 @@ ${_lane_degraded} Controller: \`${MODEL}\` (${MODEL_WHY}). Effective roles now: designer=\`${MISSION_DESIGNER_MODEL}\` planner=\`${MISSION_PLANNER_MODEL}\` executor=\`${MISSION_EXECUTOR_MODEL}\` evaluator=\`${MISSION_EVALUATOR_MODEL}\`. Driver log: \`${LOG}\`. If this repeats across fires, the lane is down — check the bucket, and check that this mission's plist carries a PATH that reaches the CLI (the World mission lost five iterations to exactly that)." log "LANE DEGRADED this fire:$(printf '%s' "$_lane_degraded" | tr '\n' ' ')" - ailang messages send controlplane "$_deg_body" \ - --title "Mission ${MISSION_NAME}: executor/planner lane degraded" --from "$MSG_FROM" 2>/dev/null \ - || log "WARNING: lane-degradation notice FAILED to send via ailang messages" - if [ -n "${MISSION_GH_ISSUE:-}" ]; then - gh issue comment "$MISSION_GH_ISSUE" --repo "$MISSION_REPO" --body "$_deg_body" >/dev/null 2>&1 \ - || log "WARNING: lane-degradation notice FAILED to post to issue #${MISSION_GH_ISSUE}" - else - log "WARNING: lane degraded but MISSION_GH_ISSUE is unset — no issue notice possible" - fi + _mc_notify "Mission ${MISSION_NAME}: executor/planner lane degraded" "$_deg_body" "lane-degradation" +fi + +# DRIVER-PIN NOTICE — same site and same reasoning as the lane notice above: after every early +# exit, so a fire that does not run cannot post. Emitted only when the pin actually FAILED, i.e. +# only when stale code really did run. The shared clone being behind is not itself reportable — +# once drivers pin, that drift is harmless, and posting it every 90 minutes would train the +# channel to be ignored, which is how the original silent fallback survived twelve commits. +if [ -n "$_pin_degraded" ]; then + _pin_body="**Driver ran UNPINNED on this fire** — recorded before the iteration ran. +${_pin_degraded} + +Mission \`${MISSION_NAME}\`. Driver log: \`${LOG}\`. The fire still ran; only its code provenance is +unknown. Fix: reconcile that clone with \`origin/dev\`, or find why the fetch failed. Until then +every fire silently runs whatever that working tree happens to hold — the class \`#558\` tracks, +measured twice (2026-08-03 \`#556\`, 2026-08-12 \`564cc4640\`)." + _mc_notify "Mission ${MISSION_NAME}: driver ran UNPINNED (code provenance unknown)" "$_pin_body" "driver-pin" fi log "=== mission iteration starting (controller=$MODEL via ${MODEL_WHY}, timeout=${HARD_TIMEOUT}s | bg-wait-ceiling=${CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS}ms | roles: designer=$MISSION_DESIGNER_MODEL planner=$MISSION_PLANNER_MODEL executor=$MISSION_EXECUTOR_MODEL evaluator=$MISSION_EVALUATOR_MODEL) ===" diff --git a/tools/launchd/test_driver_notify.sh b/tools/launchd/test_driver_notify.sh new file mode 100755 index 000000000..836895738 --- /dev/null +++ b/tools/launchd/test_driver_notify.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# Stubbed-channel tests over the REAL emit blocks in mission-control.sh. +# The blocks are awk-extracted from the file, never retyped: a retyped copy tests the copy. +set -uo pipefail + +SP="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +DRV="$REPO_ROOT/tools/launchd/mission-control.sh" +LAB="${TMPDIR:-/tmp}/emitlab.$$"; rm -rf "$LAB"; mkdir -p "$LAB" + +awk '/^_mc_notify\(\) \{/,/^\}/' "$DRV" > "$LAB/notify.sh" +awk '/^if \[ -n "\$_pin_degraded" \]; then/,/^fi$/' "$DRV" > "$LAB/pin_block.sh" +awk '/^if \[ -n "\$_lane_degraded" \]; then/,/^fi$/' "$DRV" > "$LAB/lane_block.sh" + +for f in notify pin_block lane_block; do + if [ ! -s "$LAB/$f.sh" ]; then echo "FATAL: extraction of $f produced nothing"; exit 1; fi +done +echo "extracted: notify=$(wc -l < "$LAB/notify.sh") pin=$(wc -l < "$LAB/pin_block.sh") lane=$(wc -l < "$LAB/lane_block.sh") lines" + +PASS=0; FAIL=0 +ok(){ PASS=$((PASS+1)); echo " PASS: $1"; } +bad(){ FAIL=$((FAIL+1)); echo " FAIL: $1"; echo " trace: $2"; } +check(){ case "$2" in *"$3"*) ok "$1";; *) bad "$1" "$(printf '%s' "$2"|tr '\n' '|')";; esac; } +checkno(){ case "$2" in *"$3"*) bad "$1" "$(printf '%s' "$2"|tr '\n' '|')";; *) ok "$1";; esac; } + +run() { # $1=block $2=degraded-value -> prints trace; env AILANG_RC/GH_RC/ISSUE tweak it + local block="$1" val="$2" + /bin/bash -c ' + set -uo pipefail + TRACE="" + log() { TRACE="$TRACE +LOG:$*"; } + ailang() { TRACE="$TRACE +AILANG:$*"; return ${AILANG_RC:-0}; } + gh() { TRACE="$TRACE +GH:$*"; return ${GH_RC:-0}; } + MISSION_NAME=v1; MISSION_REPO=sunholo-data/ailang; MSG_FROM=mission-control + MISSION_GH_ISSUE="${ISSUE-635}"; LOG=/tmp/x.log; REPO=/tmp/repo + MODEL=claude-opus-5; MODEL_WHY="probe ok" + MISSION_DESIGNER_MODEL=d; MISSION_PLANNER_MODEL=p; MISSION_EXECUTOR_MODEL=e; MISSION_EVALUATOR_MODEL=v + _pin_degraded=""; _lane_degraded="" + . "$1" # _mc_notify + eval "$3=\"$4\"" # set the ledger under test + . "$2" # the block + printf "%s" "$TRACE" + echo " +RC:$?" + ' _ "$LAB/notify.sh" "$LAB/$block.sh" \ + "$( [ "$block" = pin_block ] && echo _pin_degraded || echo _lane_degraded )" "$val" 2>&1 +} + +echo "== pin notice ==" +T=$(run pin_block "- driver pin: FAILED") +check "fires on both channels (ailang)" "$T" "AILANG:messages send controlplane" +check "posts to the bookkeeping issue" "$T" "GH:issue comment 635" +check "titled as UNPINNED" "$T" "driver ran UNPINNED" +check "names the tracking issue" "$T" "#558" + +echo "== pin notice: SILENT when healthy (the control) ==" +T=$(run pin_block "") +checkno "no ailang call" "$T" "AILANG:" +checkno "no gh call" "$T" "GH:" + +echo "== failed post is LOUD, and never aborts ==" +T=$(AILANG_RC=1 run pin_block "- x") +check "warns on send failure" "$T" "LOG:WARNING: driver-pin notice FAILED to send" +check "block still exits 0" "$T" "RC:0" +T=$(GH_RC=1 run pin_block "- x") +check "warns on issue failure" "$T" "LOG:WARNING: driver-pin notice FAILED to post" +check "block still exits 0" "$T" "RC:0" + +echo "== unset issue warns rather than silently skipping ==" +T=$(ISSUE= run pin_block "- x") +check "warns on unset issue" "$T" "MISSION_GH_ISSUE is unset" +checkno "no gh call attempted" "$T" "GH:issue" + +echo "== REGRESSION: lane block still works through the extracted _mc_notify ==" +T=$(run lane_block "- codex lane down") +check "lane fires on ailang" "$T" "AILANG:messages send controlplane" +check "lane posts to the issue" "$T" "GH:issue comment 635" +check "lane keeps its own title" "$T" "executor/planner lane degraded" +check "lane logs its summary" "$T" "LOG:LANE DEGRADED this fire" +T=$(run lane_block "") +checkno "lane SILENT when healthy" "$T" "AILANG:" + +echo "" +echo "==== $PASS passed, $FAIL failed ====" +[ "$FAIL" -eq 0 ] diff --git a/tools/launchd/test_pin_root.sh b/tools/launchd/test_pin_root.sh new file mode 100755 index 000000000..24b4442da --- /dev/null +++ b/tools/launchd/test_pin_root.sh @@ -0,0 +1,117 @@ +#!/bin/bash +# End-to-end test for tools/launchd/lib/pin-root.sh under bash 3.2. +# Builds a real origin + a deliberately-stale clone, then asserts on OBSERVED behaviour. +set -uo pipefail + +SP="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +SRC_HELPER="$REPO_ROOT/tools/launchd/lib/pin-root.sh" +T="${TMPDIR:-/tmp}/pinlab.$$" +rm -rf "$T"; mkdir -p "$T" +# Normalise: macOS $TMPDIR carries a trailing slash, so the raw string would not match the +# realpath the driver reports, and the path assertions below would fail on punctuation. +T="$(cd "$T" && pwd)" +trap 'rm -rf "$T"' EXIT + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " PASS: $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL: $1"; echo " got: $2"; } +check(){ # name haystack needle + case "$2" in *"$3"*) ok "$1";; *) bad "$1" "$(printf '%s' "$2" | tr '\n' '|' | tail -c 300)";; esac +} +checkno(){ case "$2" in *"$3"*) bad "$1" "$(printf '%s' "$2" | tr '\n' '|' | tail -c 300)";; *) ok "$1";; esac; } + +# ---- build origin ---------------------------------------------------------- +git init --quiet --bare "$T/origin.git" +git clone --quiet "$T/origin.git" "$T/seed" 2>/dev/null +cd "$T/seed" +git checkout --quiet -b dev +mkdir -p tools/launchd/lib +cp "$SRC_HELPER" tools/launchd/lib/pin-root.sh +cat > tools/launchd/fake-driver.sh <<'DRIVER' +#!/usr/bin/env bash +set -uo pipefail +REPO="${MISSION_WORKDIR:-$(cd "$(dirname "$0")/../.." && pwd)}" +cd "$REPO" || exit 1 +LOG=/dev/null +log() { echo "[drv] $*"; } +if [ -f "$REPO/tools/launchd/lib/pin-root.sh" ]; then + . "$REPO/tools/launchd/lib/pin-root.sh" + pin_root_to_committed_ref "$@" +else + PIN_STATUS="STALE"; PIN_NOTE="helper absent" +fi +echo "STATUS=$PIN_STATUS" +echo "DRIFT=$PIN_DRIFT" +echo "NOTE=$PIN_NOTE" +echo "REPO=$REPO" +echo "MARKER=$(cat "$REPO/MARKER" 2>/dev/null)" +echo "ARGS=$*" +DRIVER +chmod +x tools/launchd/fake-driver.sh +echo "STALE-CONTENT" > MARKER +git add -A >/dev/null; git commit --quiet -m "base" +git push --quiet origin dev 2>/dev/null + +# clone that will go stale +git clone --quiet --branch dev "$T/origin.git" "$T/clone" 2>/dev/null + +# advance origin ONE commit: the clone is now genuinely behind +echo "FRESH-CONTENT" > MARKER +git commit --quiet -am "advance" +git push --quiet origin dev 2>/dev/null +cd "$T" + +DRV="$T/clone/tools/launchd/fake-driver.sh" +export AILANG_DRIVER_PIN_DIR="$T/pinwt" + +echo "== 1. happy path: stale clone re-execs into committed origin/dev ==" +OUT=$(/bin/bash "$DRV" alpha beta 2>&1) +check "status is pinned" "$OUT" "STATUS=pinned" +check "drift measured as 1" "$OUT" "DRIFT=1" +check "ROOT moved to the pin worktree" "$OUT" "REPO=$T/pinwt" +check "reads FRESH content, not stale" "$OUT" "MARKER=FRESH-CONTENT" +checkno "stale content is NOT read" "$OUT" "MARKER=STALE-CONTENT" +check "args survive the re-exec" "$OUT" "ARGS=alpha beta" + +echo "== 2. control: the clone really was stale (instrument check) ==" +CTL=$(AILANG_DRIVER_PIN=0 /bin/bash "$DRV" 2>&1) +check "opt-out reports disabled" "$CTL" "STATUS=disabled" +check "opt-out reads STALE content" "$CTL" "MARKER=STALE-CONTENT" +check "opt-out root stays in the clone" "$CTL" "REPO=$T/clone" + +echo "== 3. no recursion: second pass returns without re-exec ==" +REC=$(AILANG_DRIVER_PINNED=deadbee AILANG_DRIVER_DRIFT=7 /bin/bash "$DRV" 2>&1) +check "already-pinned short-circuits" "$REC" "STATUS=pinned" +check "carries drift across the exec" "$REC" "DRIFT=7" + +echo "== 4. fetch failure is STALE + loud, never silent-ok ==" +git -C "$T/clone" remote set-url origin "$T/does-not-exist.git" +FF=$(AILANG_DRIVER_FETCH_TIMEOUT=20 /bin/bash "$DRV" 2>&1) +check "status STALE" "$FF" "STATUS=STALE" +check "reason names the fetch" "$FF" "git fetch origin failed" +check "driver still ran (fail-open)" "$FF" "MARKER=STALE-CONTENT" +checkno "never reports pinned" "$FF" "STATUS=pinned" +git -C "$T/clone" remote set-url origin "$T/origin.git" + +echo "== 5. driver absent FROM THE REF => refuse to exec into nothing ==" +# The driver must EXIST locally (so $0 resolves and the guard is what fires) but be absent from +# origin/dev — the shape a rename-on-dev would produce. A test asserting bash's own +# "No such file" would pass without the guard ever running. +cp "$T/clone/tools/launchd/fake-driver.sh" "$T/clone/tools/launchd/local-only.sh" +MD=$(AILANG_DRIVER_PIN_DIR="$T/pinwt2" /bin/bash "$T/clone/tools/launchd/local-only.sh" 2>&1) +check "guard fires, not bash" "$MD" "has no tools/launchd/local-only.sh" +check "status STALE" "$MD" "STATUS=STALE" +checkno "did NOT exec into nothing" "$MD" "No such file" + +echo "== 6. non-repo source => STALE, not a crash ==" +mkdir -p "$T/plain/tools/launchd/lib" +cp "$SRC_HELPER" "$T/plain/tools/launchd/lib/pin-root.sh" +cp "$T/clone/tools/launchd/fake-driver.sh" "$T/plain/tools/launchd/" +NR=$(/bin/bash "$T/plain/tools/launchd/fake-driver.sh" 2>&1) +check "non-repo is STALE" "$NR" "STATUS=STALE" +check "reason names the repo problem" "$NR" "not a git repository" + +echo "" +echo "==== $PASS passed, $FAIL failed ====" +[ "$FAIL" -eq 0 ] From f09cea9d7ca373c50a67f33b8b3b00dd4ef5b579 Mon Sep 17 00:00:00 2001 From: "Voight-Kampff (bot)" <151556158+sunholo-voight-kampff@users.noreply.github.com> Date: Wed, 12 Aug 2026 10:07:34 +0200 Subject: [PATCH 2/3] ci(launchd): actually RUN the driver tests, on a bash 3.2 runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit added `make test-launchd-drivers` and no job invoked it. A suite that only runs when someone types the command is decoration — and it is the same shape as the bug this branch fixes: a protection that is silently not in effect. macOS runner deliberately. The rig is bash 3.2.57 and GitHub's macOS images still ship 3.2 as /bin/bash; ubuntu ships bash 5, where every `declare -A` and `${v,,}` that would break the rig passes clean. Testing rig shell portability on bash 5 measures the wrong instrument. A guard step asserts BASH_VERSINFO[0] == 3 and fails the job if the runner is ever upgraded, so the coverage cannot silently degrade into a bash-5 run that still reports green. Also made test_pin_root.sh hermetic: it builds a real git lab, and a CI runner has no user.name/user.email and may default init.defaultBranch anywhere, so the lab commits would have failed there and the suite would have reported a git-config problem as a pin failure — green on the author's box, red everywhere else. Now pins identity and neutralises global and system git config. Verified by running it with HOME=/nonexistent and the identity env unset: 20/20. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 21 +++++++++++++++++++++ tools/launchd/test_pin_root.sh | 7 +++++++ 2 files changed, 28 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8ea6bb4e9..e8a195cda 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -444,6 +444,27 @@ jobs: make install-lint make lint + launchd-drivers: + name: launchd drivers (bash 3.2) + # macOS DELIBERATELY, and it is the whole point of this job: the rig runs bash 3.2.57 and + # GitHub's macOS runners still ship 3.2 as /bin/bash, while ubuntu ships bash 5. A suite + # green on bash 5 proves nothing about the constraint that actually bites here — every + # `declare -A` or ${v,,} that reaches tools/launchd/ breaks the rig and passes on ubuntu. + # No Go, no cache: these are shell + git tests and nothing else. + runs-on: macos-latest + + steps: + - uses: actions/checkout@v7 + + - name: Confirm /bin/bash really is 3.2 (guard against a silent runner upgrade) + run: | + /bin/bash --version | head -1 + /bin/bash -c '[ "${BASH_VERSINFO[0]}" -eq 3 ]' \ + || { echo "::error::/bin/bash is no longer 3.x — this job no longer tests the rig's shell"; exit 1; } + + - name: Run launchd driver tests + run: make test-launchd-drivers + govulncheck: name: govulncheck (vuln gate) runs-on: ubuntu-latest diff --git a/tools/launchd/test_pin_root.sh b/tools/launchd/test_pin_root.sh index 24b4442da..2a78aa082 100755 --- a/tools/launchd/test_pin_root.sh +++ b/tools/launchd/test_pin_root.sh @@ -13,6 +13,13 @@ rm -rf "$T"; mkdir -p "$T" T="$(cd "$T" && pwd)" trap 'rm -rf "$T"' EXIT +# Hermetic identity + branch name. A CI runner has no user.name/user.email and may default +# `init.defaultBranch` to anything, so without these the lab commits fail there and the suite +# reports a git-config problem as a pin failure — a test that only passes on the author's box. +export GIT_AUTHOR_NAME=pinlab GIT_AUTHOR_EMAIL=pinlab@invalid +export GIT_COMMITTER_NAME=pinlab GIT_COMMITTER_EMAIL=pinlab@invalid +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null + PASS=0; FAIL=0 ok() { PASS=$((PASS+1)); echo " PASS: $1"; } bad() { FAIL=$((FAIL+1)); echo " FAIL: $1"; echo " got: $2"; } From 9e95aba3157561574e20cf0a0ba1c6f62958ab63 Mon Sep 17 00:00:00 2001 From: "Voight-Kampff (bot)" <151556158+sunholo-voight-kampff@users.noreply.github.com> Date: Wed, 12 Aug 2026 10:09:13 +0200 Subject: [PATCH 3/3] docs(mission): queue m-driver-pin-rollout, gated on measured fires rather than on a date MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the five remaining launchd entry points to the V1 queue as a single rollout item, with the gate written as evidence (>=3 consecutive fires logging a successful pin, read from the driver log rather than from the file's presence on disk) instead of a delay. Three things recorded deliberately, because each has already cost an iteration: * INERT UNTIL RECONCILED. #666 changes nothing on the rig until the shared clone receives it once, and that is a human branch op — the standing fast-forward authorisation does not cover it (local dev is 1 ahead, not 0). This is exactly what deferred m-planner-codex-lane at iter-131: a sprint that lands green, passes CI, and changes nothing while the report claims the capability was gained. * ORDER. Merge #666 first, reconcile second. The reverse brings the lane fix without the pin, and the clone starts drifting again immediately. * THE GATE'S LIMIT. Three green fires exercise the re-exec, the root move and the fetch, but not the failure path, which production will not produce on demand. That arm is covered by the test suite alone, so a passing gate is evidence about the happy path and must not be reported as confidence in the whole fix. ailang-world is explicitly out of scope: different repo, hand-synced fork (513 lines vs 671, differently-shaped fallback site), portable only by hand. Handed over as msg_20260812_085746; World's loop decides when. Co-Authored-By: Claude Opus 5 --- design_docs/v1-mission.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/design_docs/v1-mission.md b/design_docs/v1-mission.md index 488ab7906..d145be255 100644 --- a/design_docs/v1-mission.md +++ b/design_docs/v1-mission.md @@ -1654,6 +1654,43 @@ frozen; contracts projection live). evaluator when the gemini lane is unavailable). Rationale: the opus bucket dried Thursday this week at ~55% duty cycle; these two move the remaining Anthropic-heavy sub-agent roles off-bucket so the week stretches toward Monday-to-Monday. +- **[BLOCKED-ON-EVIDENCE — do NOT pick before the gate below is measured] m-driver-pin-rollout** + (`#558`; mission infra, not a language change; does not gate v1.0). **Prerequisite already + shipped**: PR `#666` (2026-08-12, attended) adds `tools/launchd/lib/pin-root.sh` and wires it + into `mission-control.sh` ONLY — the driver re-execs from a worktree pinned to committed + `origin/dev`, moving the driver, the skill and the charter together, and posting *"driver ran + UNPINNED"* on both human channels when the pin fails. This item is the rollout to the + remaining **five** entry points: `nightly-eval.sh`, `nightly-lang-eval.sh`, + `mission-recovery.sh`, `os-rotation-filler.sh`, `rig-watchdog.sh`. + + **Why one root, not five patches** (Principle 3): everything a fire reads hangs off `REPO` at + `mission-control.sh:40`, so the driver (`$0`), the skill (`cwd`) and the charter (`cwd`) go + stale *together*. Measured four times — `#556`'s retired qwen3.5 running 24/24 (iter-131), a + stale skill (iter-128), a stale charter (iter-129), and `564cc4640` inert on V1 at 12 commits + behind (2026-08-12) while both sibling missions had it. Two one-time human reconciles, zero + durable fixes, until `#666`. + + **INERT UNTIL RECONCILED, and say so rather than reporting the capability as gained** — the + same trap that deferred `m-planner-codex-lane` at iter-131. `#666` cannot take effect until + the shared clone receives it once, and that is a human branch op (the standing fast-forward + authorisation above does NOT cover it: local dev is 1 ahead, not 0). **Order matters:** merge + `#666` FIRST, reconcile SECOND — reconciling first brings the lane fix but not the pin, and + the clone simply starts drifting again. + + **THE GATE (evidence, not a delay):** ≥3 consecutive V1 fires logging `driver pin: running + committed origin/dev @ ` with a normal iteration completing. Read it from + `/tmp/ailang-mission-control.log`, not from the file's presence on disk — measuring the wrong + copy is the same class of error as the bug. **Known limit of that gate, stated up front:** three + green fires exercise the re-exec, the root move and the fetch, but NOT the failure path, which + production will not produce on demand. That arm is covered only by `make test-launchd-drivers` + (37 assertions, bash 3.2 CI job) — so a passing gate is evidence about the happy path alone and + must not be reported as whole-fix confidence. + + **Not in scope, and not ours to schedule:** `ailang-world` is a **different repo** + (`sunholo-data/ailang-world`) whose driver is a hand-synced *fork* — 513 lines against this + repo's 671, with a differently-shaped fallback site. It can never take this by copy, only by + port. Handed over on the cross-mission channel as `msg_20260812_085746`; World's loop decides + when. Do not "fix" it from here. - **[QUORUM-BLOCKED 2026-08-11 (iter-179) — doc LANDED, PR `#657` → squash `0a84f5377`, Gate 3b GREEN (4/4 required, `checks=20`, zero not-green). **Reclassified P0**: `#616` is a STATIC EFFECT-SOUNDNESS hole, not the DX/error-message item it was queued as — a function with NO effect annotation calling a