Production soak-test of pull/work-stealing coordination (#1081) before default-on

[vybe]

Summary

The push→pull ("pool-based") coordination migration (#1081) is implemented but has never been exercised at full scale under real production load — only local workflows. Before pull mode can be considered for default-on, we need a controlled soak test on a live production instance carrying real agent traffic, watching for lost/phantom executions, lease/re-delivery behavior, and any regressions surfaced by the instance monitor.

Context

Raised in the 2026-07-24 Intelligence Design weekly. The platform has become notably more reliable on the current (push) path, which paradoxically raises the risk of switching — the current mode is working, so the switch has to be proven, not assumed. The Direction-B rails are already in place (#1402 re-delivery cap + async operator-queue human-gate, #1408 tool-side effect gates, #1084 effect-idempotency); what's missing is production evidence. Plan: update the target instance to the latest build first, flip pull mode for a bounded pilot set of agents (PULL_MODE_PILOT_AGENTS), and let the instance monitor report anomalies over a soak window.

Acceptance Criteria

• Target production instance updated to the latest build before enabling pull mode
• Pull mode enabled for a bounded pilot set of agents (PULL_MODE_PILOT_AGENTS)
• Soak window observed (several days of real traffic) with the instance monitor active
• No lost or phantom executions; leases re-queue / poison-park as designed; no slot/backlog drift (canary green throughout)
• Findings written up — what broke, what held, and whether pull is ready to widen — and linked back to feat: pull / work-stealing coordination (push→pull dispatch) #1081
• Any regression found is filed; if a blocker, fixed before widening the pilot

Technical Notes

• Parent epic: Agent Infrastructure (Epic: Agent Infrastructure #1045); centerpiece work item feat: pull / work-stealing coordination (push→pull dispatch) #1081.
• Rails/gates: feat: MAX_REDELIVERY cap + async operator-queue human-gate lever (lease-reaper backstop) #1402, feat: deterministic tool-side gate on capability-confined irreversible rails (v2 gap) #1408, feat: effect-scoped idempotency keys for outbound side effects #1084; cleanup-collapse refactor: collapse 9-path cleanup pyramid once agent is authoritative #429 sequences after phase 1.
• The physical-occupancy shadow meter and lease reaper are already dark-shipped (inert until an agent is piloted) — this test lights them up under load.
• Priority left at P2 for now; may warrant P1 given feat: pull / work-stealing coordination (push→pull dispatch) #1081 is the reliability spine's centerpiece.
• Owner: Vas.

[obasilakis]

Prep status + two blockers found (2026-08-04)

Code prep — PR #1982 (open, against dev)

Two defects would have made a flag-flip soak unable to produce the evidence the ACs above ask for:

1. The pilot flag was purely additive — push and pull ran in parallel inside one agent. CapacityManager.acquire had no pilot branch, so a free slot still meant a push and rows only queued on capacity overflow; when one did queue, the backend's own backlog_service.drain_next raced the agent's worker through the same db.claim_next_queued (no double-run — the claim is one atomic UPDATE — but the backend was structurally favoured: it drains on slot release plus a 60s orphan sweep, a worker idles up to 15s). Pull coverage was therefore near-zero and drain-biased. Two independent capacity counters (Redis ZSET vs the container's worker pool) also meant a pilot could run up to 2x max_parallel_tasks, invisible to canary S-02 (ZCARD only) and S-01 (excludes leased rows by design). The flag is now a true either/or for autonomous triggers; interactive turns keep the synchronous path.
2. E-05 fired on every pull turn past 60s. A pull-claimed row is running with a NULL claude_session_id by design, and mark_no_session_executions_failed — the sweep E-05 watches — already carries lease_expires_at IS NULL for that reason. Unfixed it breaks the "canary green throughout" AC and buries a real bug: Cleanup service misses 'skipped' executions and slow to detect no-session failures #106 regression in noise for the whole window.

PR also adds the soak measurement set (docs/testing/PULL_MIGRATION_TESTING.md §9): 8 queries mapped to the ACs above, a pre-flight baseline step, abort criteria, rollback.

Blockers

• Canary never runs on prod compose (CANARY_ENABLED not forwarded), and has no cross-worker leader lease when it does #1881 (canary leader lease) is now a hard prerequisite. Its packaging half already shipped (fix(canary): wire CANARY_ENABLED/_SLACK_WEBHOOK_URL into prod backend env #1876), so only the missing lease remains — and eu2 runs --workers 2. Enabling the canary there without it double-probes 12 agents per cycle and double-persists canary_violations, which is exactly what M7 counts. The go/no-go numbers would be doubled.
• AC 1 is real work: eu2 runs 9f766d11 (built 2026-07-20); dev has moved well past it.

eu2 state (recorded so it isn't lost)

• Not deployed. Still on 9f766d11. Nothing from fix(pull): make the pilot flag a true either/or + soak measurement set (#1766) #1982 is on the box.
• ~/trinity/.env now has CANARY_ENABLED=1 + CANARY_SLACK_WEBHOOK_URL (copied from dev, the only other instance with it). Backup: ~/trinity/.env.bak-20260804T140531Z.
• Both are inert — compose bakes env at container create; the backend has not been recreated. Intentional, pending Canary never runs on prod compose (CANARY_ENABLED not forwarded), and has no cross-worker leader lease when it does #1881.
• CANARY_ENABLED must be exactly 1 — canary_service.py:130 is os.getenv("CANARY_ENABLED","0") == "1", so true reads as disabled (see CANARY_ENABLED silently disables on any truthy value except "1" #1988).
• eu2 is PostgreSQL; §9's queries carry the required ::timestamptz casts (timestamp columns are TEXT).

Flip procedure is three steps, not one

PULL_MODE_PILOT_AGENTS is process env (backend restart) and the pilot agent must be recreated — there is no config-drift predicate for pull mode, so TRINITY_PULL_MODE only bakes in at create/recreate. Restart alone leaves the agent not pulling, which is how you soak a no-op for two days.

Still open before the window

Pilot agent not chosen. Baseline (M4/M6/M7) not taken. Prior evidence: a live pilot on 2026-07-08 already proved claim → run → CAS terminal → requeue x3 → poison-park end-to-end (PULL_MIGRATION_TESTING.md §6), so the smoke pass is largely redundant — the remaining gates are §8's.

[vybe]

Prep PR #1982 merged to dev (pilot flag now a true either/or + soak measurement set). No closing keyword by design — the soak itself is this ticket and remains to be run; leaving status-ready so it stays pickable.