Summary
CleanupService currently runs 9 distinct reconciliation paths writing to schedule_executions.status:
- Phase 0 — reconciliation against agent
/api/running
- Phase 1 — stale >120min → FAILED
- Phase 1b — no-session >60s → FAILED
- Phase 1c — orphan-skipped → finalize
- Phase 3 — slot reclaim with re-verify (patch for FAILED→SUCCESS race)
- Slot TTL (timeout + 5min buffer)
- Backlog 24h expiry
- TES httpx timeout → terminate + FAILED
- Activity stale >120min → FAILED
Each exists because the layer below can't be trusted. schedule_executions.status has 12 writers; the FAILED→SUCCESS race is possible and currently mitigated by Phase 3 just-in-time re-verify.
Target
One periodic reconciliation: DB.running ⟷ agent./api/running (agent wins). Capacity is recomputed from DB, not TTL'd. Single writer per status transition.
Depends on
Gating condition — DO NOT SHIP EARLY
#306 must be in production for ≥2 weeks with zero observed orphan recoveries before this lands. The watchdog pyramid is load-bearing right now; ripping it out before push-based completion has soaked would trade known bugs for unknown ones.
Scope
- Delete Phase 1 / 1b / 1c / 3 reconciliation from
cleanup_service.py.
- Simplify slot TTL logic — capacity becomes "count DB rows in running state," no expiry.
- Reduce
schedule_executions.status writers to ~4 (create, start, finish, external-cancel).
- Update
architecture.md cleanup section.
- Write a migration guide for ops — what signals were previously "phase X recovered N" and where they live now.
Success criteria
- Grep for
mark_stale_*, mark_no_session_*, finalize_orphaned_*, fail_stale_slot_* returns zero hits.
- Single writer per status transition, verifiable by audit query.
- Phase 0 reconciliation still exists but is the only reconciliation loop.
Context
See docs/planning/ORCHESTRATION_RELIABILITY_2026-04.md — Tier 2.5 Simplification.
Summary
CleanupServicecurrently runs 9 distinct reconciliation paths writing toschedule_executions.status:/api/runningEach exists because the layer below can't be trusted.
schedule_executions.statushas 12 writers; the FAILED→SUCCESS race is possible and currently mitigated by Phase 3 just-in-time re-verify.Target
One periodic reconciliation:
DB.running ⟷ agent./api/running(agent wins). Capacity is recomputed from DB, not TTL'd. Single writer per status transition.Depends on
Gating condition — DO NOT SHIP EARLY
#306 must be in production for ≥2 weeks with zero observed orphan recoveries before this lands. The watchdog pyramid is load-bearing right now; ripping it out before push-based completion has soaked would trade known bugs for unknown ones.
Scope
cleanup_service.py.schedule_executions.statuswriters to ~4 (create, start, finish, external-cancel).architecture.mdcleanup section.Success criteria
mark_stale_*,mark_no_session_*,finalize_orphaned_*,fail_stale_slot_*returns zero hits.Context
See
docs/planning/ORCHESTRATION_RELIABILITY_2026-04.md— Tier 2.5 Simplification.