Skip to content

refactor: collapse 9-path cleanup pyramid once agent is authoritative #429

Description

@vybe

Summary

CleanupService currently runs 9 distinct reconciliation paths writing to schedule_executions.status:

  1. Phase 0 — reconciliation against agent /api/running
  2. Phase 1 — stale >120min → FAILED
  3. Phase 1b — no-session >60s → FAILED
  4. Phase 1c — orphan-skipped → finalize
  5. Phase 3 — slot reclaim with re-verify (patch for FAILED→SUCCESS race)
  6. Slot TTL (timeout + 5min buffer)
  7. Backlog 24h expiry
  8. TES httpx timeout → terminate + FAILED
  9. Activity stale >120min → FAILED

Each exists because the layer below can't be trusted. schedule_executions.status has 12 writers; the FAILED→SUCCESS race is possible and currently mitigated by Phase 3 just-in-time re-verify.

Target

One periodic reconciliation: DB.running ⟷ agent./api/running (agent wins). Capacity is recomputed from DB, not TTL'd. Single writer per status transition.

Depends on

Gating condition — DO NOT SHIP EARLY

#306 must be in production for ≥2 weeks with zero observed orphan recoveries before this lands. The watchdog pyramid is load-bearing right now; ripping it out before push-based completion has soaked would trade known bugs for unknown ones.

Scope

  1. Delete Phase 1 / 1b / 1c / 3 reconciliation from cleanup_service.py.
  2. Simplify slot TTL logic — capacity becomes "count DB rows in running state," no expiry.
  3. Reduce schedule_executions.status writers to ~4 (create, start, finish, external-cancel).
  4. Update architecture.md cleanup section.
  5. Write a migration guide for ops — what signals were previously "phase X recovered N" and where they live now.

Success criteria

  • Grep for mark_stale_*, mark_no_session_*, finalize_orphaned_*, fail_stale_slot_* returns zero hits.
  • Single writer per status transition, verifiable by audit query.
  • Phase 0 reconciliation still exists but is the only reconciliation loop.

Context

See docs/planning/ORCHESTRATION_RELIABILITY_2026-04.md — Tier 2.5 Simplification.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions