bug: recovery paths write execution terminals without closing the paired activity — agent shows as running for up to 2h, then logs a fabricated 120-min failure

[obasilakis]

Summary

Every terminal writer outside the dispatching coroutine writes the terminal to schedule_executions but never closes the paired agent_activities row. The activity stays activity_state='started' until the generic 120-minute sweep closes it — so the Dashboard Timeline renders the agent as still working for up to 2 hours after the execution is terminal, and then renders it as a ~120-minute failure with a fabricated duration_ms that is never corrected.

Observed on a live instance: two chat_start activities left open for >80 min against executions that had been terminal failed for 70 min. Same class as #45 (tool-call activities orphaned) and #767 (CB probes inflating timeline duration) — both fixed at a single producer, neither changed the ownership model.

Root cause

The activity id is a local variable in the dispatching coroutine:

# services/task_execution_service.py:1107
activity_id = await activity_service.track_activity(..., related_execution_id=execution_id, ...)

and the close is gated on that same coroutine winning the CAS:

# services/task_execution_service.py:816-826  (_write_terminal_and_gate)
won = db.update_execution_status(execution_id=..., status=status, ...)   # CAS
...
if won and activity_id:                                                   # <- the gate
    await activity_service.complete_activity(...)

So the activity is closed only when (a) the original coroutine is still alive AND (b) it wins the CAS. Two independent ways that fails:

(a) Process death. execute_task has no finally that closes the activity — the close happens exclusively through this applier. If the backend process dies mid-turn, activity_id dies with it and nothing else knows the row exists. (With --reload in dev, every file save that lands mid-execution mints one.)

(b) CAS loss. A late in-process writer that loses the CAS to a recovery path hits won=False and skips line 825. Note the asymmetry — the SUCCESS applier does handle CAS loss (task_execution_service.py:1841 if not won: closes with activity_state_for_terminal(...), #1332); _write_terminal_and_gate has no equivalent branch.

And the writer that wins the CAS never takes over the responsibility. Both recovery paths stop at the row:

# services/cleanup_service.py:2031-2048  (watchdog orphan / auto-terminate)
updated = db.mark_execution_failed_by_watchdog(execution_id, combined_error)
await capacity.release_if_matches(agent_name, execution_id)   # slot: released
await self._broadcast_watchdog_event(...)                     # WS: broadcast
#                                                               activity: never closed

# services/cleanup_service.py:2383-2387  (startup restart recovery)
db.mark_execution_failed_by_watchdog(..., "Execution orphaned — recovered on backend restart")
await capacity.release(agent_name, execution["id"])
#                                                               activity: never closed

The bulk sweeps _sweep_stale_executions (cleanup_service.py:602) and _sweep_no_session_executions (:631) have the same gap.

The helper already exists and is two methods away. db/activities.py:183 get_open_activity_id_for_execution() — filtered to chat_start|schedule_start + state='started' — was written for exactly this. It has precisely two callers, both #1083/#429-specific: _close_stale_slot_activity (cleanup_service.py:1549) and _close_reaped_activity (:579). None of the writers above call it.

Impact

1. Timeline shows phantom in-progress work. ReplayTimeline.vue:670 — const isInProgress = event.status === 'started'. (The agent tile is unaffected: agent_service/stats.py:202 derives activityState from recency, not activity_state.)
2. Permanently wrong duration. The backstop db/activities.py:331 sets duration_ms = now − started_at, so a 15-minute run closes as a 120-minute failure. Nothing recomputes it. This is exactly the misleading-timeline symptom of bug(timeline): CB probe executions left open until backend restart inflate failure duration on timeline #767, one layer down.
3. Self-heals slowly, not correctly — ACTIVITY_STALE_TIMEOUT_MINUTES = 120 (cleanup_service.py:56) via _sweep_stale_activities (:703).

Reproduction

1. Trigger a long execution (POST /api/agents/{name}/schedules/{id}/trigger).
2. While it runs, restart the backend (or save a file under --reload).
3. Startup recovery / the watchdog marks the execution failed.
4. SELECT * FROM agent_activities WHERE activity_state='started' — the chat_start row is still open, related_execution_id pointing at the now-terminal execution.
5. Dashboard Timeline renders the agent as in-progress until started_at + 120min, then as a 2-hour failure bar.

Suggested fix

Move the responsibility from "whoever holds the local activity_id" to "whoever wins the CAS":

1. Call get_open_activity_id_for_execution() + complete_activity() inside both _recover_execution implementations (cleanup_service.py:1991, :2378) and after the two bulk sweeps (rows are already collected in collect_failed for the bug: bulk watchdog sweeps emit no task-completion event — swept executions never wake their subscriber (#1578 residual) #1714 event emit — the same list can drive the activity close).
2. Add the missing CAS-loss branch to _write_terminal_and_gate (:825), mirroring the SUCCESS applier at :1841.
3. Preferably fold the lookup into the terminal-write contract so a recovery path cannot land a terminal without it — otherwise producer fix: add missing logging_config.py to backend Dockerfile #4 repeats this. See the refactor: collapse 9-path cleanup pyramid once agent is authoritative #429 comment: the pull sink (pull_coordination_service.apply_task_result) and the lease reaper's re-queue branch already have the same gap, and refactor: collapse 9-path cleanup pyramid once agent is authoritative #429's success criterion (grep mark_stale_* → zero hits) would remove the 120-min backstop without adding an owner.

Prior art

• bug: Tool call activities created but never completed, accumulating as orphans #45 — tool-call activities created but never completed
• bug(timeline): CB probe executions left open until backend restart inflate failure duration on timeline #767 — CB probe executions left open, inflating timeline failure duration

[AndriiPasternak31]

/claim — picking this up (AndriiPasternak31). Worktree + plan in progress; will open a draft PR with Fixes #1804.

[github-actions]

Claimed by @AndriiPasternak31. Label status-in-progress added.

Next steps:

1. Create branch: feature/1804-<short-slug>
2. Implement the feature
3. Open a PR with Fixes #1804