tracker-intake: duplicate worker/PR spawned for an open issue whose worker was terminated after opening its PR (regression window from #2777)

[illegalcall]

Summary

After #2777, the tracker-intake poller can spawn a second worker (and a second PR) for an issue that already has an in-flight PR, when the issue's original worker was terminated (killed or crashed) after opening its PR but before the PR merged.

#2777 correctly fixed #2746 (a terminated session used to "claim" its issue forever, so a dead worker silently blocked re-spawn). It did so by excluding terminated sessions from the seen set in seenIssueIDs. The trade-off: intake now frees an issue the moment its session is terminated — with no awareness of whether that session already produced an open PR — so a terminated-but-productive worker triggers a duplicate spawn.

Why the state is reachable (no crash or race required)

The simplest trigger is a routine operator action:

1. Intake spawns worker A for open issue #N. A opens PR fix: three spawn regressions from PR #88 #100 (Fixes #N), which stays open (unmerged). Issue #N stays open — Fixes #N closes the issue only on merge.
2. An operator kills worker A (stuck/misbehaving/rebalancing). Manager.Kill marks the session IsTerminated=true and deletes the restore marker so it will not be resurrected (backend/internal/session_manager/manager.go:663-666). Killing the session does not close PR fix: three spawn regressions from PR #88 #100.
3. Next intake tick (default 1 minute, observer.go:22): the only session for #N is terminated, so seenIssueIDs drops it. Intake has no PR surface and Spawn has no issue-level dedup, so it spawns worker B for #N. B opens PR feat: first-class orchestrator session + file-based system prompt #101.

Result: two workers and two PRs for one issue.

The same end-state is reachable via a worker crash/SIGKILL (reaper observes the runtime clearly dead → IsTerminated=true, manager.go:113-118) or the agent CLI exiting (ActivityExited → IsTerminated=true, manager.go:203-205). There is no runtime crash-auto-respawn: Restore/RestoreAll run only from the boot-time Reconcile pass (manager.go:1071), so the terminated session lingers until the next boot — well past the 1-minute tick.

Root cause

• ListAllSessions returns all sessions including terminated (SELECT ... FROM sessions, no is_terminated filter — backend/internal/storage/sqlite/queries/sessions.sql), so terminated rows persist indefinitely.
• seenIssueIDs dedups only on live sessions (sess.IssueID != "" && !sess.IsTerminated, backend/internal/observe/trackerintake/observer.go:244-252).
• Intake's Store interface (observer.go:35-38) exposes only ListProjects/ListAllSessions — no PR facts — so pollProject cannot tell that an open PR already exists (observer.go:196-217).
• Manager.Spawn has no issue-level dedup — it unconditionally creates a session (backend/internal/session_manager/manager.go:246).

So once the live session disappears, nothing else records that the issue is already being worked.

Reproduction

Deterministic unit reproduction of the intake decision (drives the real Observer.Poll and seenIssueIDs; stubs store/tracker/spawner; the crash/kill transition is modeled by flipping the session to IsTerminated, the state Kill/reaper/ActivityExited are proven to produce):

Poll 1: issue #12 open, no sessions      -> spawn worker A (A opens PR #100, open/unmerged)
(worker A killed/crashed)                -> its session IsTerminated=true; PR #100 open, issue #12 open
Poll 2: only session for #12 is terminated -> spawn worker B (DUPLICATE; B opens PR #101)

• Against the merged code: two spawns for #12.
• Against the pre-fix(tracker-intake): exclude terminated sessions from seen issue set #2777 line (if sess.IssueID != ""): one spawn — confirming this window is introduced by fix(tracker-intake): exclude terminated sessions from seen issue set #2777 (which in turn was necessary to fix bug(tracker-intake): terminated sessions block re-spawn of their issue (intake never re-assigns killed/crashed issues) #2746).

Full-daemon/live-agent reproduction (boot daemon → real worker opens a real PR → ao kill → observe a second spawn on the next tick) has not been run; the above is unit-level plus exhaustive code tracing.

Suggested fix direction

Give intake a claim signal that is independent of session liveness, so a terminated-but-productive worker does not free its issue:

• Extend the intake Store (or add a small port) to expose "issue already has an open or merged PR" (e.g. last-attributed PR per canonical issue id), and skip the issue in pollProject when one exists.

This closes the window without reintroducing #2746: a worker that died without ever producing a PR still frees its issue for re-spawn.

Distinct from #2745 (reaper cannot terminate stuck sessions) — that is a separate root cause.

Found during review of #2777.

[somewherelostt]

Thanks for contributing to Agent Orchestrator.

This issue is being picked up by the current external contributor on-call pair:

• @illegalcall
• @Pulkit7070

If someone is already working on this, please continue as usual.
The on-call pair is added for visibility, tracking, and support, not to take over the work.
If you need help with review, direction, reproduction, or next steps, please tag @illegalcall and @Pulkit7070 here.

For faster context or live questions, you can also join the AO Discord.

Join the session here:
https://discord.gg/H6ZDcUXmq

Come by if you want to see what is being built, ask questions, or just hang around with the community.