feat(streaming): wake the agent server-side when a notify_on_complete background task finishes while the WebUI tab/session is idle

[Isla-Liu]

Thinking Path

• A core Hermes Agent guarantee is that a terminal(background=true, notify_on_complete=true) task wakes the agent when it finishes, so the agent can act on the result on its own. The CLI and the gateway hosts (Telegram, Discord, etc.) already do this by draining the completion queue inside their own run loops.
• In the WebUI this guarantee did not hold. The background process exited, its completion was queued in the agent process registry, and nothing read from it — the agent only resumed when a human typed the next message. The agent had told the user "I'll continue when this finishes," and then silently never did.
• The merged PR Fix WebUI stream completion recovery gaps #2279 closed part of this gap with a next-turn safety-net drain: at the start of the next user-initiated turn it drains any pending completions and prepends them to the user's message. That is the correct backstop, but it cannot help the primary case this PR targets: fire a long task, close the tab, come back later — an autonomous agent in that state has no "next user turn," and a closed tab can never re-POST anything to the server.
• The structurally correct fix is therefore to start the wakeup turn on the server, with no browser round-trip — exactly the way the CLI and gateway hosts already self-wake — and to keep an open tab able to watch that server-started turn live.

What Changed

This PR layers a proactive, server-side wakeup on top of the merged #2279
next-turn drain. It carries none of #2279's code (that is upstream now);
it is rebased onto current master.

• Server-side wakeup turn (primary path). A startup drain thread reads the agent process completion queue and, when the owning session is idle, starts the agent turn directly on the server via a new headless entrypoint (api/routes.py::start_session_turn) that reuses the exact session-load → workspace → model/provider → worker-thread path /api/chat/start already uses. No second agent-construction path is introduced. If a turn is already active, it does not start a competing turn and instead leaves the completion for the merged Fix WebUI stream completion recovery gaps #2279 next-turn drain.
• Persistent per-session live-view channel (GET /api/session/stream). A long-lived SSE channel keyed on the WebUI session_id (so it survives across turns, unlike the per-turn chat stream). A reaper thread collects channels with no subscribers after a grace period and idle channels after a configurable TTL; a channel with a live subscriber is never collected. When a turn is started server-side, an open tab attaches the existing chat-stream renderer to it and shows it live; a closed tab simply gets the persisted result on next load (parity with CLI behavior).
• SSE write-deadline backpressure. All long-lived SSE endpoints now arm a socket write deadline, so a slow or backgrounded tab whose TCP receive window has filled can no longer pin a server worker thread indefinitely (which previously also blocked the live-view reaper).
• Cold model-catalog resolution no longer blocks a wakeup. A server-side wakeup fires exactly when the model catalog is cold; the cold rebuild could make a blocking network call (provider token exchange) that hung the turn on restricted networks. The wakeup now resolves from warm/disk cache or a network-free minimal catalog, and the already-persisted session model always wins. Integrated with the existing upstream model-resolve fast-path.
• Fast-task completion race. When a task finishes in the narrow window while a turn is tearing down, the wakeup prompt is persisted and redelivered by a turn-teardown idle hook, making the busy-at-completion case behave the same as the idle case. The claim is atomic, so it fires exactly once and never double-fires with the merged Fix WebUI stream completion recovery gaps #2279 drain.
• Cross-session wakeup isolation. Per-turn session identity is bound to a context variable for the duration of the turn (plus a completion-time target cross-check as defense in depth), so with multiple concurrent WebUI sessions a completed background task always wakes the session that launched it, never another open session.
• Copilot review comments on this PR have been addressed (agent working notes removed from the repo root, a misleading ack response field corrected, owner-unknown streams no longer broadcast cross-session, the dedupe contract asserted instead of silently swallowing a registry rename, and local absolute paths stripped from a test).

Why It Matters

The headline use case for notify_on_complete is "start something long, walk
away, let the agent pick it up when it's done." Before this PR a WebUI user
who did that — especially with the tab closed — got nothing: the agent
appeared to hang forever. After this PR the agent wakes itself on the server
the moment the task completes, an open tab sees that turn render live, and a
closed tab finds the completed turn already persisted on next load. WebUI
now matches the wakeup behavior the CLI and gateway hosts have always had.

Relationship to #2279

PR #2279 (merged into master) is the next-turn safety-net drain — it
delivers a pending completion at the start of the next user turn. This PR is
the proactive server-side layer on top of it: it wakes the agent without
waiting for a next user turn and without a browser round-trip. The two share a
single completion-consumed dedupe key (the contract #2279 established), so a
completion wakes the agent exactly once regardless of which path fires first;
neither double-fires and there is no wakeup loop. This PR carries none of
#2279's code.

Refs #2278 (root issue — WebUI never fires notify_on_complete wakeups;
closed by #2279 for the next-turn case, completed here for the
idle/closed-tab case). Builds on #2279. The #2262 compression-marker fix was
shipped upstream via #2279 and is intentionally not re-added here.

Verification

• Full Python regression suite: 0 regressions vs the post-Fix WebUI stream completion recovery gaps #2279-merge master baseline.
• Wakeup-critical suites pass, covering: closed-tab self-wake when idle; fast vs. slow completion (completion-during-teardown); live-view of a server-started turn plus SSE write-deadline backpressure; no double-fire / no loop across the merged Fix WebUI stream completion recovery gaps #2279 and this PR in both interleavings; cold model-catalog resolution surviving a hung network probe; cross-session wakeup isolation under concurrent sessions.
• Upstream Fix WebUI stream completion recovery gaps #2279 contract suites remain green and unchanged.
• Live-instance verification: a real WebUI instance self-woke on both a fast and a slow notify_on_complete background task with the tab closed, and an open tab rendered the server-started turn live (verified with Playwright).
• git diff --check clean; touched modules py-compile clean.

Risks / Follow-ups

• Adds two long-lived server threads (the completion drain and the live-view reaper). Both are daemon threads with bounded work; the reaper's grace period and idle TTL are configurable.
• The live-view channel is keyed on session_id and survives across turns; the reaper TTL is the upper bound on an abandoned channel's lifetime.
• The completion-consumed dedupe key is the contract established by the merged Fix WebUI stream completion recovery gaps #2279; if a future change alters that contract, both the next-turn drain and this proactive path must be updated together.

Model Used

AI-assisted change with repository inspection, source-level root-cause
analysis, targeted editing, automated regression suites, and live-instance
verification. AI assisted with code, tests, verification, and PR text.

[Isla-Liu]

Self-review Q1-Q4 from internal reviewer (decisions, ahead of upstream review):

Q1 Event name — keeping process_complete. Matches outline/RFC/tests/CHANGELOG; renaming pre-merge isnt worth the churn.

Q2 /api/process-complete-ack auth — keeping current (mirror /api/goal-continue ownership check). Endpoint is diagnostic-only; the real auth-bearing path is /api/chat/start.

Q3 watch-pattern UX — keeping single event channel; will branch toast text in frontend on data.type as a follow-up commit if maintainers want polish. Splitting events doubles the contract surface for a UX nit.

Q4 hermes-agent ±5 LOC companion patch — deferring indefinitely. WebUI-side env export is sufficient and matches the CLI/gateway pattern.

Heads-up for maintainers: PR #2279 was opened today by another contributor as the Option A drain implementation for issue #2278. We are rebasing this PR on top of #2279 to position it as the Option B proactive-notification layer (per maintainer endorsement in #2278: "Option B can land later if there is demand"). #2262 compression-marker fix attribution stays with #2279. Updated PR body + integrated branch coming after local verification.

[nesquena-hermes]

Maintainer signal on Q1–Q4 so you can move forward:

Decisions

Q1 — Event name process_complete: keep as-is. Agreed, renaming pre-merge is churn.

Q2 — /api/process-complete-ack auth model: the ownership check mirroring /api/goal-continue is fine for the diagnostic ack endpoint. The real auth-bearing path is /api/chat/start, as you noted. Approved.

Q3 — Single event channel + frontend branching on data.type: approved. Splitting events would double the contract surface for a UX polish concern. If toast-text variants are wanted later, branching in messages.js based on the payload is the right place.

Q4 — Defer hermes-agent companion patch: approved. WebUI-side env export matches the CLI/gateway pattern, no need for an agent-side change in this slice.

Status

PR #2279 already merged (May 15) as the Option A drain layer. Path forward for this PR:

1. Rebase on current master. v0.51.78 / v0.51.79 / v0.51.80 have shipped since you opened this. mergeable: CONFLICTING right now.
2. Reposition the PR body to reflect the new layering: Option B proactive-notification layer on top of Fix WebUI stream completion recovery gaps #2279's drain helper.
3. CI run. Once rebased and green, mark "ready for review" — we'll route through independent review for the streaming changes (api/streaming.py + api/background_process.py are sensitive paths).

If the rebase is clean, no maintainer intervention needed — just push the rebase + flip out of draft and we'll review.

Ping back if any of the Q1–Q4 decisions affect implementation enough that you need to revisit before rebasing.

[Copilot]

Pull request overview

This PR layers proactive server-side agent wakeup ("Option Z") and a persistent per-session SSE live-view channel ("Option X") on top of the recently merged upstream PR #2279 (next-turn drain for terminal(notify_on_complete=true)). It adds a new api/background_process.py module containing a drain thread that reads tools.process_registry.completion_queue and, when the owning session is idle, starts a wakeup agent turn directly server-side — making the "fire a background task, close the tab, come back later" use case work the way CLI/gateway hosts already do. It also adds several follow-ups: an SSE write-deadline to prevent thread exhaustion, a model-resolve cache-only fast path to avoid a Copilot token-exchange hang during cold-cache wakeups, and a defer-race teardown hook so fast background tasks that complete during a turn's teardown window are redelivered when the session goes idle.

Changes:

• New api/background_process.py (drain thread, SessionChannel registry + reaper, deferred-wakeup persistence, server-side wakeup starter) and a new /api/session/stream SSE endpoint and start_session_turn headless entrypoint in api/routes.py.
• api/config.py gains get_available_models(prefer_cache=True) + a bounded live-rebuild worker (_LIVE_REBUILD_BUDGET_SECONDS, _minimal_static_models_catalog) and new state (DEFERRED_PROCESS_WAKEUPS, PROCESS_SESSION_INDEX, SESSION_CHANNEL_* TTLs); api/streaming.py adds _sse_set_write_deadline, env wiring for HERMES_SESSION_CHAT_ID, and a turn-teardown drain hook.
• Frontend (static/messages.js, static/sessions.js) opens/closes a per-session SSE stream, dedupes process_complete events module-wide, and attaches the existing attachLiveStream renderer when the server fans a server_turn_started frame.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
api/background_process.py	New module: drain thread, SessionChannel + reaper, deferred-wakeup persistence, server-side wakeup runner.
api/config.py	Adds prefer_cache mode, bounded rebuild worker, minimal static catalog, and new state buckets for wakeup/SessionChannel.
api/routes.py	Adds start_session_turn, /api/session/stream + /api/process-complete-ack handlers, threads prefer_cached_catalog, arms SSE write deadlines, fans server_turn_started.
api/streaming.py	Adds _sse_set_write_deadline, exports HERMES_SESSION_CHAT_ID, registers per-session process mapping, calls turn-teardown deferred-wakeup drain.
server.py	Starts/stops the drain thread and SessionChannel reaper.
static/messages.js	Module-scope dedupe set, process_complete listener, startSessionStream/stopSessionStream, server_turn_started attach-existing-renderer logic.
static/sessions.js	Wires session-stream start/stop into newSession/loadSession.
CHANGELOG.md	Long entries describing the Option Z pivot, Option X channel, model-resolve fix, defer-race fix, and SSE backpressure fix.
tests/test_session_channel_option_x.py, test_process_complete_wakeup.py, test_process_complete_ab_coexistence.py, test_optionz_liveview_perf.py, test_wakeup_model_resolve_hang.py, test_wakeup_defer_race.py, manual_repro_wakeup_hang.py	New tests for SessionChannel lifecycle, AB-coexistence dedupe, live-view fan-out + SSE backpressure, model-resolve hang fix, and defer-race teardown hook.
tests/test_bugbatch_apr2026.py	Source-grep updated for new get_available_models( signature.
wakeup-model-resolve-hang-fix.md, wakeup-defer-race-fix.md, optionz-liveview-perf-fix.md	New design/repro notes (markdown only).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Isla-Liu]

background task working good behavior image:

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

[Isla-Liu]

Thanks — agreed. These three files were working notes, not maintained project documentation, and the substance is already captured (operator details stripped) in the [Unreleased] section of CHANGELOG.md. I've removed all three from the repo root rather than relocating them, since nothing in them is unique to keep. Done in commit a76c050.

[Isla-Liu]

Good catch — that field and comment were stale after the Option Z pivot and contradicted the function's own docstring. Since the endpoint is now a pure no-op for state (the wakeup turn is started server-side; the browser never re-POSTs /api/chat/start), I've dropped "pending_consumed": False and replaced it with "noop": True plus a comment that matches the documented behavior. I verified nothing reads the old field first — the frontend fires the ack fire-and-forget (.catch(() => {})) without inspecting the body, and no test asserts it. Fixed in commit cee897a.

[Isla-Liu]

Resolved by skipping non-matching streams when the owner is unknown — the option you preferred. _emit_to_session_streams now does if owner_sid != session_id: continue (previously if owner_sid and owner_sid != session_id), so a stream with no matching ACTIVE_RUNS row is no longer broadcast to, removing the cross-session-leak surface.

This was justified with a deterministic Playwright reproduction of the related open-tab live-view defect: the per-session SessionChannel (emitted unconditionally below this loop), now backed by an on-subscribe recovery replay, is the sole authoritative cross-turn / server-initiated live-view carrier after the Option X/Z work. The STREAMS loop is in-turn defense-in-depth only and is never the path that delivers a between-turns or server-initiated wakeup, so the broadcast fallback is not load-bearing for any live-view path. Covered by regression tests (owner-unknown skip, matching-owner still delivered, cross-session isolation). Fixed in commit 29cdce2.

[Isla-Liu]

Addressed with the defensive option (b). The read side already uses the public process_registry.is_completion_consumed(); only the write side reached into the private _lock / _completion_consumed, because there is no public mark_completion_consumed() upstream. I've extracted _mark_registry_completion_consumed() which documents the coupling and narrows the exception handling: ImportError stays best-effort (the registry is legitimately absent in non-agent unit-test contexts), but an AttributeError/TypeError from a future rename or retype is now logged at ERROR with an explicit contract-violation message instead of being swallowed silently at DEBUG. A future rename therefore fails loud rather than silently reintroducing the double-wakeup.

I also added a contract test (test_registry_completion_consumed_contract) plus a fail-loud test in tests/test_process_complete_ab_coexistence.py that pin the expected ProcessRegistry private surface, so an upstream rename breaks CI here visibly. Fixed in commit 9f3c7b6.

For the longer term, option (a) — a public mark_completion_consumed(process_id) / is_completion_consumed(process_id) pair on ProcessRegistry upstream — is the durable fix; I'm happy to submit that upstream and switch this call site to it if you'd like.

[Isla-Liu]

Moved it out of the test root: git mv tests/manual_repro_wakeup_hang.py tests/manual/repro_wakeup_hang.py. I stripped the hardcoded absolute interpreter path from the docstring (the run instruction is now parametric — python tests/manual/repro_wakeup_hang.py from the repo root) and fixed its repo-root sys.path computation for the deeper location.

For the collection exclusion I used collect_ignore_glob = ["manual/*.py"] in the existing top-level tests/conftest.py rather than a nested tests/manual/conftest.py: a second conftest module collides with the suite's top-level from conftest import TEST_BASE/TEST_STATE_DIR imports (the suite uses rootdir-relative module names, no packages). Verified pytest --collect-only tests/ collects cleanly with no collection errors and the manual script no longer surfaces. Fixed in commit 3b61772.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Isla-Liu]

Rebased onto latest origin/master to resolve the merge conflicts (the branch had fallen 41 commits behind).

Rebase summary

• Semantic rebase (not merge): 15 commits replayed linearly on origin/master, 0 merge commits, original authorship preserved. New head d6753e55.
• Zero semantic loss: git diff <merge-base>..<old-head> --stat is byte-identical to git diff origin/master..<new-head> --stat (20 files, +4935/−20 both sides). The only content delta is one indentation change on a single static/sessions.js line, grafted into upstream's coalesced IIFE.
• Conflict files resolved hunk-by-hunk preserving both intents: api/streaming.py (our Option B/Z + Option X SSE alongside upstream's reasoning-replay / fallback-warning / keyless-endpoint changes), api/routes.py (our wakeup/SSE dispatch + the prefer_cached_catalog fix integrated with upstream's bug: /api/chat/start and /api/sessions can wedge for ~125s while health remains responsive #1855 fast-path, both kept), api/config.py, static/sessions.js, static/messages.js, CHANGELOG.md (upstream release sections byte-untouched).
• All 5 Copilot review items (Portability #1–fix(css): mobile responsive layout and dvh viewport fix #5) and the prior maintainer Q1–Q4 resolutions are preserved through the rebase.

Verification on the rebased tree

• Wakeup-critical + RCA + fingerprint suites: 85/85 pass.
• Full suite: 5961 passed; the only 3 failures are tests/test_issue1106_custom_providers_models.py — an upstream-owned test file none of these commits touch. Root-caused (4-phase) as the already-accepted model-resolve-hang budget tripping under this runner's slow DNS: re-running with the documented HERMES_WEBUI_MODELS_REBUILD_BUDGET=0 switch → 17/17 pass. Not a rebase regression; not investigated further per that prior decision.
• Live-instance self-wake verified on the rebased code (d6753e55): both the fast path (background process completing during turn-teardown) and the slow path (completing after teardown, idle-hook) trigger a server-initiated wakeup turn that renders live in an already-open tab with no refresh.

Ready for independent maintainer review of the sensitive streaming paths.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

[Isla-Liu]

If the rebase is clean, no maintainer intervention needed — just push the rebase + flip out of draft and we'll review.

Ping back if any of the Q1–Q4 decisions affect implementation enough that you need to revisit before rebasing.

@nesquena-hermes @nesquena
Hope for some idea or advice, thanks!

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.