From a6f008ec3896969a4e02c3bee4a0674eb565772c Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Tue, 28 Jul 2026 17:46:16 +0200 Subject: [PATCH 1/9] docs(backlog): investigate VRAM/lifecycle bug + durable-fix exploration TASK-1: stuck model loads (>1200s), unkillable abandoned workers holding device TASK-2: exploration of durable fixes (flush caches, CPU engine, timeout, shorter text) --- ...iVoice-VRAM-lifecycle-stuck-model-loads.md | 42 +++++++++++++++++++ ...rable-fixes-for-OmniVoice-VRAM-pressure.md | 35 ++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md create mode 100644 backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md diff --git a/backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md b/backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md new file mode 100644 index 000000000..23658ee7b --- /dev/null +++ b/backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md @@ -0,0 +1,42 @@ +--- +id: TASK-1 +title: Investigate OmniVoice VRAM/lifecycle bug (stuck model loads, unkillable abandoned jobs) +status: To Do +assignee: [] +created_date: '2026-07-28 17:00' +updated_date: '2026-07-28 17:00' +labels: + - omnivoice + - bug + - vram + - lifecycle + - mps +dependencies: [] +priority: high +ordinal: 1000 +--- + +## Description + +OmniVoice (MPS backend) is chronically VRAM-starved, causing a recurring lifecycle failure: + +1. A heavy model load exceeds the 1200s execution-time budget. +2. OmniVoice "abandons" the GPU-pool worker, but **cannot kill it**. The abandoned worker keeps running and **holds the MPS device**. +3. Every subsequent synth (REST `/v1/audio/speech`, `/generate`, and even the MCP `generate_speech`) queues behind the device-holding abandoned worker and hangs (60s-180s+ timeouts). + +Evidence from `~/Library/Application Support/OmniVoice/omnivoice.log` (read 2026-07-28): +- `Model load exceeded 1200.0s; resetting GPU pool` appears **3 times**: 2026-07-20 11:34, 2026-07-28 16:22, 2026-07-28 16:58 (recurred ~36 min apart the same day, i.e. a retry cascade: stuck, retry, still stuck because the device is held). +- `abandoned ... cannot be killed: it keeps running and keeps holding the device` appears **34 times**. +- VRAM / memory-pressure events: **40 times**. + +OmniVoice's own message references internal issues #730 / #1190. + +The MCP path (`mcp__omnivoice__generate_speech`) works when the device is free (succeeded 2026-07-28 12:17 @35s and 12:30 @13s), but hangs once the abandoned worker holds the device (16:22 onward). v0.4.2 (released 2026-07-28) did NOT introduce this per its changelog (update-UX + model-repair + localization), so it predates the release. + +## Acceptance Criteria + +- [ ] Root cause of the VRAM starvation: which resident model + which load contends, and why the load exceeds 1200s of compute. +- [ ] Root cause of the unkillable abandoned worker (the lifecycle bug behind #730/#1190): why OmniVoice cannot kill/clean up an abandoned GPU-pool worker. +- [ ] A reproducer (input/state that reliably triggers the stuck load). +- [ ] Fix so abandoned workers are actually killed (release the device) OR so loads cannot starve past the budget. +- [ ] Verify the fix prevents recurrence under sustained use (no stuck loads in a long-run test). diff --git a/backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md b/backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md new file mode 100644 index 000000000..d060571cf --- /dev/null +++ b/backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md @@ -0,0 +1,35 @@ +--- +id: TASK-2 +title: Exploration - durable fixes for OmniVoice VRAM pressure / synth reliability +status: To Do +assignee: [] +created_date: '2026-07-28 17:00' +updated_date: '2026-07-28 17:00' +labels: + - omnivoice + - exploration + - vram + - reliability +dependencies: + - TASK-1 +priority: medium +ordinal: 2000 +--- + +## Description + +While TASK-1 investigates the root lifecycle bug, evaluate these durable mitigations (from OmniVoice's own error message + the voice-on-reaction integration findings) so the TTS is reliable for unattended / reaction-triggered use. Document the tradeoff of each, then pick the one(s) to adopt. + +Candidates to explore: + +1. **Flush caches / Unload the resident model** (OmniVoice UI: Settings -> Models, or an API call) before a heavy/long synth, to free VRAM so the load does not starve. Question: can this be automated (API or script) so it runs before each long synth or on a schedule? +2. **Set the engine to CPU** (Settings -> Models) to remove MPS VRAM contention; stable but slower. Quantify the speed/quality tradeoff and whether it is acceptable for the voice-on-reaction use case. +3. **Raise `OMNIVOICE_GENERATE_TIMEOUT_S`** to tolerate long generations. Caveat: this does NOT fix the unkillable-worker hang (the device stays held regardless); evaluate whether it helps or only delays the failure. +4. **Shorter text per synth**: chunk long messages into multiple shorter synths to reduce per-call load. Evaluate the chunking strategy and how to concatenate the audio. +5. **Plugin-side PocketTTS fallback** (deployed 2026-07-28 in the hermes-agent `table_image_fallback` voice module, `_voice.py`): on the OmniVoice-trigger emoji, try OmniVoice; on timeout/failure, fall back to PocketTTS. Makes the voice-on-reaction feature robust regardless of OmniVoice's state (OmniVoice quality when healthy, PocketTTS speed as fallback). This complements, not replaces, TASK-1. + +## Acceptance Criteria + +- [ ] For each candidate: documented tradeoff (reliability gain vs cost/complexity/quality). +- [ ] Pick the durable fix(es) to adopt for unattended use; record the decision (and link TASK-1's root cause). +- [ ] If a candidate is automated (e.g. auto-flush before a long synth), implement and verify it prevents the stuck-load recurrence. From c2955dbe923d0face83d414249fec87ae735fc9a Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Tue, 28 Jul 2026 17:58:56 +0200 Subject: [PATCH 2/9] chore(backlog): initialize Backlog.md project structure --- backlog/config.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 backlog/config.yml diff --git a/backlog/config.yml b/backlog/config.yml new file mode 100644 index 000000000..720e5705e --- /dev/null +++ b/backlog/config.yml @@ -0,0 +1,16 @@ +project_name: "omnivoice-studio" +default_status: "To Do" +statuses: ["To Do", "In Progress", "Done"] +labels: [] +date_format: yyyy-mm-dd +max_column_width: 20 +default_editor: "vi" +auto_open_browser: false +default_port: 6420 +remote_operations: true +auto_commit: false +filesystem_only: false +bypass_git_hooks: false +check_active_branches: true +active_branch_days: 30 +task_prefix: "task" From 5229a9504c5bd2d8926199d1da9fe0f9603d9a5c Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Thu, 30 Jul 2026 16:10:36 +0200 Subject: [PATCH 3/9] chore: untrack local backlog/ task tracker (gitignored) --- .gitignore | 3 ++ backlog/config.yml | 16 ------- ...iVoice-VRAM-lifecycle-stuck-model-loads.md | 42 ------------------- ...rable-fixes-for-OmniVoice-VRAM-pressure.md | 35 ---------------- 4 files changed, 3 insertions(+), 93 deletions(-) delete mode 100644 backlog/config.yml delete mode 100644 backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md delete mode 100644 backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md diff --git a/.gitignore b/.gitignore index 1f1741210..62ef03843 100644 --- a/.gitignore +++ b/.gitignore @@ -150,3 +150,6 @@ playwright-report/ # probe — generated HTML reports tests/probe/reports/ + +# local Backlog.md task tracker; never tracked upstream +backlog/ diff --git a/backlog/config.yml b/backlog/config.yml deleted file mode 100644 index 720e5705e..000000000 --- a/backlog/config.yml +++ /dev/null @@ -1,16 +0,0 @@ -project_name: "omnivoice-studio" -default_status: "To Do" -statuses: ["To Do", "In Progress", "Done"] -labels: [] -date_format: yyyy-mm-dd -max_column_width: 20 -default_editor: "vi" -auto_open_browser: false -default_port: 6420 -remote_operations: true -auto_commit: false -filesystem_only: false -bypass_git_hooks: false -check_active_branches: true -active_branch_days: 30 -task_prefix: "task" diff --git a/backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md b/backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md deleted file mode 100644 index 23658ee7b..000000000 --- a/backlog/tasks/task-1 - Investigate-OmniVoice-VRAM-lifecycle-stuck-model-loads.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: TASK-1 -title: Investigate OmniVoice VRAM/lifecycle bug (stuck model loads, unkillable abandoned jobs) -status: To Do -assignee: [] -created_date: '2026-07-28 17:00' -updated_date: '2026-07-28 17:00' -labels: - - omnivoice - - bug - - vram - - lifecycle - - mps -dependencies: [] -priority: high -ordinal: 1000 ---- - -## Description - -OmniVoice (MPS backend) is chronically VRAM-starved, causing a recurring lifecycle failure: - -1. A heavy model load exceeds the 1200s execution-time budget. -2. OmniVoice "abandons" the GPU-pool worker, but **cannot kill it**. The abandoned worker keeps running and **holds the MPS device**. -3. Every subsequent synth (REST `/v1/audio/speech`, `/generate`, and even the MCP `generate_speech`) queues behind the device-holding abandoned worker and hangs (60s-180s+ timeouts). - -Evidence from `~/Library/Application Support/OmniVoice/omnivoice.log` (read 2026-07-28): -- `Model load exceeded 1200.0s; resetting GPU pool` appears **3 times**: 2026-07-20 11:34, 2026-07-28 16:22, 2026-07-28 16:58 (recurred ~36 min apart the same day, i.e. a retry cascade: stuck, retry, still stuck because the device is held). -- `abandoned ... cannot be killed: it keeps running and keeps holding the device` appears **34 times**. -- VRAM / memory-pressure events: **40 times**. - -OmniVoice's own message references internal issues #730 / #1190. - -The MCP path (`mcp__omnivoice__generate_speech`) works when the device is free (succeeded 2026-07-28 12:17 @35s and 12:30 @13s), but hangs once the abandoned worker holds the device (16:22 onward). v0.4.2 (released 2026-07-28) did NOT introduce this per its changelog (update-UX + model-repair + localization), so it predates the release. - -## Acceptance Criteria - -- [ ] Root cause of the VRAM starvation: which resident model + which load contends, and why the load exceeds 1200s of compute. -- [ ] Root cause of the unkillable abandoned worker (the lifecycle bug behind #730/#1190): why OmniVoice cannot kill/clean up an abandoned GPU-pool worker. -- [ ] A reproducer (input/state that reliably triggers the stuck load). -- [ ] Fix so abandoned workers are actually killed (release the device) OR so loads cannot starve past the budget. -- [ ] Verify the fix prevents recurrence under sustained use (no stuck loads in a long-run test). diff --git a/backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md b/backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md deleted file mode 100644 index d060571cf..000000000 --- a/backlog/tasks/task-2 - Exploration-durable-fixes-for-OmniVoice-VRAM-pressure.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -id: TASK-2 -title: Exploration - durable fixes for OmniVoice VRAM pressure / synth reliability -status: To Do -assignee: [] -created_date: '2026-07-28 17:00' -updated_date: '2026-07-28 17:00' -labels: - - omnivoice - - exploration - - vram - - reliability -dependencies: - - TASK-1 -priority: medium -ordinal: 2000 ---- - -## Description - -While TASK-1 investigates the root lifecycle bug, evaluate these durable mitigations (from OmniVoice's own error message + the voice-on-reaction integration findings) so the TTS is reliable for unattended / reaction-triggered use. Document the tradeoff of each, then pick the one(s) to adopt. - -Candidates to explore: - -1. **Flush caches / Unload the resident model** (OmniVoice UI: Settings -> Models, or an API call) before a heavy/long synth, to free VRAM so the load does not starve. Question: can this be automated (API or script) so it runs before each long synth or on a schedule? -2. **Set the engine to CPU** (Settings -> Models) to remove MPS VRAM contention; stable but slower. Quantify the speed/quality tradeoff and whether it is acceptable for the voice-on-reaction use case. -3. **Raise `OMNIVOICE_GENERATE_TIMEOUT_S`** to tolerate long generations. Caveat: this does NOT fix the unkillable-worker hang (the device stays held regardless); evaluate whether it helps or only delays the failure. -4. **Shorter text per synth**: chunk long messages into multiple shorter synths to reduce per-call load. Evaluate the chunking strategy and how to concatenate the audio. -5. **Plugin-side PocketTTS fallback** (deployed 2026-07-28 in the hermes-agent `table_image_fallback` voice module, `_voice.py`): on the OmniVoice-trigger emoji, try OmniVoice; on timeout/failure, fall back to PocketTTS. Makes the voice-on-reaction feature robust regardless of OmniVoice's state (OmniVoice quality when healthy, PocketTTS speed as fallback). This complements, not replaces, TASK-1. - -## Acceptance Criteria - -- [ ] For each candidate: documented tradeoff (reliability gain vs cost/complexity/quality). -- [ ] Pick the durable fix(es) to adopt for unattended use; record the decision (and link TASK-1's root cause). -- [ ] If a candidate is automated (e.g. auto-flush before a long synth), implement and verify it prevents the stuck-load recurrence. From 9615cd5294e29ff33911988d0699151ef7a8a821 Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Sat, 15 Aug 2026 13:49:47 +0200 Subject: [PATCH 4/9] =?UTF-8?q?fix(events):=20sync=20endpoints=20dropped?= =?UTF-8?q?=20their=20WS=20events=20=E2=80=94=20rename/delete=20left=20eve?= =?UTF-8?q?ry=20open=20tab=20stale?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PUT/DELETE /profiles (rename, delete, revoke consent) and the history/export mutators are sync FastAPI endpoints: their bodies run in threadpool workers where asyncio.get_running_loop() raises, so event_bus.emit() hit the RuntimeError branch and silently dropped the "profiles" event. The UI only refetches the voice list on that event, so after a rename the list kept stale state, and a reload during that window could land on an empty panel (no retry on the initial load either) — which reads to a user as "all my voices are gone" even though nothing was deleted. emit() now captures the serving loop in subscribe() and hands off from foreign threads via call_soon_threadsafe (async callers are unchanged). Also: the initial list loads in useAppData retry until FIRST success via retryInitialLoad — a WS-triggered reload failure still keeps the previous list, but the first load has nothing to keep. Loaders gained {rethrow: true} for the initial path so the retry actually engages (they swallow errors by design elsewhere); an integration test pins that wiring. Tests: tests/test_event_bus_thread_emit.py fails on the old emit (verified by stashing the fix) and passes with it; a live two-instance probe confirmed PUT rename → WS event arrives on the fixed build and never on the original. --- CHANGELOG.md | 1 + backend/core/event_bus.py | 48 ++++++++++--- frontend/src/hooks/useAppData.js | 81 ++++++++++------------ frontend/src/test/initialLoadRetry.test.js | 67 ++++++++++++++++++ frontend/src/utils/initialLoadRetry.js | 29 ++++++++ tests/test_event_bus_thread_emit.py | 81 ++++++++++++++++++++++ 6 files changed, 252 insertions(+), 55 deletions(-) create mode 100644 frontend/src/test/initialLoadRetry.test.js create mode 100644 frontend/src/utils/initialLoadRetry.js create mode 100644 tests/test_event_bus_thread_emit.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 83497edc8..aa11eddcd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ the frozen-backend fallback mirror it for their toolchains. ### Fixed +- Renaming, deleting, or revoking consent on a voice (and starring/clearing history, recording exports) now live-updates every open tab again: those sync API routes run in worker threads where the WebSocket event was silently dropped, so the UI kept stale lists until a reload — which could look like "all my voices are gone". Events are now handed off to the serving loop thread-safely, and the initial list load retries until first success instead of leaving an empty panel on one transient failure. - Crash reports now carry the crashed run's own stderr: the shared error log is append-only with per-run offsets, so a restart can no longer overwrite the dying process's final output with the replacement's healthy startup. (#1510) - Wayland: a stale portal identity no longer kills the dictation shortcut for the whole session. The desktop entry the app writes for the GlobalShortcuts portal could point at a binary that has since moved (a `cargo clean`, a relocated AppImage) — GNOME then refuses the bind with "App info not found" and the hotkey silently dies. The entry is validated and rewritten at startup now. (#1526) - The guard that keeps transcription on the degrading ASR loader now scans the whole backend, not just the routers — a service that transcribes on a request's behalf skipped `ensure_loaded()` just as thoroughly. (#1519) — thanks @ahov520! diff --git a/backend/core/event_bus.py b/backend/core/event_bus.py index baa3b995c..74fd662ce 100644 --- a/backend/core/event_bus.py +++ b/backend/core/event_bus.py @@ -23,9 +23,17 @@ _listeners: list[asyncio.Queue] = [] _lock = asyncio.Lock() +# The loop that serves /ws/events, captured on first use. Sync FastAPI +# endpoints (rename/delete profile, revoke consent) run in threadpool workers +# where `asyncio.get_running_loop()` raises, which used to silently drop their +# events — the UI then never refetched the voice list (#1158 class). +_serving_loop: asyncio.AbstractEventLoop | None = None + async def subscribe() -> asyncio.Queue: """Register a new listener. Returns a Queue that receives event dicts.""" + global _serving_loop + _serving_loop = asyncio.get_running_loop() q: asyncio.Queue = asyncio.Queue(maxsize=64) async with _lock: _listeners.append(q) @@ -56,12 +64,34 @@ def emit(kind: str, payload: dict[str, Any] | None = None) -> None: **(payload or {}), } event_str = json.dumps(event) + loop = asyncio.get_running_loop() if _on_event_loop() else _serving_loop + if loop is None: + # No serving loop yet — nobody to notify; dropping is correct. + logger.debug("No event loop — event dropped: %s", kind) + return try: - loop = asyncio.get_running_loop() - loop.create_task(_broadcast(event_str)) + if _on_event_loop(): + loop.create_task(_broadcast(event_str)) + else: + # Threadpool worker (sync endpoint body): the only thread-safe way in. + loop.call_soon_threadsafe(_schedule_broadcast, event_str) except RuntimeError: - # No event loop running (unlikely in FastAPI context but safe) - logger.debug("No event loop — event dropped: %s", kind) + # The serving loop closed between capture and use (app shutdown). + logger.debug("Event loop closed — event dropped: %s", kind) + + +def _on_event_loop() -> bool: + """True when called from the running event loop (async-context emit).""" + try: + asyncio.get_running_loop() + except RuntimeError: + return False + return True + + +def _schedule_broadcast(event_str: str) -> None: + """Run `_broadcast` on the serving loop; called via call_soon_threadsafe.""" + asyncio.get_running_loop().create_task(_broadcast(event_str)) async def _broadcast(event_str: str) -> None: @@ -73,11 +103,11 @@ async def _broadcast(event_str: str) -> None: q.put_nowait(event_str) except asyncio.QueueFull: # Slow consumer — drop oldest, then push. Not a race (#1163): - # every queue op runs on the single event loop, and there is - # no await between the QueueFull and this get_nowait/put_nowait - # pair — no consumer can interleave, so get_nowait cannot raise - # QueueEmpty here. emit() from a foreign thread drops the event - # before ever touching a queue (see the RuntimeError branch). + # every queue op runs on the single event loop (a foreign + # thread's emit() hands off via call_soon_threadsafe first), + # and there is no await between the QueueFull and this + # get_nowait/put_nowait pair — no consumer can interleave, so + # get_nowait cannot raise QueueEmpty here. try: q.get_nowait() q.put_nowait(event_str) diff --git a/frontend/src/hooks/useAppData.js b/frontend/src/hooks/useAppData.js index 66fe47917..58f225ffd 100644 --- a/frontend/src/hooks/useAppData.js +++ b/frontend/src/hooks/useAppData.js @@ -10,6 +10,7 @@ import { useModelStatus } from '../api/hooks'; import useRealtimeEvents from './useRealtimeEvents'; import { mergeDescribedAttrs } from '../utils/voiceInstruct'; import { sanitizeOmniUi } from '../utils/omniUiSchema'; +import { retryInitialLoad } from '../utils/initialLoadRetry'; /** * Encapsulates all data-loading effects, localStorage persistence, @@ -122,43 +123,27 @@ export default function useAppData() { }, [modelStatus, modelSubStage, modelDetail, modelError, modelProgress]); // ── Data loading callbacks ── - // Failures keep the previous list (better than blanking the UI), but are - // logged so "my voices/history vanished" reports carry a cause (#1158). - const loadProfiles = useCallback(async () => { - try { - setProfiles(await listProfiles()); - } catch (e) { - console.warn('Failed to load voice profiles:', e); - } - }, []); - const loadHistory = useCallback(async () => { - try { - setHistory(await listHistory()); - } catch (e) { - console.warn('Failed to load generation history:', e); - } - }, []); - const loadDubHistory = useCallback(async () => { - try { - setDubHistory(await listDubHistory()); - } catch (e) { - console.warn('Failed to load dub history:', e); - } - }, []); - const loadProjects = useCallback(async () => { - try { - setStudioProjects(await listProjects()); - } catch (e) { - console.warn('Failed to load projects:', e); - } - }, []); - const loadExportHistory = useCallback(async () => { - try { - setExportHistory(await listExportHistory()); - } catch (e) { - console.warn('Failed to load export history:', e); - } - }, []); + // WS-triggered reloads swallow failures: keeping the previous list is + // better than blanking the UI, and the warn gives "my voices vanished" + // reports a cause (#1158). The INITIAL load passes `{ rethrow: true }` so + // retryInitialLoad can retry — there is no previous list to keep yet. + const makeLoader = (fetch, set, label) => + useCallback( + async ({ rethrow } = {}) => { + try { + set(await fetch()); + } catch (e) { + console.warn(`Failed to load ${label}:`, e); + if (rethrow) throw e; + } + }, + [], + ); + const loadProfiles = makeLoader(listProfiles, setProfiles, 'voice profiles'); + const loadHistory = makeLoader(listHistory, setHistory, 'generation history'); + const loadDubHistory = makeLoader(listDubHistory, setDubHistory, 'dub history'); + const loadProjects = makeLoader(listProjects, setStudioProjects, 'projects'); + const loadExportHistory = makeLoader(listExportHistory, setExportHistory, 'export history'); // ── WebSocket real-time updates ── useRealtimeEvents({ @@ -171,10 +156,10 @@ export default function useAppData() { // ── Initial data load with backend retry ── useEffect(() => { - let cancelled = false; + const cancelledRef = { cancelled: false }; const loadAll = async () => { let delay = 1000; - while (!cancelled) { + while (!cancelledRef.cancelled) { try { await apiModelStatus(); break; @@ -182,12 +167,16 @@ export default function useAppData() { await new Promise((r) => setTimeout(r, delay)); delay = Math.min(delay * 2, 4000); } - if (cancelled) return; - loadProfiles(); - loadHistory(); - loadDubHistory(); - loadProjects(); - loadExportHistory(); + if (cancelledRef.cancelled) return; + // Initial loads retry until FIRST success (#1158 class): a later + // (WS-triggered) reload failure keeps the previous list, but the first + // load has no previous list to keep — one transient failure used to + // leave the panel empty, which read as "my voices are gone". + retryInitialLoad(() => loadProfiles({ rethrow: true }), cancelledRef); + retryInitialLoad(() => loadHistory({ rethrow: true }), cancelledRef); + retryInitialLoad(() => loadDubHistory({ rethrow: true }), cancelledRef); + retryInitialLoad(() => loadProjects({ rethrow: true }), cancelledRef); + retryInitialLoad(() => loadExportHistory({ rethrow: true }), cancelledRef); }; loadAll(); // Restore local UI state @@ -242,7 +231,7 @@ export default function useAppData() { if (saved.showOverrides !== undefined) setShowOverrides(saved.showOverrides); } catch (e) {} return () => { - cancelled = true; + cancelledRef.cancelled = true; }; }, []); diff --git a/frontend/src/test/initialLoadRetry.test.js b/frontend/src/test/initialLoadRetry.test.js new file mode 100644 index 000000000..3cd0dd637 --- /dev/null +++ b/frontend/src/test/initialLoadRetry.test.js @@ -0,0 +1,67 @@ +import { describe, it, expect, vi } from 'vitest'; +import { retryInitialLoad } from '../utils/initialLoadRetry'; + +// #1158 class: the initial data loads (profiles/history/…) ran exactly once +// after the backend became reachable. A transient failure on that single call +// (backend restarting, WS reconnect window, a 502 from the LAN gate) left the +// voices panel empty with no retry — which reads to users as "all my voices +// are gone". The load itself already keeps the previous list on later (WS) +// reloads; only the INITIAL load must retry until first success. + +describe('retryInitialLoad (#1158 class)', () => { + it('retries a failing loader until it succeeds', async () => { + let calls = 0; + const loader = vi.fn(async () => { + calls += 1; + if (calls < 3) throw new Error('transient'); + }); + await retryInitialLoad(loader, { baseDelayMs: 1 }); + expect(calls).toBe(3); + }); + + it('does not retry after first success', async () => { + const loader = vi.fn(async () => {}); + await retryInitialLoad(loader, { baseDelayMs: 1 }); + expect(loader).toHaveBeenCalledTimes(1); + }); + + it('stops retrying when cancelled', async () => { + const loader = vi.fn(async () => { + throw new Error('down'); + }); + const opts = { baseDelayMs: 1 }; + const p = retryInitialLoad(loader, opts); + opts.cancelled = true; + await p; + // At most one attempt plus whatever was already in flight; the point is + // it terminates instead of looping forever after cancellation. + expect(loader.mock.calls.length).toBeLessThan(5); + }); + + // #1158 wiring contract: the initial load must pass loaders that REJECT on + // failure. useAppData's loadProfiles-style loaders swallow errors by design + // (WS reloads keep the previous list); if the initial-load call site ever + // stops using { rethrow: true }, the retry helper resolves on attempt one + // and the fix is silently inert again (skeptic finding F1). + it('integration: useAppData wires rethrowing loaders into the retry', async () => { + const { readFileSync } = await import('node:fs'); + // jsdom replaces the global URL; Node's fileURLToPath needs a REAL node: + // URL instance, so build one from the specifier's pathname directly. + const nodeUrl = await import('node:url'); + const NodeURL = nodeUrl.URL; + const hookPath = new NodeURL('../hooks/useAppData.js', import.meta.url).pathname; + const src = readFileSync(hookPath, 'utf8'); + const initialBlock = src.slice(src.indexOf('retryInitialLoad(')); + for (const loader of [ + 'loadProfiles', + 'loadHistory', + 'loadDubHistory', + 'loadProjects', + 'loadExportHistory', + ]) { + expect(initialBlock).toContain(`${loader}({ rethrow: true })`); + } + // and the loader factory must actually rethrow when asked + expect(src).toContain('if (rethrow) throw e;'); + }); +}); diff --git a/frontend/src/utils/initialLoadRetry.js b/frontend/src/utils/initialLoadRetry.js new file mode 100644 index 000000000..a300eb1c0 --- /dev/null +++ b/frontend/src/utils/initialLoadRetry.js @@ -0,0 +1,29 @@ +/** + * retryInitialLoad — run an initial data load until first success. + * + * The INITIAL app load differs from later (WS-triggered) reloads: on reload a + * failure keeps the previous list, but on first load there IS no previous + * list, so a single transient failure left the panel empty forever and read + * to users as "my voices are gone" (#1158 class). Retry with backoff until + * the first success; afterwards the caller's normal keep-previous-list + * behavior takes over. + * + * `opts.cancelled` is the caller's mount-cancellation flag — set it to true + * and any pending wait resolves without another attempt. + */ +export async function retryInitialLoad(loader, opts = {}) { + const baseDelayMs = opts.baseDelayMs ?? 500; + const maxDelayMs = opts.maxDelayMs ?? 4000; + let delay = 0; // first attempt is immediate; only failures pay the backoff + for (;;) { + if (opts.cancelled) return; + try { + await loader(); + return; + } catch { + // transient — retry + } + await new Promise((r) => setTimeout(r, delay || baseDelayMs)); + delay = Math.min((delay || baseDelayMs) * 2, maxDelayMs); + } +} diff --git a/tests/test_event_bus_thread_emit.py b/tests/test_event_bus_thread_emit.py new file mode 100644 index 000000000..3d0765181 --- /dev/null +++ b/tests/test_event_bus_thread_emit.py @@ -0,0 +1,81 @@ +"""event_bus.emit must deliver from threadpool threads (sync FastAPI endpoints). + +PUT /profiles/{id} (rename), DELETE /profiles/{id}, and DELETE .../consent are +sync endpoints, so Starlette runs their bodies in a threadpool worker with no +running event loop. event_bus.emit() used to call asyncio.get_running_loop() +and silently drop the event (DEBUG log only) from exactly those threads, which +reached users as "I renamed a voice and the list went stale / looked empty" +(the frontend only refetches the voice list on the WS "profiles" event). + +The test drives the real failure shape: subscribe on the serving loop, call +emit() from a plain thread (as the threadpool does), and assert the event +arrives in the listener queue. +""" + +from __future__ import annotations + +import asyncio +import json +import threading + +import pytest + + +@pytest.fixture +def bus(): + """Resolve the module per test — binding it at collection lets another + suite's `sys.modules` rebinding make this exercise a different object.""" + return __import__("core.event_bus", fromlist=["emit"]) + + +def test_emit_from_thread_reaches_serving_loop(bus, tmp_path): + loop = asyncio.new_event_loop() + received: list[str] = [] + started = threading.Event() + done = threading.Event() + + def run_loop(): + asyncio.set_event_loop(loop) + loop.run_until_complete(_serve(bus, received, started, done)) + + t = threading.Thread(target=run_loop, name="test-serving-loop") + t.start() + try: + assert started.wait(2.0), "serving loop never subscribed" + + # The bug's exact context: no running loop in this thread, like a + # Starlette threadpool worker executing a sync endpoint body. + def sync_endpoint_body(): + bus.emit("profiles", {"action": "updated", "id": "abc123"}) + + worker = threading.Thread(target=sync_endpoint_body, name="threadpool-worker") + worker.start() + worker.join(2.0) + + assert done.wait(2.0), ( + "emit() from a threadpool thread was dropped — the WS 'profiles' " + "event never reached the serving loop's listener queue" + ) + payload = json.loads(received[0]) + assert payload["kind"] == "profiles" + assert payload["action"] == "updated" + assert payload["id"] == "abc123" + finally: + loop.call_soon_threadsafe(done.set) + t.join(2.0) + loop.close() + + +async def _serve(bus, received: list[str], started: threading.Event, done: threading.Event): + q = await bus.subscribe() + started.set() + # Poll the queue (not `done`) — `done` is what the main thread waits on. + deadline = asyncio.get_event_loop().time() + 2.0 + while asyncio.get_event_loop().time() < deadline: + try: + received.append(q.get_nowait()) + break + except asyncio.QueueEmpty: + await asyncio.sleep(0.01) + done.set() + await bus.unsubscribe(q) From dcaed7cbf4cdf3b4cc46823b8e2beeb403867cf4 Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Sat, 15 Aug 2026 13:54:13 +0200 Subject: [PATCH 5/9] docs(changelog): one-line Unreleased entry with issue ref + credit (Greptile P2) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa11eddcd..293562273 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,7 +34,7 @@ the frozen-backend fallback mirror it for their toolchains. ### Fixed -- Renaming, deleting, or revoking consent on a voice (and starring/clearing history, recording exports) now live-updates every open tab again: those sync API routes run in worker threads where the WebSocket event was silently dropped, so the UI kept stale lists until a reload — which could look like "all my voices are gone". Events are now handed off to the serving loop thread-safely, and the initial list load retries until first success instead of leaving an empty panel on one transient failure. +- Renaming, deleting, or revoking consent on a voice (and starring/clearing history, recording exports) now live-updates every open tab again — the sync routes' WebSocket events were silently dropped, which could look like "all my voices are gone" (#1561) — thanks @paoloantinori! - Crash reports now carry the crashed run's own stderr: the shared error log is append-only with per-run offsets, so a restart can no longer overwrite the dying process's final output with the replacement's healthy startup. (#1510) - Wayland: a stale portal identity no longer kills the dictation shortcut for the whole session. The desktop entry the app writes for the GlobalShortcuts portal could point at a binary that has since moved (a `cargo clean`, a relocated AppImage) — GNOME then refuses the bind with "App info not found" and the hotkey silently dies. The entry is validated and rewritten at startup now. (#1526) - The guard that keeps transcription on the degrading ASR loader now scans the whole backend, not just the routers — a service that transcribes on a request's behalf skipped `ensure_loaded()` just as thoroughly. (#1519) — thanks @ahov520! From fd30a6c4ad55e11f0384d02c71e938c5fea53a63 Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Sat, 15 Aug 2026 14:10:38 +0200 Subject: [PATCH 6/9] fix(review): last-write-wins loaders + no sleep-polling in the emit test CodeRabbit #1562 findings, both real: - makeLoader is now generation-guarded: the initial retry loop overlaps freely with WS-triggered reloads, and a slow in-flight response could resolve AFTER a fresher reload and overwrite its list with stale data. Each invocation bumps a generation; only the newest may setState. - The regression test awaited the queue via sleep-polling; it now uses asyncio.wait_for(q.get()) so a failure surfaces as TimeoutError instead of depending on 10ms poll timing (repo rule: no sleeps as sync). --- frontend/src/hooks/useAppData.js | 13 ++++++++++--- tests/test_event_bus_thread_emit.py | 17 +++++++---------- 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/frontend/src/hooks/useAppData.js b/frontend/src/hooks/useAppData.js index 58f225ffd..c38ed8582 100644 --- a/frontend/src/hooks/useAppData.js +++ b/frontend/src/hooks/useAppData.js @@ -127,11 +127,17 @@ export default function useAppData() { // better than blanking the UI, and the warn gives "my voices vanished" // reports a cause (#1158). The INITIAL load passes `{ rethrow: true }` so // retryInitialLoad can retry — there is no previous list to keep yet. - const makeLoader = (fetch, set, label) => - useCallback( + // Each loader is last-write-wins by invocation order: a slow in-flight + // request (the initial retry loop overlaps freely with WS reloads) must + // not overwrite the fresher list a later reload already applied. + const makeLoader = (fetch, set, label) => { + const genRef = { current: 0 }; + return useCallback( async ({ rethrow } = {}) => { + const gen = ++genRef.current; try { - set(await fetch()); + const data = await fetch(); + if (gen === genRef.current) set(data); } catch (e) { console.warn(`Failed to load ${label}:`, e); if (rethrow) throw e; @@ -139,6 +145,7 @@ export default function useAppData() { }, [], ); + }; const loadProfiles = makeLoader(listProfiles, setProfiles, 'voice profiles'); const loadHistory = makeLoader(listHistory, setHistory, 'generation history'); const loadDubHistory = makeLoader(listDubHistory, setDubHistory, 'dub history'); diff --git a/tests/test_event_bus_thread_emit.py b/tests/test_event_bus_thread_emit.py index 3d0765181..49b9960ce 100644 --- a/tests/test_event_bus_thread_emit.py +++ b/tests/test_event_bus_thread_emit.py @@ -69,13 +69,10 @@ def sync_endpoint_body(): async def _serve(bus, received: list[str], started: threading.Event, done: threading.Event): q = await bus.subscribe() started.set() - # Poll the queue (not `done`) — `done` is what the main thread waits on. - deadline = asyncio.get_event_loop().time() + 2.0 - while asyncio.get_event_loop().time() < deadline: - try: - received.append(q.get_nowait()) - break - except asyncio.QueueEmpty: - await asyncio.sleep(0.01) - done.set() - await bus.unsubscribe(q) + # Await the event itself (no sleep-polling): a failure surfaces as + # asyncio.TimeoutError, which fails the test with a clear traceback. + try: + received.append(await asyncio.wait_for(q.get(), 2.0)) + finally: + done.set() + await bus.unsubscribe(q) From 030bc4751552d9aaba2124c1a32bc9e4ae3b9eab Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Sat, 15 Aug 2026 15:08:20 +0200 Subject: [PATCH 7/9] =?UTF-8?q?fix(lint):=20makeLoader=20can't=20call=20us?= =?UTF-8?q?eCallback=20(rules-of-hooks)=20=E2=80=94=20generation=20state?= =?UTF-8?q?=20in=20a=20ref=20instead?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- frontend/src/hooks/useAppData.js | 41 +++++++++++++++----------------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/frontend/src/hooks/useAppData.js b/frontend/src/hooks/useAppData.js index 0fd707283..5c56caac6 100644 --- a/frontend/src/hooks/useAppData.js +++ b/frontend/src/hooks/useAppData.js @@ -1,4 +1,4 @@ -import { useState, useEffect, useLayoutEffect, useCallback, useRef } from 'react'; +import { useState, useEffect, useLayoutEffect, useRef } from 'react'; import { useAppStore } from '../store'; import { listProfiles } from '../api/profiles'; import { listHistory } from '../api/generate'; @@ -170,28 +170,25 @@ export default function useAppData() { // retryInitialLoad can retry — there is no previous list to keep yet. // Each loader is last-write-wins by invocation order: a slow in-flight // request (the initial retry loop overlaps freely with WS reloads) must - // not overwrite the fresher list a later reload already applied. - const makeLoader = (fetch, set, label) => { - const genRef = { current: 0 }; - return useCallback( - async ({ rethrow } = {}) => { - const gen = ++genRef.current; - try { - const data = await fetch(); - if (gen === genRef.current) set(data); - } catch (e) { - console.warn(`Failed to load ${label}:`, e); - if (rethrow) throw e; - } - }, - [], - ); + // not overwrite the fresher list a later reload already applied. Plain + // per-render closures over stable imports/setters — a useRef-free module + // would need hooks inside a helper, which rules-of-hooks forbids. + const loadersRef = useRef({ profiles: 0, history: 0, dub: 0, projects: 0, exports: 0 }); + const makeLoader = (key, fetch, set, label) => async ({ rethrow } = {}) => { + const gen = ++loadersRef.current[key]; + try { + const data = await fetch(); + if (gen === loadersRef.current[key]) set(data); + } catch (e) { + console.warn(`Failed to load ${label}:`, e); + if (rethrow) throw e; + } }; - const loadProfiles = makeLoader(listProfiles, setProfiles, 'voice profiles'); - const loadHistory = makeLoader(listHistory, setHistory, 'generation history'); - const loadDubHistory = makeLoader(listDubHistory, setDubHistory, 'dub history'); - const loadProjects = makeLoader(listProjects, setStudioProjects, 'projects'); - const loadExportHistory = makeLoader(listExportHistory, setExportHistory, 'export history'); + const loadProfiles = makeLoader('profiles', listProfiles, setProfiles, 'voice profiles'); + const loadHistory = makeLoader('history', listHistory, setHistory, 'generation history'); + const loadDubHistory = makeLoader('dub', listDubHistory, setDubHistory, 'dub history'); + const loadProjects = makeLoader('projects', listProjects, setStudioProjects, 'projects'); + const loadExportHistory = makeLoader('exports', listExportHistory, setExportHistory, 'export history'); // ── WebSocket real-time updates ── useRealtimeEvents({ From be007e9d774b0a123d91d315b31cacedd6dbcbab Mon Sep 17 00:00:00 2001 From: Paolo Antinori Date: Sat, 15 Aug 2026 15:19:49 +0200 Subject: [PATCH 8/9] style(frontend): oxfmt useAppData (CI format check) --- frontend/src/hooks/useAppData.js | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/frontend/src/hooks/useAppData.js b/frontend/src/hooks/useAppData.js index 5c56caac6..f4fe496dc 100644 --- a/frontend/src/hooks/useAppData.js +++ b/frontend/src/hooks/useAppData.js @@ -174,21 +174,28 @@ export default function useAppData() { // per-render closures over stable imports/setters — a useRef-free module // would need hooks inside a helper, which rules-of-hooks forbids. const loadersRef = useRef({ profiles: 0, history: 0, dub: 0, projects: 0, exports: 0 }); - const makeLoader = (key, fetch, set, label) => async ({ rethrow } = {}) => { - const gen = ++loadersRef.current[key]; - try { - const data = await fetch(); - if (gen === loadersRef.current[key]) set(data); - } catch (e) { - console.warn(`Failed to load ${label}:`, e); - if (rethrow) throw e; - } - }; + const makeLoader = + (key, fetch, set, label) => + async ({ rethrow } = {}) => { + const gen = ++loadersRef.current[key]; + try { + const data = await fetch(); + if (gen === loadersRef.current[key]) set(data); + } catch (e) { + console.warn(`Failed to load ${label}:`, e); + if (rethrow) throw e; + } + }; const loadProfiles = makeLoader('profiles', listProfiles, setProfiles, 'voice profiles'); const loadHistory = makeLoader('history', listHistory, setHistory, 'generation history'); const loadDubHistory = makeLoader('dub', listDubHistory, setDubHistory, 'dub history'); const loadProjects = makeLoader('projects', listProjects, setStudioProjects, 'projects'); - const loadExportHistory = makeLoader('exports', listExportHistory, setExportHistory, 'export history'); + const loadExportHistory = makeLoader( + 'exports', + listExportHistory, + setExportHistory, + 'export history', + ); // ── WebSocket real-time updates ── useRealtimeEvents({ From afe013a6bccd418e382b677f009129d0f78543ce Mon Sep 17 00:00:00 2001 From: debpalash <4178343+debpalash@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:47:45 +0530 Subject: [PATCH 9/9] fix(events): dispatch foreign loops through serving loop --- backend/core/event_bus.py | 26 ++++++--------- tests/test_event_bus_thread_emit.py | 52 +++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 15 deletions(-) diff --git a/backend/core/event_bus.py b/backend/core/event_bus.py index 74fd662ce..a9242ed36 100644 --- a/backend/core/event_bus.py +++ b/backend/core/event_bus.py @@ -64,31 +64,27 @@ def emit(kind: str, payload: dict[str, Any] | None = None) -> None: **(payload or {}), } event_str = json.dumps(event) - loop = asyncio.get_running_loop() if _on_event_loop() else _serving_loop - if loop is None: + try: + caller_loop = asyncio.get_running_loop() + except RuntimeError: + caller_loop = None + target_loop = _serving_loop or caller_loop + if target_loop is None: # No serving loop yet — nobody to notify; dropping is correct. logger.debug("No event loop — event dropped: %s", kind) return try: - if _on_event_loop(): - loop.create_task(_broadcast(event_str)) + if caller_loop is target_loop: + target_loop.create_task(_broadcast(event_str)) else: - # Threadpool worker (sync endpoint body): the only thread-safe way in. - loop.call_soon_threadsafe(_schedule_broadcast, event_str) + # Sync endpoints and async producers on a foreign loop must both + # hand off: the lock and listener queues belong to serving_loop. + target_loop.call_soon_threadsafe(_schedule_broadcast, event_str) except RuntimeError: # The serving loop closed between capture and use (app shutdown). logger.debug("Event loop closed — event dropped: %s", kind) -def _on_event_loop() -> bool: - """True when called from the running event loop (async-context emit).""" - try: - asyncio.get_running_loop() - except RuntimeError: - return False - return True - - def _schedule_broadcast(event_str: str) -> None: """Run `_broadcast` on the serving loop; called via call_soon_threadsafe.""" asyncio.get_running_loop().create_task(_broadcast(event_str)) diff --git a/tests/test_event_bus_thread_emit.py b/tests/test_event_bus_thread_emit.py index 49b9960ce..16002fc30 100644 --- a/tests/test_event_bus_thread_emit.py +++ b/tests/test_event_bus_thread_emit.py @@ -66,6 +66,58 @@ def sync_endpoint_body(): loop.close() +def test_emit_from_foreign_running_loop_reaches_serving_loop(bus): + """An async producer may run on a worker loop, but listener state belongs + to the WebSocket serving loop and must only be touched there.""" + serving_loop = asyncio.new_event_loop() + serving_loop.set_debug(True) + received: list[str] = [] + started = threading.Event() + done = threading.Event() + + async def serve_with_waiter_ready(): + q = await bus.subscribe() + waiter = asyncio.create_task(q.get()) + await asyncio.sleep(0) # q.get() has installed its serving-loop Future + started.set() + try: + received.append(await asyncio.wait_for(waiter, 0.5)) + except asyncio.TimeoutError: + pass + finally: + done.set() + await bus.unsubscribe(q) + + def run_serving_loop(): + asyncio.set_event_loop(serving_loop) + serving_loop.run_until_complete(serve_with_waiter_ready()) + + serving_thread = threading.Thread( + target=run_serving_loop, name="test-serving-loop" + ) + serving_thread.start() + try: + assert started.wait(2.0), "serving loop never subscribed" + + async def foreign_async_caller(): + assert asyncio.get_running_loop() is not serving_loop + bus.emit("profiles", {"action": "updated", "id": "foreign-loop"}) + await asyncio.sleep(0.05) + + asyncio.run(foreign_async_caller()) + + assert done.wait(2.0), ( + "emit() ran listener delivery on the caller's foreign loop" + ) + assert received, "foreign-loop event never reached the serving loop" + payload = json.loads(received[0]) + assert payload["id"] == "foreign-loop" + finally: + serving_loop.call_soon_threadsafe(done.set) + serving_thread.join(2.0) + serving_loop.close() + + async def _serve(bus, received: list[str], started: threading.Event, done: threading.Event): q = await bus.subscribe() started.set()