perf(watermark): background-prefetch the AudioSeal generator at startup

CHANGELOG.md

@@ -18,6 +18,7 @@ the frozen-backend fallback mirror it for their toolchains.
 - The backend binds its port immediately and reports startup progress live — `/health` answers 503-with-step and a new `/startup/progress` endpoint lists every step while PyTorch, API routes, and database migrations load in the background, so "starting at step X" is never mistakable for "dead"; the desktop splash narrates each step (#1550)
 
 ### Added
+- The locally cached AudioSeal watermark generator warms on a background thread ~35s after boot (`OMNIVOICE_PRELOAD_WATERMARK=0` opts out; explicitly setting `=1` may download it), so the first synthesis no longer serializes the audioseal import + model load inline — measured at ~42s on a cold filesystem, 3s short of a 90s client timeout (#1576) — thanks @paoloantinori!
 - Voices you've cloned stay "warm" across restarts — encoded references now persist to disk (~10 KB each), so the first generation of a session skips the re-encode and any transcription pass; `OMNIVOICE_PROMPT_DISK_CACHE=0` opts out (#1565)
 - Optional FlashInfer acceleration for the default engine on CUDA (`OMNIVOICE_FLASHINFER=1`, ~2.2x measured) — needs the optional `flashinfer-python` package; missing package or kernel failure logs why and falls back to the standard path (#1565)
 - The bug reporter notices when you're on an outdated build and offers the latest release before filing — with a "File anyway" escape hatch — and stamps a `Build status` line into every report so up-to-date reports are tellable from stale ones (#1547)
@@ -31,6 +32,7 @@ the frozen-backend fallback mirror it for their toolchains.
 - The OmniVoice guide now covers combining style attributes with a reference clip (consistent instruct stabilizes cloning; the reference wins conflicts), inline pronunciation control (pinyin / CMU phonemes), and corrects the claim that the default engine can't do voice design — it can, from attributes (#1565)
 
 ### Fixed
+- Watermark embedding failures now log the full traceback instead of just the exception message, so a silently-unmarked-audio incident (audio passes through unmarked by design) is diagnosable from the log alone (#1576) — thanks @paoloantinori!
 - Stored artifact subpaths now resolve after moving a data directory between Windows, macOS, Linux, and Docker, while traversal and symlink escapes remain blocked (#1559) — thanks @Eman-Yousaf!
 - A remote browser hitting an API-key-configured server's admin 403 now gets the API-key login form instead of endless console 403s, while desktop and PIN-only/no-key servers keep the plain loopback error so guests are never offered a login no key can satisfy (#1568) — thanks @paoloantinori!
 - The crash-isolated ASR sidecar and its download preflight now agree on which model to load — setting the shared faster-whisper model variable applies to both variants instead of the sidecar quietly using a different one (#1556)

backend/api/routers/archetypes.py

@@ -357,15 +357,11 @@ def _infer(seed: int):
     # Runs on the dedicated watermark pool (#1190): AudioSeal embedding is CPU
     # work that holds no VRAM, so it must not occupy a GPU worker ahead of the
     # next generate on 1-worker hosts.
-    from services.watermark import mark_synthetic
-    from services.model_manager import get_watermark_pool
-    import functools
-    audio_tensor = await run_on_gpu_pool_guarded(
-        functools.partial(mark_synthetic, audio_tensor, model.sampling_rate,
-                          context="archetypes.render"),
-        what="Archetype watermark",
+    from services.watermark import mark_synthetic_async
+    audio_tensor = await mark_synthetic_async(
+        audio_tensor, model.sampling_rate,
+        context="archetypes.render",
         timeout=generate_timeout_s(""),
-        executor=get_watermark_pool(),
     )
 
     out_path.parent.mkdir(parents=True, exist_ok=True)

backend/api/routers/batch.py

@@ -413,19 +413,15 @@ def _gen(text=seg_text, lang=target_lang, dur=seg_duration):
         # unmarked while the interactive dub pipeline marked every segment.
         # One whole-track embed (chunked internally, #1045) is equivalent to
         # dub_generate's per-segment marks: the 16-bit message repeats
-        # throughout. Runs in the GPU pool like generate's finalize; never
-        # raises (degrades to unmarked on failure, same as every producer).
+        # throughout. Never raises (degrades to unmarked on failure, same as
+        # every producer).
         # Dispatched to the dedicated watermark pool, not the GPU pool (#1190):
         # AudioSeal embedding is CPU work that holds no VRAM, and a whole-track
         # embed is long enough that occupying a GPU worker with it stalled the
         # next language's segments on 1-worker hosts.
-        from services.watermark import mark_synthetic
-        from services.model_manager import get_watermark_pool
-        import functools
-        full_audio = await loop.run_in_executor(
-            get_watermark_pool(),
-            functools.partial(mark_synthetic, full_audio, sr,
-                              context="batch.dub_track"),
+        from services.watermark import mark_synthetic_async
+        full_audio = await mark_synthetic_async(
+            full_audio, sr, context="batch.dub_track",
         )
 
         # Same assembly pattern as dub_generate.py:390 — `full_audio` is a

backend/api/routers/generation.py

@@ -870,7 +870,6 @@ async def _finalize_generation(
     Returns ``(watermarked_tensor, meta)`` where ``meta`` carries
     ``id`` / ``filename`` / ``duration`` / ``gen_time``.
     """
-    loop = asyncio.get_running_loop()
     # Invisible AudioSeal provenance watermark on the final audio. Embedding
     # was previously only wired into the dub pipeline (dub_generate.py), so
     # plain TTS came out unmarked despite the setting being on — and the same
@@ -882,12 +881,9 @@ async def _finalize_generation(
     # AudioSeal embedding is CPU work that holds no VRAM, so occupying a GPU
     # worker with it only delays the next generate on 1-worker hosts.
     if not already_marked:
-        from services.watermark import mark_synthetic
-        from services.model_manager import get_watermark_pool
-        audio_tensor = await loop.run_in_executor(
-            get_watermark_pool(),
-            functools.partial(mark_synthetic, audio_tensor, sample_rate,
-                              context="generate.finalize"),
+        from services.watermark import mark_synthetic_async
+        audio_tensor = await mark_synthetic_async(
+            audio_tensor, sample_rate, context="generate.finalize",
         )
     gen_time = round(time.time() - start_time, 2)
 
@@ -1822,12 +1818,10 @@ def _line(obj) -> bytes:
                     # (#1190): AudioSeal embedding is CPU work that owns no
                     # VRAM, and on a 1-worker host it used to serialize
                     # directly ahead of the next generate.
-                    from services.watermark import mark_synthetic
-                    from services.model_manager import get_watermark_pool
-                    _preview = await asyncio.get_running_loop().run_in_executor(
-                        get_watermark_pool(),
-                        functools.partial(mark_synthetic, audio_tensor, sample_rate,
-                                          context="generate.stream_preview"),
+                    from services.watermark import mark_synthetic_async
+                    _preview = await mark_synthetic_async(
+                        audio_tensor, sample_rate,
+                        context="generate.stream_preview",
                     )
                     yield _line({"type": "chunk", "seq": 0, "pcm": _pcm16_b64(_preview)})
                 else:
@@ -1849,12 +1843,10 @@ def _line(obj) -> bytes:
                         # Provenance-mark the streamed copy off the GPU pool
                         # (#1169 mark, #1190 placement): CPU-only AudioSeal
                         # work must not occupy a GPU worker between chunks.
-                        from services.watermark import mark_synthetic
-                        from services.model_manager import get_watermark_pool
-                        preview = await asyncio.get_running_loop().run_in_executor(
-                            get_watermark_pool(),
-                            functools.partial(mark_synthetic, preview, sample_rate,
-                                              context="generate.stream_preview"),
+                        from services.watermark import mark_synthetic_async
+                        preview = await mark_synthetic_async(
+                            preview, sample_rate,
+                            context="generate.stream_preview",
                         )
                         if i == 0:
                             # After the first render so lazy-loading engines

backend/main.py

@@ -1,3 +1,4 @@
+import math
 import os
 import sys
 
@@ -369,19 +370,36 @@ def _env_flag(name: str, default: bool = False) -> bool:
 _EAGER = _env_flag("OMNIVOICE_EAGER_INIT", default=("pytest" in sys.modules))
 
 
+def _env_float(name: str, default: float) -> float:
+    """Parse a float env override, rejecting negative and non-finite values.
+
+    Shared by the preload-delay / timeout knobs: NaN would silently never
+    fire, a negative would fire during startup I/O, so both fall back to the
+    default instead (the bug class CodeRabbit flagged on the watermark knob
+    in PR #1577 — latent in the older copies too, closed here for all)."""
+    raw = os.environ.get(name, "")
+    try:
+        value = float(raw) if raw.strip() else default
+    except ValueError:
+        return default
+    return value if math.isfinite(value) and value >= 0 else default
+
+
 def _capture_preload_delay_s() -> float:
     """Seconds after boot before the dictation (capture ASR) model warms.
 
     Late enough that it never competes with startup I/O or the TTS preload;
     overridable via OMNIVOICE_CAPTURE_PRELOAD_DELAY (mostly for tests)."""
-    raw = os.environ.get("OMNIVOICE_CAPTURE_PRELOAD_DELAY", "")
-    try:
-        v = float(raw)
-        if v >= 0:
-            return v
-    except (TypeError, ValueError):
-        pass
-    return 30.0
+    return _env_float("OMNIVOICE_CAPTURE_PRELOAD_DELAY", 30.0)
+
+def _watermark_preload_delay_s() -> float:
+    """Seconds after boot before the AudioSeal generator warm-up fires.
+
+    Own knob, NOT ``_capture_preload_delay_s`` + offset: a capture-specific
+    env override must not retime the watermark warm too, and the two cold
+    imports shouldn't fire on the same tick (CodeRabbit, PR #1577). Default
+    35s sits ~5s past the capture-ASR warm for the same reason."""
+    return _env_float("OMNIVOICE_PRELOAD_WATERMARK_DELAY", 35.0)
 
 
 def _capture_preload_ram_ok(min_free_bytes: int = 4 * 1024**3) -> bool:
@@ -398,14 +416,7 @@ def _capture_preload_ram_ok(min_free_bytes: int = 4 * 1024**3) -> bool:
 def _mcp_start_timeout_s() -> float:
     """Seconds to wait for the MCP session manager to start before giving up
     and serving without it (#632). Overridable via OMNIVOICE_MCP_START_TIMEOUT_S."""
-    raw = os.environ.get("OMNIVOICE_MCP_START_TIMEOUT_S", "")
-    try:
-        v = float(raw)
-        if v > 0:
-            return v
-    except (TypeError, ValueError):
-        pass
-    return 30.0
+    return max(_env_float("OMNIVOICE_MCP_START_TIMEOUT_S", 30.0), 0.001)
 
 
 async def _serve_mcp(session_manager, ready: "asyncio.Event", stop: "asyncio.Event") -> None:
@@ -852,6 +863,8 @@ async def _phase_b(app: FastAPI) -> None:
     # #1174: arm model loads for THIS run — an in-process relaunch may carry a
     # stale shutting-down flag from a previous lifespan.
     model_loads_reset_shutdown()
+    from services.model_manager import begin_watermark_pool_lifecycle
+    begin_watermark_pool_lifecycle()
     app.state.idle_task = asyncio.create_task(idle_worker())
     app.state.worker_task = asyncio.create_task(task_manager.worker())
     # Warm the TTS model in the background so first /generate is instant.
@@ -905,6 +918,50 @@ def _warm():
     else:
         logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
 
+    # Watermark: warm the AudioSeal generator in the background so the first
+    # mark_synthetic doesn't serialize the audioseal import + model load
+    # inside the first synthesis (measured ~42 s inline on a cold filesystem,
+    # 2026-08-17 macOS report — 3 s short of the client's 90 s timeout).
+    # Small model on CPU; deferred a few seconds past the capture-ASR warm so
+    # the two cold imports don't contend for the same disk, and no RAM guard
+    # is needed. Runs on the watermark pool — where the model is used — not
+    # the shared default executor.
+    if _env_flag("OMNIVOICE_PRELOAD_WATERMARK", default=True):
+        async def _preload_watermark():
+            await asyncio.sleep(_watermark_preload_delay_s())
+            loop = asyncio.get_running_loop()
+            from services import watermark as _watermark
+
+            # Gate BEFORE touching get_watermark_pool(): the pool is lazy so
+            # hosts with watermarking disabled never spawn its thread, and
+            # creating it unconditionally would break that invariant. The
+            # race with a first embed is benign — pool creation is itself
+            # lock-guarded.
+            if not _watermark.will_mark():
+                logger.debug("Watermark preload skipped (disabled or audioseal absent)")
+                return
+            from services.model_manager import get_watermark_pool
+
+            # Default startup may warm an existing local checkpoint but may
+            # not fetch one. Only an explicit user opt-in permits a download.
+            raw_preload = os.environ.get("OMNIVOICE_PRELOAD_WATERMARK", "")
+            allow_download = raw_preload.strip().lower() in {"1", "true", "yes", "on"}
+
+            try:
+                await loop.run_in_executor(
+                    get_watermark_pool(),
+                    lambda: _watermark.prefetch_generator(
+                        allow_download=allow_download
+                    ),
+                )
+            except Exception:
+                # prefetch_generator swallows its own errors; this guards the
+                # setup half (imports, pool construction) so a broken warm-up
+                # is visible now, not as an unretrieved exception at shutdown.
+                logger.warning("Watermark preload task failed", exc_info=True)
+
+        app.state.watermark_preload_task = asyncio.create_task(_preload_watermark())
+
     # ── MCP session manager (Wave 2.2) ────────────────────────────────────
     # Run it in its OWN task owning the full enter→exit lifecycle (anyio
     # task-affinity, see _serve_mcp); only wait, with a timeout, for ready —
@@ -1080,8 +1137,20 @@ async def lifespan(app: FastAPI):
         getattr(app.state, "worker_task", None),
         getattr(app.state, "preload_task", None),
         getattr(app.state, "capture_preload_task", None),
+        getattr(app.state, "watermark_preload_task", None),
         timeout=20.0,
     )
+    # The watermark warm-up runs on its dedicated 1-worker pool. Cancellation
+    # detaches the asyncio future but cannot kill a thread inside AudioSeal,
+    # so drain it fully before lifespan teardown reports completion.
+    try:
+        from services.model_manager import shutdown_watermark_pool as _wm_drain
+
+        _wm_drain()
+    except Exception:
+        # Best-effort drain: a failure here must not abort the remaining
+        # shutdown steps (model unload, MCP teardown) below.
+        logger.warning("Watermark pool drain failed at shutdown", exc_info=True)
     # Unload the model and free GPU memory
     try:
         import services.model_manager as mm