feat(omnivoice): voice prompts that survive restarts + opt-in FlashInfer (~2.2x)

CHANGELOG.md

@@ -18,12 +18,15 @@ the frozen-backend fallback mirror it for their toolchains.
 - The backend binds its port immediately and reports startup progress live — `/health` answers 503-with-step and a new `/startup/progress` endpoint lists every step while PyTorch, API routes, and database migrations load in the background, so "starting at step X" is never mistakable for "dead"; the desktop splash narrates each step (#1550)
 
 ### Added
+- Voices you've cloned stay "warm" across restarts — encoded references now persist to disk (~10 KB each), so the first generation of a session skips the re-encode and any transcription pass; `OMNIVOICE_PROMPT_DISK_CACHE=0` opts out (#1565)
+- Optional FlashInfer acceleration for the default engine on CUDA (`OMNIVOICE_FLASHINFER=1`, ~2.2x measured) — needs the optional `flashinfer-python` package; missing package or kernel failure logs why and falls back to the standard path (#1565)
 - The bug reporter notices when you're on an outdated build and offers the latest release before filing — with a "File anyway" escape hatch — and stamps a `Build status` line into every report so up-to-date reports are tellable from stale ones (#1547)
 - Settings → Performance & Device gains a compute-device override (Auto / CUDA / ROCm / XPU / MPS / CPU, or `OMNIVOICE_DEVICE`) — pin the device when auto-detect picks wrong; only devices your machine actually has are offered (#1557)
 
 ### Docs
 - The READMEs now lead with download buttons and a three-step first-clone walkthrough, and a new benchmarks page anchors measured per-engine/per-device numbers on the in-repo harness (#1555)
 - Every engine now has its own guide — 21 new pages under docs/engines plus an index covering all 16 TTS and 11 ASR engines, linked from both READMEs (#1556)
+- The OmniVoice guide now covers combining style attributes with a reference clip (consistent instruct stabilizes cloning; the reference wins conflicts), inline pronunciation control (pinyin / CMU phonemes), and corrects the claim that the default engine can't do voice design — it can, from attributes (#1565)
 
 ### Fixed
 - The crash-isolated ASR sidecar and its download preflight now agree on which model to load — setting the shared faster-whisper model variable applies to both variants instead of the sidecar quietly using a different one (#1556)

backend/services/engine_env.py

@@ -57,6 +57,99 @@ def _force_compile_requested() -> bool:
     return value.strip().lower() in {"1", "true", "yes", "on"}
 
 
+# ── FlashInfer opt-in (upstream k2-fsa port) ────────────────────────────────
+# Explicit power-user opt-in, CUDA-only: OMNIVOICE_FLASHINFER=1 patches the
+# OmniVoice model with flashinfer packed attention (~2x per upstream's
+# benchmarks); =graph additionally captures CUDA graphs (best at batch=1).
+# Off by default — `flashinfer` is not a shipped dependency, and an
+# optimization must never be a point of failure. Session-sticky failure
+# latch mirrors torch.compile's (#278).
+_FLASHINFER_ENV = "OMNIVOICE_FLASHINFER"
+_flashinfer_runtime_failure: Optional[str] = None
+
+
+def flashinfer_mode() -> str:
+    """The user's ``OMNIVOICE_FLASHINFER`` request: 'off' | 'on' | 'graph'.
+
+    Unknown values normalize to 'off' with a log line naming the env var, so
+    a typo degrades to the default path instead of half-applying.
+    """
+    value = os.environ.get(_FLASHINFER_ENV, "").strip().lower()
+    if value in {"", "0", "false", "no", "off"}:
+        return "off"
+    if value in {"1", "true", "yes", "on"}:
+        return "on"
+    if value == "graph":
+        return "graph"
+    logger.warning(
+        "%s=%r not recognized (valid: 0, 1, graph) — FlashInfer stays off.",
+        _FLASHINFER_ENV, value,
+    )
+    return "off"
+
+
+def should_flashinfer(device: str) -> str:
+    """Resolve the FlashInfer request against this host: 'off' | 'on' | 'graph'.
+
+    Requires all of: the ``OMNIVOICE_FLASHINFER`` opt-in, device == "cuda"
+    (flashinfer is CUDA-only), the ``flashinfer`` package importable, and no
+    earlier runtime failure this session. Every refusal is logged with the
+    reason and the knob's name — the user asked for it, so silence would read
+    as "the setting doesn't work".
+    """
+    mode = flashinfer_mode()
+    if mode == "off":
+        return "off"
+    if device != "cuda":
+        logger.warning(
+            "%s requested but the compute device is %r — FlashInfer is "
+            "CUDA-only, continuing without it.", _FLASHINFER_ENV, device,
+        )
+        return "off"
+    if importlib.util.find_spec("flashinfer") is None:
+        logger.warning(
+            "%s requested but the `flashinfer` package is not installed — "
+            "continuing without it. Install with: uv pip install "
+            "flashinfer-python flashinfer-jit-cache "
+            "--extra-index-url https://flashinfer.ai/whl/cu128/ "
+            "(pick the index matching your CUDA build).", _FLASHINFER_ENV,
+        )
+        return "off"
+    if _flashinfer_runtime_failure is not None:
+        logger.info(
+            "FlashInfer skipped: failed earlier this session (%s) — using the "
+            "standard path.", _flashinfer_runtime_failure,
+        )
+        return "off"
+    return mode
+
+
+def mark_flashinfer_runtime_failure(reason: str) -> None:
+    """Latch a FlashInfer apply/runtime failure for the rest of the process,
+    same contract as ``mark_compile_runtime_failure``."""
+    global _flashinfer_runtime_failure
+    try:
+        # Import/kernel errors embed absolute paths (wheels under the user's
+        # home) — redact before latching, since the reason is logged here and
+        # re-logged on every later skip.
+        from core.failure import sanitize
+
+        reason = sanitize(reason)
+    except Exception:
+        # Fail closed: if the redactor itself breaks, latching the raw text
+        # would defeat the redaction. Keep only the exception class (the part
+        # before ':' in our "Type: message" reasons) and drop the message.
+        reason = (
+            f"{(reason or '').split(':', 1)[0][:80]} "
+            "(details redacted: sanitizer unavailable)"
+        ).strip()
+    _flashinfer_runtime_failure = reason or "unknown FlashInfer runtime failure"
+    logger.warning(
+        "FlashInfer disabled for this session after a runtime failure: %s",
+        _flashinfer_runtime_failure,
+    )
+
+
 def _cuda_arch_supported_for_compile() -> "tuple[bool, str]":
     """Check the GPU's architecture against this torch build's arch list.
 

backend/services/model_manager.py

@@ -1376,6 +1376,122 @@
     _model.generate = _generate_with_compile_fallback
 
 
+# ── FlashInfer runtime fallback (upstream k2-fsa port) ──────────────────────
+
+
+def _is_flashinfer_runtime_failure(exc: BaseException) -> bool:
+    """True when an exception originates in the FlashInfer fast path (the
+    flashinfer package, our omnivoice_flashinfer patch module, or CUDA-graph
+    capture/replay) rather than in the model or the request itself. Same
+    chain/traceback walk as ``_is_compile_runtime_failure``."""
+    import traceback as _tb
+
+    tb_markers = ("/flashinfer/", "omnivoice_flashinfer")
+    msg_markers = ("flashinfer", "cuda graph", "cudagraph")
+    seen: set[int] = set()
+    cur: BaseException | None = exc
+    while cur is not None and id(cur) not in seen:
+        seen.add(id(cur))
+        mod = type(cur).__module__ or ""
+        if mod.startswith("flashinfer"):
+            return True
+        msg = str(cur).lower()
+        if any(marker in msg for marker in msg_markers):
+            return True
+        try:
+            for frame in _tb.extract_tb(cur.__traceback__):
+                filename = (frame.filename or "").replace("\\", "/")
+                if any(marker in filename for marker in tb_markers):
+                    return True
+        except Exception:
+            pass
+        if cur.__cause__ is not None:
+            cur = cur.__cause__
+        elif not cur.__suppress_context__:
+            cur = cur.__context__
+        else:
+            cur = None
+    return False
+
+
+def _unapply_flashinfer(_model) -> None:
+    """Restore the standard execution path on a FlashInfer-patched model.
+
+    ``apply_flashinfer`` works entirely through *instance-level* state —
+    MethodType-bound ``forward``/``_generate_iterative`` overrides and
+    ``_fi_*`` attributes — so deleting those attributes restores the class
+    implementations exactly. The attention implementation is restored to the
+    one captured before apply (``_fi_orig_attn_impl`` — could be
+    flash_attention_2, not just sdpa), and use_cache is re-enabled."""
+    llm = getattr(_model, "llm", None)
+    orig_attn = getattr(_model, "_fi_orig_attn_impl", None) or "sdpa"
+    if llm is not None:
+        for module in llm.modules():
+            if "forward" in vars(module):
+                del module.forward
+            for attr in ("_fi_w_qkv", "_fi_qkv_split", "_fi_rope_theta", "_fi_w_gate_up"):
+                if attr in vars(module):
+                    delattr(module, attr)
+        try:
+            llm.set_attn_implementation(orig_attn)
+        except Exception:
+            logger.exception(
+                "failed to restore %s attention after FlashInfer", orig_attn
+            )
+        llm.config.use_cache = True
+    for attr in (
+        "_fi_orig_attn_impl",
+        "_generate_iterative",
+        "_fi_runner",
+        "_fi_graph_cache",
+        "_fi_enable_cuda_graph",
+        "_fi_graph_buckets",
+        "_fi_overhead_budget",
+    ):
+        if attr in vars(_model):
+            delattr(_model, attr)
+
+
+def _install_flashinfer_fallback(_model) -> None:
+    """Wrap ``model.generate`` so a FlashInfer failure at inference time falls
+    back to the standard path instead of failing the generation — the same
+    contract as ``_install_compile_fallback`` (#278): an optimization must
+    never turn a working generation into an error."""
+    orig_generate = _model.generate
+
+    def _generate_with_flashinfer_fallback(*args, **kwargs):
+        try:
+            return orig_generate(*args, **kwargs)
+        except Exception as exc:
+            if not _is_flashinfer_runtime_failure(exc):
+                raise
+            logger.warning(
+                "FlashInfer runtime failure during generation (%s: %s) — "
+                "restoring the standard path and disabling FlashInfer for "
+                "this session. Generation is being retried without it.",
+                type(exc).__name__, exc,
+            )
+            from services import engine_env
+            engine_env.mark_flashinfer_runtime_failure(
+                f"{type(exc).__name__}: {exc}"
+            )
+            # Unapply BEFORE exposing the eager path: while the teardown
+            # mutates modules, _model.generate still routes through the
+            # thread-affinity wrapper, so a concurrent render queues behind
+            # this call instead of racing the half-restored model (Greptile,
+            # #1565 round 2). Only a fully restored model is published.
+            _unapply_flashinfer(_model)
+            _model.generate = orig_generate
+            try:
+                return orig_generate(*args, **kwargs)
+            except Exception as plain_exc:
+                # `from None`: a genuine standard-path failure must not be
+                # chained to — and misread as — the FlashInfer error.
+                raise plain_exc from None
+
+    _model.generate = _generate_with_flashinfer_fallback
+
+
 # ── #315: thread affinity for cudagraph-compiled models ─────────────────────
 # `torch.compile(mode="reduce-overhead")` captures CUDA graphs, and captured
 # graph state is **thread-local** (torch/_inductor/cudagraph_trees keys its
@@ -2117,14 +2233,65 @@
                     "to stop preloading it alongside TTS."
                 ) from asr_exc
 
+        # FlashInfer opt-in (upstream k2-fsa port): packed CFG attention +
+        # fused kernels, ~2x on upstream's benchmarks. Applied INSTEAD of
+        # torch.compile — both rewrite the llm's execution and they do not
+        # compose. Best-effort: any apply failure latches the session off and
+        # the standard path continues untouched.
+        flashinfer_applied = False
+        try:
+            from services.engine_env import (
+                mark_flashinfer_runtime_failure,
+                should_flashinfer,
+            )
+
+            fi_mode = should_flashinfer(device)
+            if fi_mode != "off":
+                _set_loading("compiling", "Applying FlashInfer kernels…")
+                try:
+                    from omnivoice.models.omnivoice_flashinfer import apply_flashinfer
+
+                    # Captured BEFORE apply so unapply (either the failure
+                    # branch below or the generate-time fallback) restores
+                    # the true prior implementation.
+                    _model._fi_orig_attn_impl = getattr(
+                        _model.llm.config, "_attn_implementation", "sdpa"
+                    )
+                    apply_flashinfer(_model, enable_cuda_graph=(fi_mode == "graph"))
+                except Exception as fi_exc:  # noqa: BLE001 — perf opt, never fatal
+                    mark_flashinfer_runtime_failure(
+                        f"{type(fi_exc).__name__}: {fi_exc}"
+                    )
+                    # apply_flashinfer mutates the model as it goes — a
+                    # failure partway leaves half-patched modules that would
+                    # crash the next render (Greptile, #1565). Restore fully.
+                    _unapply_flashinfer(_model)
+                else:
+                    flashinfer_applied = True
+                    _install_flashinfer_fallback(_model)
+                    # BOTH modes pin inference to one thread. Graph mode for
+                    # the #315 reason (captured CUDA-graph state is
+                    # thread-local); eager mode because the FlashInfer
+                    # attention wrapper and packed position ids are planned
+                    # per generation in module state — two _gpu_pool workers
+                    # interleaving plan() and run() would corrupt each
+                    # other's layout (CodeRabbit/Greptile, #1565).
+                    _install_compile_thread_affinity(_model)
+                    logger.info(
+                        "FlashInfer applied (mode=%s) — torch.compile skipped "
+                        "for this load.", fi_mode,
+                    )
+        except Exception:
+            logger.exception("FlashInfer opt-in check failed; continuing without")
+
         try:
             # plan-02 (#65): gate on Triton availability (+ user setting), not
             # just device==cuda. Triton has no Windows wheel, so the old
             # cuda-only check OOM'd on Windows+CUDA; should_torch_compile()
             # falls back to eager there.
             from services.engine_env import should_torch_compile
 
-            if should_torch_compile(device):
+            if not flashinfer_applied and should_torch_compile(device):
                 _set_loading("compiling", "Compiling model (torch.compile)…")
                 try:
                     _model.llm = torch.compile(_model.llm, mode=_TORCH_COMPILE_MODE)