diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f2ce07a6..d7df6ba38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,12 +10,14 @@ the frozen-backend fallback mirror it for their toolchains. **Highlights** -- The backend now answers within a second of launch and narrates its startup step by step -- Reporting a bug from an outdated build now offers the latest release first -- The backend is only announced ready once it can actually serve, and crash-loop restarts now pace themselves +- Dictation now stays bound to the app where it started and recovers locally from silent recognizer output (#1175) +- The backend now answers within a second of launch and narrates its startup step by step (#1550) +- Reporting a bug from an outdated build now offers the latest release first (#1547) +- The backend is only announced ready once it can actually serve, and crash-loop restarts now pace themselves (#1548) - Invisible watermarking no longer stalls — or silently skips — the first take of a session (#1615) ### Changed +- Dictation now carries one native output session from shortcut-down through final delivery, restores text, HTML, image, or file-list clipboards only when untouched, keeps Wayland copy-safe unless current-focus insertion is explicitly enabled, and retries silent Sherpa speech only through an already-installed local ASR model (#1175) - The backend binds its port immediately and reports startup progress live — `/health` answers 503-with-step and a new `/startup/progress` endpoint lists every step while PyTorch, API routes, and database migrations load in the background, so "starting at step X" is never mistakable for "dead"; the desktop splash narrates each step (#1550) ### Added @@ -37,6 +39,12 @@ the frozen-backend fallback mirror it for their toolchains. - The OmniVoice guide now covers combining style attributes with a reference clip (consistent instruct stabilizes cloning; the reference wins conflicts), inline pronunciation control (pinyin / CMU phonemes), and corrects the claim that the default engine can't do voice design — it can, from attributes (#1565) ### Fixed +- Dictation on a WebView that refuses a 16 kHz audio context (WKWebView) now low-passes before downsampling, so frequencies above 8 kHz stop folding into the speech the recognizer is fed (#1610) +- A microphone context that cannot be resumed now reports a mic error instead of leaving the dictation pill on "Listening" while capturing nothing (#1610) +- Dictation no longer retains a whole session's audio for silent-model recovery — an open mic grew that buffer by ~115 MB an hour; the recent two minutes are kept instead (#1610) +- The clipboard-delivery status is now translated in all 21 languages, so Wayland users — where clipboard delivery is the default — no longer see an English string (#1610) +- A native sherpa-onnx load failure of any exception type now degrades to "engine unavailable" instead of taking the dictation WebSocket down (#1610) +- Dictation now ships Whisper Tiny as its one cross-platform default, avoiding Parakeet's measured empty decoding on Windows while keeping Parakeet selectable behind runtime fallback (#1175) - Re-mixing a dub no longer decodes, rewrites, and re-reads every cached segment — same-rate cached audio is reused directly (and rejected if truncated), switching timing modes can't reuse slot-truncated audio as natural-rate, and RVC respects natural-rate modes (#1594) - PocketTTS French works again — pocket-tts only ships a 24-layer French model and rejected the name the sidecar asked for, so every French request failed at model load; French now always loads `french_24l` (#1613) — thanks @paoloantinori! - Installing IndexTTS 2.5 no longer fails claiming an interrupted download — the weights repo ships `config.yaml` and VoiceStudio demanded a `config_v2_5.yaml` that exists in no upstream release; both names are accepted, so a hand-renamed checkout keeps working (#1611) — thanks @zuiaiyutu! diff --git a/README.md b/README.md index f3bd54e1d..ed2c1d4ac 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,7 @@ Use `bun run dev` for the browser UI. See [Contributing](.github/CONTRIBUTING.md | **Voice Design** | Create a voice from age, accent, pitch, style, and delivery instructions | | **Video Dubbing** | Transcribe, translate, preserve speakers, synthesize, and export video | | **Stories and audiobooks** | Multi-voice scripts · EPUB/PDF import · chapter rendering · `.m4b` export | -| **Dictation Widget** | System-wide shortcut, live transcription, optional local-LLM cleanup | +| **[Dictation Widget](docs/features/dictation.md)** | System-wide shortcut, live transcription, optional local-LLM cleanup | | **Vocal Isolation** | Demucs speech/background separation | | **Speaker Diarization** | Pyannote and WhisperX speaker assignment | | **Batch Queue** | Queue large sets of audio and video jobs with per-job progress | diff --git a/backend/api/routers/capture_ws.py b/backend/api/routers/capture_ws.py index c021087ef..b7b13d8ab 100644 --- a/backend/api/routers/capture_ws.py +++ b/backend/api/routers/capture_ws.py @@ -27,6 +27,10 @@ "detail": "..."} — error ("detail" kept for legacy) + Sherpa ``final`` frames additionally carry + ``"final_kind": "utterance"|"summary"``. Utterances are mid-session + commits; the summary is the authoritative whole-session result at EOF. + Every ``final`` text is normalised by services.text_polish (leading capital for Latin scripts, terminal punctuation, single-spaced) so the pasted result reads like typed text. Partials are raw. @@ -35,6 +39,7 @@ import asyncio import logging +import math import os import tempfile import time @@ -70,17 +75,46 @@ _AEC_FAR = 0x01 # playback reference frame (feed the echo model only) -def _requested_pcm_sample_rate(query_params) -> int | None: - """Return a bounded PCM rate for ``?pcm=1``/``?aec=1`` sessions.""" - raw_pcm = query_params.get("pcm") in ("1", "true", "on") - aec = query_params.get("aec") in ("1", "true", "on") - if not raw_pcm and not aec: - return None +# Client-supplied ``?sr=`` values outside the range real capture devices use +# are replaced with 16 kHz. The rate sizes server-side state — RecoveryTail +# multiplies it by RECOVERY_TAIL_SECONDS to compute its byte ceiling — so an +# absurd rate must never be believed: it would re-open the unbounded-memory +# path the recovery-tail cap closed. +SR_MIN, SR_MAX = 8000, 96000 + + +def _bounded_sample_rate(query_params) -> int: try: sample_rate = int(query_params.get("sr", "16000")) except (TypeError, ValueError): return 16000 - return sample_rate if 8000 <= sample_rate <= 96000 else 16000 + return sample_rate if SR_MIN <= sample_rate <= SR_MAX else 16000 + + +def _requested_pcm_sample_rate(query_params) -> int | None: + """Return the bounded rate when the client transport is raw PCM. + + Sherpa clients omit ``pcm=1`` because the selected model already defines + that transport. If the model is demoted or its runtime is unavailable, the + legacy recognizer fallback must still decode those same bytes as PCM. + """ + raw_pcm = query_params.get("pcm") in ("1", "true", "on") + aec = query_params.get("aec") in ("1", "true", "on") + sherpa_pcm = False + requested_model = query_params.get("model") + if requested_model: + try: + from services.sherpa_dictation import is_sherpa_model + sherpa_pcm = is_sherpa_model(requested_model) + except Exception: # noqa: BLE001 + # A broken sherpa install must not decide the framing question — + # sherpa_pcm stays False and the session negotiates the + # MediaRecorder path; availability is re-probed (and reported) + # when the model is actually selected. + sherpa_pcm = False + if not raw_pcm and not aec and not sherpa_pcm: + return None + return _bounded_sample_rate(query_params) def _demux_aec_frame(data: bytes) -> tuple[str, bytes]: @@ -137,16 +171,27 @@ def _select_sherpa_spec(websocket: WebSocket): from services import sherpa_dictation as sd except Exception: return None + + def _usable_spec(model_id): + spec = sd.get_spec(model_id) + if spec is not None and sd.is_demoted(spec.id): + logger.warning( + "dictation model %s is demoted — using the capture ASR fallback", + spec.id, + ) + return None + return spec + requested = websocket.query_params.get("model") if requested: - return sd.get_spec(requested) # explicit selection (may be None if bad) + return _usable_spec(requested) # explicit selection (may be unavailable) # Fall back to the persisted dictation pref. try: from services.asr_backend import dictation_model_id mid = dictation_model_id() except Exception: mid = None - return sd.get_spec(mid) if mid else None + return _usable_spec(mid) if mid else None @router.websocket("/ws/transcribe") @@ -422,6 +467,64 @@ async def process_partials(): SHERPA_OFFLINE_RMS_FLOOR = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_RMS", "0.01")) +#: Seconds of audio retained for silent-model recovery. Recovery only needs +#: enough speech to prove the model is broken and to re-transcribe what was +#: said; retaining the whole session grew ~115 MB/hour at 16 kHz on an open +#: mic, unbounded, and only ever got read when the fallback fired. +RECOVERY_TAIL_DEFAULT_SECONDS = 120.0 +RECOVERY_TAIL_MAX_SECONDS = 300.0 + + +def _bounded_recovery_tail_seconds(value: str | None) -> float: + """Parse the recovery tail override without allowing unbounded buffers.""" + try: + seconds = float(value) if value is not None else RECOVERY_TAIL_DEFAULT_SECONDS + except (TypeError, ValueError): + return RECOVERY_TAIL_DEFAULT_SECONDS + if not math.isfinite(seconds) or seconds <= 0: + return RECOVERY_TAIL_DEFAULT_SECONDS + return min(seconds, RECOVERY_TAIL_MAX_SECONDS) + + +RECOVERY_TAIL_SECONDS = _bounded_recovery_tail_seconds( + os.environ.get("OMNIVOICE_DICTATION_RECOVERY_TAIL_S") +) + + +class RecoveryTail: + """The most recent ``RECOVERY_TAIL_SECONDS`` of session audio. + + Keeps the *tail* rather than the head: a long dictation's useful speech is + what the user just said, and the silent-model check cares about how much + audio the session carried overall — which ``total_bytes`` still reports + truthfully after trimming. + """ + + __slots__ = ("_buf", "_max", "total_bytes") + + def __init__(self, sample_rate: int, seconds: float = RECOVERY_TAIL_SECONDS): + # int16 mono → 2 bytes/sample. Floor of one frame so a nonsense rate + # or seconds value can't produce a zero-length buffer. + self._max = max(2, int(seconds * max(1, sample_rate)) * 2) + self._buf = bytearray() + self.total_bytes = 0 + + def extend(self, pcm: bytes) -> None: + self._buf.extend(pcm) + self.total_bytes += len(pcm) + excess = len(self._buf) - self._max + if excess > 0: + # int16 mono: trim whole samples only. A split frame can carry an + # odd byte count, and an odd trim would leave the tail starting + # mid-sample — every later sample byte-shifted, and the recovery + # transcription fed noise. + excess += excess % 2 + del self._buf[:excess] + + def tail(self) -> bytes: + return bytes(self._buf) + + def is_model_silent(text: str, heard_speech: bool, pcm_bytes: int) -> bool: """True when the dictation model produced NO text despite real speech. @@ -448,19 +551,74 @@ def _pcm16_to_f32(pcm: bytes): return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0 +def _pcm16_rms(pcm: bytes) -> float: + samples = _pcm16_to_f32(pcm) + if not len(samples): + return 0.0 + return float((samples * samples).mean() ** 0.5) + + +async def _recover_silent_sherpa( + spec, pcm: bytes, pcm_sr: int, +) -> tuple[str, list[dict]]: + """Retry a token-silent Sherpa session through an installed local ASR.""" + logger.warning( + "dictation model %s decoded NOTHING from %.1fs of speech-level audio " + "— falling back to the capture ASR engine for this session", + spec.id, len(pcm) / float(max(1, pcm_sr) * 2), + ) + try: + from services.asr_backend import asr_model_missing_error + fallback_missing = await asyncio.to_thread( + asr_model_missing_error, + purpose="dictation", + skip_sherpa=True, + require_installed=True, + ) + if fallback_missing is not None: + logger.warning( + "dictation silent-model fallback is not installed (%s); " + "skipping recovery to avoid an automatic download", + fallback_missing.get("missing_repo_id", "unknown"), + ) + return "", [] + + result = await _transcribe_buffer_full( + [pcm], pcm_sr=pcm_sr, skip_sherpa=True, + ) + text = polish_text(_result_text(result)) + if not text: + return "", [] + # The RMS gate can fire on fan/keyboard noise. Only another recognizer + # producing words proves the audio held speech and makes persistent + # demotion safe. + try: + from services.sherpa_dictation import demote_model + if await asyncio.to_thread(demote_model, spec.id): + logger.error( + "dictation model %s demoted on this machine — it will no longer be " + "auto-selected. Pick it again in Settings to give it another chance.", + spec.id, + ) + except Exception: + logger.exception("silent-model demotion failed") + segments = (result or {}).get("segments") or [ + {"start": 0.0, "end": None, "text": text} + ] + return text, segments + except Exception: + logger.exception("dictation silent-model fallback failed") + return "", [] + + async def _sherpa_session(websocket: WebSocket): - """Shared WS receive setup for the sherpa handlers. + """Shared WS setup for the sherpa handlers. - Returns ``(get_frame, state)`` where ``get_frame`` is an async callable - that yields the next near-end (mic) PCM bytes, ``b""`` for a keepalive/ref - frame, or ``None`` on EOF/disconnect. ``state`` carries sample rate, AEC, - and the disconnect flag for the caller's finaliser. + Returns ``(pcm_sr, aec)``: the bounded PCM sample rate for the session + and the echo canceller when ``?aec=1`` requested one (``None`` otherwise + or when AEC setup fails). """ - pcm_sr = 16000 - try: - pcm_sr = int(websocket.query_params.get("sr", "16000")) - except (TypeError, ValueError): - pcm_sr = 16000 + pcm_sr = _bounded_sample_rate(websocket.query_params) aec = None if websocket.query_params.get("aec") in ("1", "true", "on"): try: @@ -569,6 +727,8 @@ async def _run_sherpa_streaming(websocket: WebSocket, spec): last_partial = "" committed: list[str] = [] # finalized utterances this session + session_pcm = RecoveryTail(pcm_sr) # bounded audio for silent-model recovery + heard_speech = False client_disconnected = False async def _send(payload) -> bool: @@ -610,6 +770,9 @@ def _flush_final(): break if kind == "skip": continue + session_pcm.extend(pcm) + if not heard_speech and _pcm16_rms(pcm) >= SHERPA_OFFLINE_RMS_FLOOR: + heard_speech = True text, endpoint = await asyncio.to_thread(_decode_after_feed, pcm) if endpoint: # Commit this utterance (polished — it gets pasted); reset @@ -618,6 +781,7 @@ def _flush_final(): if text: committed.append(text) await _send({"type": "final", "text": text, + "final_kind": "utterance", "segments": [{"start": 0.0, "end": None, "text": text}], "language": "auto", "engine": backend.id}) rec.reset(stream) @@ -644,7 +808,28 @@ def _flush_final(): # Pieces are already polished; the join is too (polish is idempotent). full = " ".join(t for t in committed if t).strip() segments = [{"start": 0.0, "end": None, "text": t} for t in committed if t] + + model_silent = is_model_silent(full, heard_speech, session_pcm.total_bytes) + if model_silent: + recovered, recovered_segments = await _recover_silent_sherpa( + spec, session_pcm.tail(), pcm_sr, + ) + if recovered: + full = recovered + segments = recovered_segments + if not client_disconnected: + payload = {"type": "final", "text": full, "final_kind": "summary", + "segments": segments, + "language": "auto", "engine": backend.id} + if model_silent: + payload["engine"] = "capture-asr-fallback" if full else backend.id + payload["model_silent"] = spec.id + payload["warning"] = ( + f"The selected dictation model ({spec.id}) produced no text from your " + "speech. Switched to the fallback engine for this session — pick a " + "different model in Settings → Dictation." + ) if full: # Hard-bounded refinement (~4s): never delays this summary `final` # beyond OMNIVOICE_REFINE_TIMEOUT_S even with a dead LLM endpoint. @@ -653,14 +838,9 @@ def _flush_final(): refined = await maybe_refine_async(full) except Exception: refined = None - payload = {"type": "final", "text": full, "segments": segments, - "language": "auto", "engine": backend.id} if refined and refined != full: payload["refined_text"] = refined - await _send(payload) - else: - await _send({"type": "final", "text": "", "segments": [], - "language": "auto", "engine": backend.id}) + await _send(payload) try: await websocket.close() except Exception: @@ -697,7 +877,7 @@ async def _run_sherpa_offline(websocket: WebSocket, spec): # whisper/zipformer transcribe the same bytes). Keep the whole session's # audio and whether any of it was speech-level, so the finaliser can tell # "user said nothing" (fine) from "model produced nothing" (broken). - session_pcm = bytearray() + session_pcm = RecoveryTail(pcm_sr) heard_speech = False running = True client_disconnected = False @@ -716,12 +896,6 @@ async def _send(payload) -> bool: client_disconnected = True return False - def _rms(pcm: bytes) -> float: - samples = _pcm16_to_f32(pcm) - if not len(samples): - return 0.0 - return float((samples * samples).mean() ** 0.5) - def _decode_window(pcm: bytes) -> str: samples = _pcm16_to_f32(pcm) if not len(samples): @@ -740,7 +914,7 @@ async def receive(): continue buf.extend(pcm) session_pcm.extend(pcm) - if not heard_speech and _rms(pcm) >= SHERPA_OFFLINE_RMS_FLOOR: + if not heard_speech and _pcm16_rms(pcm) >= SHERPA_OFFLINE_RMS_FLOOR: heard_speech = True last_audio = time.monotonic() except WebSocketDisconnect: @@ -766,6 +940,7 @@ async def _commit(snapshot: bytes): if text: committed.append(text) await _send({"type": "final", "text": text, + "final_kind": "utterance", "segments": [{"start": 0.0, "end": None, "text": text}], "language": "auto", "engine": backend.id}) @@ -777,8 +952,8 @@ async def partials(): continue snapshot = bytes(buf) if len(snapshot) > sil_bytes and \ - _rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR: - if _rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR: + _pcm16_rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR: + if _pcm16_rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR: await _commit(snapshot) else: # Pure silence — drop it (keep the gate window for @@ -824,39 +999,18 @@ async def partials(): # quiet user — hand the session to the capture ASR backend so the user # still gets their words, and say which model let them down. Bounded to # this session; the pref is left alone so the user stays in control. - model_silent = is_model_silent(full, heard_speech, len(session_pcm)) + model_silent = is_model_silent(full, heard_speech, session_pcm.total_bytes) if model_silent: - logger.warning( - "dictation model %s decoded NOTHING from %.1fs of speech-level audio " - "— falling back to the capture ASR engine for this session", - spec.id, len(session_pcm) / float(max(1, pcm_sr) * 2), + recovered, recovered_segments = await _recover_silent_sherpa( + spec, session_pcm.tail(), pcm_sr, ) - # Demote it so the NEXT session doesn't repeat this round trip. The - # curated default can be broken on a platform we never tested (the - # NeMo-TDT decoder is, on Windows), and observing it beats guessing. - try: - from services.sherpa_dictation import demote_model - if demote_model(spec.id): - logger.error( - "dictation model %s demoted on this machine — it will no longer be " - "auto-selected. Pick it again in Settings to give it another chance.", - spec.id, - ) - except Exception: - logger.exception("silent-model demotion failed") - try: - result = await _transcribe_buffer_full([bytes(session_pcm)], pcm_sr=pcm_sr) - fb_text = polish_text((result or {}).get("text", "") or "") - if fb_text: - full = fb_text - segments = (result or {}).get("segments") or [ - {"start": 0.0, "end": None, "text": fb_text} - ] - except Exception: - logger.exception("dictation silent-model fallback failed") + if recovered: + full = recovered + segments = recovered_segments if not client_disconnected: - payload = {"type": "final", "text": full, "segments": segments, + payload = {"type": "final", "text": full, "final_kind": "summary", + "segments": segments, "language": "auto", "engine": backend.id} if model_silent: # The client surfaces this so a silently-broken model can't look @@ -884,6 +1038,35 @@ async def partials(): pass +def _result_text(result: dict | None) -> str: + """Normalize text from every ASR backend result shape. + + Some backends return a top-level ``text`` value, while WhisperX, Faster + Whisper, Moonshine, and OpenAI-compatible ASR expose only ``segments`` and + ``chunks``. Dictation partials and finals must interpret both contracts the + same way. + """ + if not isinstance(result, dict): + return "" + + text = result.get("text") + if isinstance(text, str) and text.strip(): + return text.strip() + + for key in ("segments", "chunks"): + items = result.get(key) + if not isinstance(items, (list, tuple)): + continue + text = " ".join( + str(item.get("text", "")).strip() + for item in items + if isinstance(item, dict) and item.get("text") + ).strip() + if text: + return text + return "" + + async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str: """Quick partial transcription of the current audio buffer.""" @@ -898,7 +1081,7 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) def _run(): backend = get_capture_asr_backend() result = backend.transcribe(tmp, word_timestamps=False) - return result.get("text", "") + return _result_text(result) # Bound dictation transcribes (#730): a wedged whisperx/CTranslate2 call # must not hold its GPU-pool worker forever and starve TTS / other ASR @@ -912,7 +1095,9 @@ def _run(): pass -async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict: +async def _transcribe_buffer_full( + chunks: list[bytes], *, pcm_sr: int | None = None, skip_sherpa: bool = False, +) -> dict: """Full transcription with timing info for the final result.""" tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks) if tmp is None: @@ -924,15 +1109,13 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded def _run(): - backend = get_capture_asr_backend() + backend = get_capture_asr_backend(skip_sherpa=skip_sherpa) t0 = time.perf_counter() result = backend.transcribe(tmp, word_timestamps=False) elapsed = round(time.perf_counter() - t0, 2) segments = result.get("segments", []) - full_text = result.get("text", "") - if not full_text and segments: - full_text = " ".join(s.get("text", "") for s in segments).strip() + full_text = _result_text(result) # Wave 1.1: strip Whisper hallucination loops from the final # text (the string that gets auto-pasted). Segments keep the diff --git a/backend/config/models.yaml b/backend/config/models.yaml index c020d65af..0f43e11bc 100644 --- a/backend/config/models.yaml +++ b/backend/config/models.yaml @@ -159,17 +159,16 @@ models: - repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" label: "Parakeet TDT v3 (sherpa-onnx — dictation, 25 EU langs)" role: ASR - size_gb: 0.18 + size_gb: 0.67 engine: sherpa-onnx dictation_id: sherpa-parakeet-tdt-v3 tag: offline - curated_on: [all] - note: "Recommended live-dictation default. CPU, int8 ONNX. Requires sherpa-onnx." + note: "Multilingual European-language dictation. CPU, int8 ONNX. Requires sherpa-onnx." - repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8" label: "Parakeet TDT v2 (sherpa-onnx — dictation, English)" role: ASR - size_gb: 0.17 + size_gb: 0.66 engine: sherpa-onnx dictation_id: sherpa-parakeet-tdt-v2 tag: offline @@ -178,7 +177,7 @@ models: - repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20" label: "Zipformer Bilingual (sherpa-onnx — streaming, zh+en)" role: ASR - size_gb: 0.13 + size_gb: 0.2 engine: sherpa-onnx dictation_id: sherpa-zipformer-bilingual-zh-en tag: streaming @@ -187,7 +186,7 @@ models: - repo_id: "csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en" label: "Paraformer Bilingual (sherpa-onnx — streaming, zh+en)" role: ASR - size_gb: 0.115 + size_gb: 0.24 engine: sherpa-onnx dictation_id: sherpa-paraformer-bilingual-zh-en tag: streaming @@ -196,7 +195,7 @@ models: - repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17" label: "Zipformer Streaming EN 20M (sherpa-onnx — streaming, English)" role: ASR - size_gb: 0.128 + size_gb: 0.044 engine: sherpa-onnx dictation_id: sherpa-zipformer-en-20m tag: streaming @@ -205,7 +204,7 @@ models: - repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23" label: "Zipformer Streaming ZH 14M (sherpa-onnx — streaming, Chinese)" role: ASR - size_gb: 0.074 + size_gb: 0.025 engine: sherpa-onnx dictation_id: sherpa-zipformer-zh-14m tag: streaming @@ -214,11 +213,12 @@ models: - repo_id: "csukuangfj/sherpa-onnx-whisper-tiny" label: "Whisper Tiny (sherpa-onnx — dictation, 90+ langs)" role: ASR - size_gb: 0.116 + size_gb: 0.104 engine: sherpa-onnx dictation_id: sherpa-whisper-tiny tag: offline - note: "Multilingual offline dictation (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx." + curated_on: [all] + note: "Recommended cross-platform dictation default (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx." # ── Diarisation ─────────────────────────────────────────────────────── diff --git a/backend/services/asr_backend.py b/backend/services/asr_backend.py index 5eab36093..ce2b4eda2 100644 --- a/backend/services/asr_backend.py +++ b/backend/services/asr_backend.py @@ -2995,7 +2995,7 @@ def _capture_prefers_parakeet() -> bool: return _parakeet_mlx_installed() -def get_capture_asr_backend() -> ASRBackend: +def get_capture_asr_backend(*, skip_sherpa: bool = False) -> ASRBackend: """Pick the fastest ASR engine for capture / dictation. Selection order: @@ -3020,6 +3020,9 @@ def get_capture_asr_backend() -> ASRBackend: Returns a cached singleton so the model stays warm between calls; the singleton is rebuilt if the selected sherpa model changes. + + ``skip_sherpa`` is used only to validate a token-silent Sherpa result with + the installed capture fallback before persisting model demotion. """ global _capture_backend, _capture_backend_key @@ -3028,7 +3031,7 @@ def get_capture_asr_backend() -> ASRBackend: # call get_sherpa_dictation_backend concurrently) can't both build a model. with _capture_backend_lock: # 0. Honor an explicit sherpa dictation model selection. - sherpa_id = dictation_model_id() + sherpa_id = None if skip_sherpa else dictation_model_id() if sherpa_id: ok, _ = SherpaDictationBackend.is_available() if ok: @@ -3195,7 +3198,10 @@ def _capture_whisper_repo() -> str | None: return os.environ.get("OMNIVOICE_PYTORCH_ASR_MODEL", _PYTORCH_ASR_DEFAULT) -def _recommended_asr_model(purpose: str, missing_repo: str | None) -> dict | None: +def _recommended_asr_model( + purpose: str, missing_repo: str | None, *, prefer_sherpa: bool = True, + excluded_sherpa_model_id: str | None = None, +) -> dict | None: """The catalog entry to offer in the download CTA. Offline: the missing repo itself when it's in the catalog (guarantees @@ -3215,20 +3221,38 @@ def _shape(m: dict) -> dict: by_id = {m["repo_id"]: m for m in KNOWN_MODELS} exact = by_id.get(missing_repo) if missing_repo else None - want_sherpa = False - if purpose == "dictation": - if exact is not None and exact.get("engine") == "sherpa-onnx": + + def _eligible(m: dict, *, sherpa: bool) -> bool: + if (m.get("engine") == "sherpa-onnx") != sherpa: + return False + if sherpa and m.get("dictation_id") == excluded_sherpa_model_id: + return False + return _model_supported(m) + + if purpose != "dictation": + if exact is not None and _model_supported(exact): return _shape(exact) + prefer_sherpa = False + + if purpose == "dictation" and prefer_sherpa: ok, _ = SherpaDictationBackend.is_available() - want_sherpa = ok - if not want_sherpa and exact is not None and _model_supported(exact): + if ok: + if exact is not None and _eligible(exact, sherpa=True): + return _shape(exact) + for m in KNOWN_MODELS: + if (m.get("role") == "ASR" and _eligible(m, sherpa=True) + and _model_curated(m)): + return _shape(m) + + # No usable Sherpa recommendation remains (runtime unavailable, explicit + # fallback probe, or the sole curated entry is the demoted model). Offer + # the exact capture fallback so download → retry cannot loop. + if exact is not None and _eligible(exact, sherpa=False): return _shape(exact) for m in KNOWN_MODELS: if m.get("role") != "ASR": continue - if (m.get("engine") == "sherpa-onnx") != want_sherpa: - continue - if _model_curated(m) and _model_supported(m): + if _eligible(m, sherpa=False) and _model_curated(m): return _shape(m) return None @@ -3259,7 +3283,9 @@ def _repo_installed(repo: str) -> bool: def asr_model_missing_error(*, purpose: str = "transcribe", sherpa_model_id: str | None = None, - backend_id: str | None = None) -> dict | None: + backend_id: str | None = None, + skip_sherpa: bool = False, + require_installed: bool = False) -> dict | None: """None when the active ASR selection can transcribe without downloading anything; otherwise the typed ``{"error": "asr_model_missing", ...}`` payload for a 409 / SSE / WS error with a download CTA. @@ -3271,6 +3297,11 @@ def asr_model_missing_error(*, purpose: str = "transcribe", ``?model=`` override. Installed state comes from the same HF-cache helpers the model store uses (see :func:`_repo_installed`), so the answer matches the Model Catalogue → Models install badges. + ``skip_sherpa`` probes only the non-Sherpa capture fallback; silent-model + recovery uses it before deciding whether persistent demotion is warranted. + ``require_installed`` makes unknown/custom selections fail closed for that + recovery path so it can never turn the normal fail-open policy into an + implicit model download. FAIL-OPEN rule: a repo the model catalog doesn't know (a custom ``ASR_MODEL_*`` pin, pytorch-whisper's default repo, an unrecognized @@ -3280,27 +3311,55 @@ def asr_model_missing_error(*, purpose: str = "transcribe", a broken preflight must degrade to the old behaviour, not block ASR. """ try: + prefer_sherpa_recommendation = not skip_sherpa + excluded_sherpa_model_id = None if purpose == "dictation": - sid = sherpa_model_id or dictation_model_id() + sid = None if skip_sherpa else (sherpa_model_id or dictation_model_id()) if sid: ok, _ = SherpaDictationBackend.is_available() if ok: from services import sherpa_dictation as _sd spec = _sd.get_spec(sid) + # A recognizer observed returning silence must follow the + # same capture fallback as execution, even when the + # frontend keeps sending its persisted `?model=` value. if spec is not None: - if _sd.is_installed(spec): - return None - return { - "error": ASR_MODEL_MISSING, - "missing_repo_id": spec.repo_id, - "recommended": _recommended_asr_model(purpose, spec.repo_id), - } + if _sd.is_demoted(spec.id): + excluded_sherpa_model_id = spec.id + else: + if _sd.is_installed(spec): + return None + return { + "error": ASR_MODEL_MISSING, + "missing_repo_id": spec.repo_id, + "recommended": _recommended_asr_model( + purpose, spec.repo_id, + ), + } repo = _capture_whisper_repo() else: repo = _offline_asr_repo(backend_id) if repo is None: + if require_installed: + return { + "error": ASR_MODEL_MISSING, + "missing_repo_id": "unresolved-capture-fallback", + "recommended": None, + } return None # explicit opt-in engine — can't (and shouldn't) preflight from api.routers.setup.models import get_model_catalog + if require_installed: + if _repo_installed(repo): + return None + return { + "error": ASR_MODEL_MISSING, + "missing_repo_id": repo, + "recommended": _recommended_asr_model( + purpose, repo, + prefer_sherpa=prefer_sherpa_recommendation, + excluded_sherpa_model_id=excluded_sherpa_model_id, + ), + } if get_model_catalog().get(repo) is None: return None # not installable from the CTA — fail open (see docstring) if _repo_installed(repo): @@ -3308,7 +3367,11 @@ def asr_model_missing_error(*, purpose: str = "transcribe", return { "error": ASR_MODEL_MISSING, "missing_repo_id": repo, - "recommended": _recommended_asr_model(purpose, repo), + "recommended": _recommended_asr_model( + purpose, repo, + prefer_sherpa=prefer_sherpa_recommendation, + excluded_sherpa_model_id=excluded_sherpa_model_id, + ), } except Exception: # noqa: BLE001 — preflight is best-effort, never a blocker logger.warning("ASR install preflight failed — proceeding without it", diff --git a/backend/services/sherpa_dictation.py b/backend/services/sherpa_dictation.py index 693475bbf..b135fd0b8 100644 --- a/backend/services/sherpa_dictation.py +++ b/backend/services/sherpa_dictation.py @@ -110,7 +110,7 @@ def streaming(self) -> bool: # the same HF tree API on 2026-08-07 — not estimated. Every one of the seven # was wrong before, and in both directions, which is worse than uniformly # optimistic: the two Parakeets under-reported by ~3.8x (0.18 -> 0.67 GB), -# so the recommended default quietly downloaded four times what the picker +# so installing v3 quietly downloaded four times what the picker # promised on a metered or small-disk machine; but the two low-RAM # zipformers OVER-reported by ~3x (0.128 -> 0.044), making the fallback # models look bulkier than the heavyweights they exist to rescue users @@ -129,7 +129,6 @@ def streaming(self) -> bool: kind="offline-transducer", size_gb=0.67, languages="25 European languages", - recommended=True, heavy=True, model_type="nemo_transducer", files={ @@ -223,6 +222,7 @@ def streaming(self) -> bool: kind="offline-whisper", size_gb=0.104, languages="90+ languages (auto-detect)", + recommended=True, files={ "encoder": "tiny-encoder.int8.onnx", "decoder": "tiny-decoder.int8.onnx", @@ -231,7 +231,7 @@ def streaming(self) -> bool: ), } -DEFAULT_MODEL_ID = "sherpa-parakeet-tdt-v3" +DEFAULT_MODEL_ID = "sherpa-whisper-tiny" # repo_id → model id, so the model-store list (keyed by repo_id) can be # enriched with the dictation metadata, and so capture can map either key. @@ -261,6 +261,16 @@ def sherpa_available() -> tuple[bool, str]: return True, "ready" except ImportError as e: return False, f"sherpa-onnx not installed: {e}. Install with: uv add sherpa-onnx" + except Exception as e: # noqa: BLE001 — an availability probe must fail closed + # Native wheel failures surface as OSError/RuntimeError rather than + # ImportError (missing DLL/dylib/so, loader or runtime init failure) — + # but the set is open-ended: an extension module is free to raise + # anything at init. This is an availability question, so ANY failure to + # import means "not available", never an exception escaping to the + # caller. SherpaDictationBackend.is_available() calls this directly and + # capture_ws.ws_transcribe calls that without a guard, so an unexpected + # type here took the WebSocket down instead of falling back (#1610). + return False, f"sherpa-onnx unavailable ({type(e).__name__}): {e}" def _resolve_model_dir(spec: SherpaModelSpec, *, download: bool = True) -> str: @@ -397,13 +407,10 @@ def p(role: str) -> str: # transcribe the same bytes. It is a defect inside sherpa-onnx that the app # cannot fix by configuration. # -# The curated default therefore cannot be trusted to WORK just because it is -# installed — and which platforms are affected is not knowable up front, so -# hard-coding a different default per OS would only be a guess. Instead the app -# learns from what it observes: when a session hears real speech and the model -# returns nothing, that model is demoted on THIS machine and stops being -# selected. Self-correcting wherever the breakage actually is, and a no-op -# everywhere it isn't. +# Installation alone therefore cannot prove that a recognizer works. When a +# session hears real speech and the model returns nothing, that model is +# demoted on this machine and stops being selected. This self-corrects wherever +# the decoder defect appears and is a no-op everywhere it does not. #: prefs key holding the list of model ids demoted on this machine. PREF_SILENT_MODELS = "dictation.silent_models" diff --git a/backend/tests/test_dictation_model_demotion.py b/backend/tests/test_dictation_model_demotion.py index ade81c741..e9d095d8f 100644 --- a/backend/tests/test_dictation_model_demotion.py +++ b/backend/tests/test_dictation_model_demotion.py @@ -1,16 +1,15 @@ """A dictation model that decodes nothing gets demoted, not re-selected forever. -`sherpa-parakeet-tdt-v3` is the curated default, and on Windows it installs -cleanly, loads without error, and returns an empty token list for clear speech +On Windows, `sherpa-parakeet-tdt-v3` installs cleanly, loads without error, +and returns an empty token list for clear speech (both quantisations, both decoding methods, sherpa-onnx 1.13.3 and 1.13.4) while whisper and zipformer transcribe the same bytes. The defect is inside sherpa-onnx's NeMo-TDT decoder — unfixable from here by configuration. -Hard-coding a different default per OS would be a guess: we have evidence for -one platform only. So the app observes instead. When a session hears real -speech and the model returns nothing, that model is demoted ON THIS MACHINE and -stops being auto-selected, which self-corrects wherever the breakage actually -is and is a no-op everywhere it isn't. +Whisper Tiny is now the cross-platform default, while Parakeet remains +selectable. Runtime demotion still protects users who select a recognizer that +loads successfully but decodes nothing: it is demoted on this machine and the +next session follows the capture fallback. These tests pin the demotion round trip and, critically, that the user can always take back control by re-picking the model. diff --git a/backend/tests/test_dictation_silent_model.py b/backend/tests/test_dictation_silent_model.py index 6ea89ce9c..a7e29d81d 100644 --- a/backend/tests/test_dictation_silent_model.py +++ b/backend/tests/test_dictation_silent_model.py @@ -1,13 +1,13 @@ """A dictation model that decodes NOTHING must fall back, not fail silently. -Found on Windows with the curated default `sherpa-parakeet-tdt-v3`: the model +Found on Windows with `sherpa-parakeet-tdt-v3`: the model downloads, loads with zero errors, and is correctly detected as a TDT model (`num_durations: 5`) — then returns an empty token list for clear speech. Measured against the same 18.9s WAV, on the same machine, same sherpa-onnx: sherpa-whisper-tiny -> "Alright, here we are. I hope that's all..." sherpa-zipformer-en-20m -> "ANTS BOTH IN WHAT DISGUISED THIS THAT..." - parakeet-tdt-v3 (int8) -> '' <-- the curated default + parakeet-tdt-v3 (int8) -> '' parakeet-tdt-v3 (fp32) -> '' parakeet-tdt-v2 (int8) -> '' diff --git a/docs/engines/nemo-parakeet.md b/docs/engines/nemo-parakeet.md index 097be4f6e..0277b18e6 100644 --- a/docs/engines/nemo-parakeet.md +++ b/docs/engines/nemo-parakeet.md @@ -20,8 +20,9 @@ instead — same model family, no NeMo dependency: - **Apple Silicon:** [parakeet-mlx](parakeet-mlx.md) (installed by default on mac-ARM source installs). -- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — its default - dictation model is an int8 ONNX export of Parakeet TDT v3. +- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — selectable + int8 ONNX exports of Parakeet TDT v2/v3; Whisper Tiny remains the + cross-platform dictation default. ## Selecting it diff --git a/docs/engines/sherpa-onnx-asr.md b/docs/engines/sherpa-onnx-asr.md index 765f55532..95a131f68 100644 --- a/docs/engines/sherpa-onnx-asr.md +++ b/docs/engines/sherpa-onnx-asr.md @@ -11,10 +11,10 @@ partials either way. - Ensure `sherpa-onnx` is installed (`uv add sherpa-onnx` on source installs). - Pick a dictation model in the app (Model Catalogue → Models lists the - curated set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or + selectable set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or pin `OMNIVOICE_ASR_BACKEND=sherpa-onnx-asr`. - `OMNIVOICE_SHERPA_ASR_MODEL` selects the model — default - `sherpa-parakeet-tdt-v3`. + `sherpa-whisper-tiny`. ## Best at @@ -25,17 +25,17 @@ partials either way. timestamps, which makes it a dictation/notes tool rather than a dubbing engine. -## The 7 curated models +## The 7 selectable models | Id | Type | Languages | Download | | --- | --- | --- | --- | -| `sherpa-parakeet-tdt-v3` (default) | offline | 25 European languages | 0.67 GB | +| `sherpa-parakeet-tdt-v3` | offline | 25 European languages | 0.67 GB | | `sherpa-parakeet-tdt-v2` | offline | English | 0.66 GB | | `sherpa-zipformer-bilingual-zh-en` | streaming | Chinese + English | 0.20 GB | | `sherpa-paraformer-bilingual-zh-en` | streaming | Chinese + English | 0.24 GB | | `sherpa-zipformer-en-20m` | streaming | English | 0.044 GB | | `sherpa-zipformer-zh-14m` | streaming | Chinese | 0.025 GB | -| `sherpa-whisper-tiny` | offline | 90+ languages (auto-detect) | 0.104 GB | +| `sherpa-whisper-tiny` (default, recommended) | offline | 90+ languages (auto-detect) | 0.104 GB | Sizes are measured on-disk download sizes. Weights are int8 ONNX checkpoints that download on first use through the same HF cache as everything else — @@ -45,7 +45,10 @@ allocator holds onto freed blocks). ## Platform support -CPU on every platform, by the strict cross-platform default-parity rule. +CPU on every platform, by the strict cross-platform default-parity rule. The +[upstream CPU wheels](https://k2-fsa.github.io/sherpa/onnx/python/install.html) +cover Linux, macOS, and Windows, and upstream documents Whisper as a +[supported non-streaming model family](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/whisper/index.html). `OMNIVOICE_SHERPA_ASR_PROVIDER` can override the ONNX provider on a verified GPU build, but the default never diverges. diff --git a/docs/features.yaml b/docs/features.yaml index f55e4ec9b..4c9d1b4da 100644 --- a/docs/features.yaml +++ b/docs/features.yaml @@ -95,4 +95,5 @@ docs: - docs/install/linux.md - docs/install/docker.md - docs/install/troubleshooting.md + - docs/features/dictation.md - docs/migration/real-time-voice-cloning.md diff --git a/docs/features/dictation.md b/docs/features/dictation.md new file mode 100644 index 000000000..a9cc111db --- /dev/null +++ b/docs/features/dictation.md @@ -0,0 +1,63 @@ +# Dictation + +VoiceStudio dictation records from the system-wide shortcut, transcribes +locally, and—where the desktop permits it—inserts the result into the app where +the shortcut was pressed. The pill never needs keyboard focus. + +## Use it + +1. Choose an installed dictation model in the Model Catalogue. +2. Set the shortcut and hold/toggle behavior in **Settings → Hotkey**. +3. Put the cursor in a text field, press the shortcut, speak, then release or + press again. + +Whisper Tiny is the recommended default on macOS, Windows, and Linux. It +auto-detects more than 90 languages. Parakeet TDT v3 remains available for its +25 supported European languages, but it is not selected automatically. + +The pill reports **Inserted** only after native delivery succeeds. **Copied** +means automatic insertion was unavailable and the complete final transcript is +ready for a normal paste. VoiceStudio retries a speech-level empty Sherpa decode +only through another ASR model whose weights are already installed; when that +fallback confirms the audio contains words, the silent model is demoted. This +recovery never starts a download. + +## Destination and clipboard safety + +The desktop captures the target at shortcut-down and carries its session ID +through partial, utterance, and summary messages. A late result from an older +session cannot use a newer session's target. macOS, Windows, and X11 validate +and reactivate the captured process/window before insertion. Wayland does not +expose a portable target identity or arbitrary foreign-window activation, so +the safe default leaves the transcript copied instead of guessing which app +should receive it. The GTK pill remains non-focusable. + +For paste delivery, VoiceStudio snapshots text, HTML (with its plain-text +alternative), image, or file-list clipboard content, stages the transcript, +and restores the snapshot after the target consumes it. +Streaming segments share a generation-tracked lease: a stale restore cannot +win over a newer segment, and VoiceStudio never overwrites clipboard content +you copied during transcription. Unsupported clipboard formats cannot be +round-tripped; in that case the transcript remains on the clipboard instead of +attempting a lossy restore. + +## Platform behavior + +| Platform | Automatic insertion | +| --- | --- | +| macOS | Reactivates the captured application and sends Command-V. Without Accessibility permission, the result stays copied. | +| Windows | Validates the captured window and process, requests foreground activation, then sends Ctrl-V. If Windows denies activation, the result stays copied. | +| Linux X11 | Reactivates the captured X11 window through EWMH, verifies it, then sends Ctrl-V. | +| Wayland | Leaves the complete transcript copied because a portable captured-window identity is unavailable. | +| Browser mode | Copies the transcript; browsers cannot target another desktop app. | + +Advanced Wayland users can set `VOICESTUDIO_WAYLAND_UNTARGETED_INSERT=1` to +insert into whichever client owns keyboard focus when transcription finishes. +wlroots compositors use `wtype`; KDE Plasma and GNOME can use clipboard paste +through `dotool` or `ydotool`. Helpers run with host loader variables from an +AppImage, have a bounded timeout, and never retry after one may have emitted +partial input. This opt-in cannot promise the shortcut-down target if focus +changes. `dotool` needs direct write access to `/dev/uinput`; `ydotool` 1.0+ +needs a running `ydotoold` with that access and a user-readable socket. +VoiceStudio checks these prerequisites before selection. Tray-started Wayland +dictation always stays copy-only. diff --git a/docs/install/linux.md b/docs/install/linux.md index 7021c6479..5ba62924a 100644 --- a/docs/install/linux.md +++ b/docs/install/linux.md @@ -100,6 +100,22 @@ where the protocol gives applications no say in their own placement and the compositor decides where it appears. The capsule works the same either way; only its position is out of the app's hands there. +Wayland does not expose a portable identity for the app focused at shortcut +down, so VoiceStudio safely leaves the complete transcript on the clipboard and +the pill says **Copied** instead of risking insertion into a different app. + +Advanced users can opt into current-focus insertion with +`VOICESTUDIO_WAYLAND_UNTARGETED_INSERT=1`. wlroots compositors such as Sway and +Hyprland use `wtype`; KDE Plasma and GNOME can use `dotool` or `ydotool` to +paste the Unicode clipboard payload. The +opt-in targets whichever client owns keyboard focus when transcription +finishes, not necessarily the app where dictation started. Tray-started +dictation remains copy-only. `dotool` needs direct write access to +`/dev/uinput` (normally through a distribution udev rule/group). `ydotool` +1.0+ needs the `ydotoold` daemon running with that access and its socket +available to the desktop user. VoiceStudio skips either helper when its +readiness check fails. + If the global shortcut stops working, restart your desktop's portal service, then save the shortcut again in **Settings → Hotkey** to reopen consent. Portal packages and support vary by desktop; use the backend recommended by your diff --git a/docs/specs/2026-07-16-dictation-flow-program.md b/docs/specs/2026-07-16-dictation-flow-program.md index 8221174a5..c4d212a7c 100644 --- a/docs/specs/2026-07-16-dictation-flow-program.md +++ b/docs/specs/2026-07-16-dictation-flow-program.md @@ -1,14 +1,14 @@ -# Dictation Flow Program — local WhisperFlow-class dictation on Parakeet +# Dictation Flow Program — local cross-platform flow dictation -*Spec, 2026-07-16. Research inputs: three-agent study — product landscape (Wispr Flow, jamiepine/voicebox, Handy, VoiceInk, Whispering, Talon, Claude Code `/voice`), in-repo capability map, and Parakeet TDT/Nemotron feasibility (sherpa-onnx). Sources cited inline where load-bearing.* +*Spec, 2026-07-16. Research inputs: a multi-agent product-landscape study, in-repo capability map, and local ASR feasibility review. Sources are cited inline where load-bearing.* ## Why -Dictating prompts to AI agents is the fastest-growing text-input workload (Claude Code shipped built-in `/voice`; Wispr Flow raised at ~$2B on it) — and every polished option is **cloud** (Wispr: cloud-only, no Linux, one privacy scandal already; Claude Code voice: cloud-only, no SSH). The best open competitor, **jamiepine/voicebox** (41.7k★, MIT — our refinement layer is already adapted from it), only ships reliable auto-paste on macOS. VoiceStudio already has the hard parts: a Wispr-style pill, global hotkey, sherpa-onnx streaming WS, **Parakeet TDT v3 int8 as the shipped default**, clipboard-restoring paste, and local-LLM refinement. A local, cross-platform, private flow-dictation experience is reachable and strategically differentiating — the wedge is **local + Linux/Wayland + agent-prompting**, where nobody credible plays. +Dictating prompts to AI agents is a rapidly growing text-input workload, while polished options remain cloud-first and Linux support is uneven. VoiceStudio already has the hard parts: a compact capture pill, global hotkey, sherpa-onnx streaming WS, **Whisper Tiny int8 as the shipped cross-platform default**, clipboard-restoring paste, and local-LLM refinement. A local, cross-platform, private flow-dictation experience is reachable and strategically differentiating — the wedge is **local + Linux/Wayland + agent-prompting**. ## Current state (verified in-repo) -Widget: pill webview + `tauri-plugin-global-shortcut` (`CmdOrCtrl+Shift+Space`, toggle/hold) on macOS, Windows and X11 + the GlobalShortcuts desktop portal on Wayland + browser-mode keyboard fallback; `getUserMedia` → raw-PCM WS `/ws/transcribe`; paste via arboard+enigo with clipboard restore, macOS a11y fail-loud, Windows no-activate. Backend: 7 sherpa models (Parakeet TDT v3 default), streaming path (zipformer/paraformer) + chunked-offline path (0.8 s partial cadence, **RMS silence gate**), `text_polish` on finals, opt-in LLM refinement (Ollama/LM Studio, ≤4 s wall clock). Gaps: no real VAD, no dictionary/hotwords, no per-app awareness, no command grammar, no language picker, enigo-only Linux insertion, no comprehensive dictation feature guide beyond the Linux installation note, picker understates model size ~4×. +Widget: pill webview + `tauri-plugin-global-shortcut` (`CmdOrCtrl+Shift+Space`, toggle/hold) on macOS, Windows and X11 + the GlobalShortcuts desktop portal on Wayland + browser-mode keyboard fallback; `getUserMedia` → 16 kHz raw-PCM WS `/ws/transcribe`; a native session captures the destination before the pill appears, restores an untouched clipboard by generation, reactivates macOS/Windows/X11 targets, and uses a truthful copy fallback on Wayland unless current-focus insertion is explicitly enabled. Backend: 7 sherpa models (Whisper Tiny default), streaming path (zipformer/paraformer) + chunked-offline path (0.8 s partial cadence, **RMS silence gate**), shared speech-evidence model demotion with installed-only ASR fallback, `text_polish` on finals, opt-in LLM refinement (Ollama/LM Studio, ≤4 s wall clock). Gaps: no real VAD, no dictionary/hotwords, no per-app formatting profiles, no command grammar, no language picker, picker understates model size ~4×. ## Program phases @@ -41,7 +41,7 @@ Parakeet's one real weakness is OOV technical terms — and the dictionary is Wi ### Phase 4 — insertion reliability + Wayland (beat everyone on Linux) - **Reliability engineering** (the boring 20% that reads professional; Wispr does 5 retries): retry-with-backoff on paste, transcript stays on clipboard + toast on failure, password-field refusal, Windows elevated-window detection. -- **Wayland insertion chain** replacing bare enigo on Linux: kwtype→wtype→dotool→ydotool→wl-copy+notify fallback (Handy's proven cascade), IBus/Fcitx5 input-method commit path evaluated for GNOME (highest quality, nobody mainstream ships it), libei/RemoteDesktop-portal as the forward bet. Wispr has no Linux at all; voicebox has no Linux paste — this is the moat. +- **Wayland insertion chain** replacing bare enigo on Linux: wtype→dotool→ydotool→wl-copy+notify fallback, IBus/Fcitx5 input-method commit path evaluated for GNOME (highest quality, nobody mainstream ships it), libei/RemoteDesktop-portal as the forward bet. Reliable local Linux insertion is the moat. ### Phase 5 — command mode (headline, local-only differentiator) Second hotkey → speak an instruction over selected text → local LLM rewrite → explicit Apply. Wispr charges for this; ours is local and free. Requires configured LLM; hidden otherwise (existing `llm_ready` plumbing). @@ -56,10 +56,10 @@ Second hotkey → speak an instruction over selected text → local LLM rewrite | Use case | Model | Partials | Final after pause | Disk/RAM | |---|---|---|---|---| | English, best feel | nemotron-streaming-en 160 ms (new, Ph. 1) | 200–400 ms | ~0.5–0.7 s | 0.66 GB / ~1.2 GB | -| Multilingual default | parakeet-tdt-v3 + silero-VAD (upgraded path) | 0.8 s cadence | ~0.4–0.7 s | 0.67 GB / ~1.2 GB | +| European languages (opt-in) | parakeet-tdt-v3 + silero-VAD (upgraded path) | 0.8 s cadence | ~0.4–0.7 s | 0.67 GB / ~1.2 GB | | Multilingual streaming (opt-in) | Nemotron-3.5 320 ms (new) | ~400 ms | ~0.7 s | 0.68 GB / ~1.2 GB | | Low-RAM | zipformer-20M (existing) | ~100 ms | ~0.6 s | 0.13 GB / ~0.3 GB | -| CJK / 90+ langs | whisper-tiny (existing; consider small) | n/a | seconds | 0.12 GB | +| Multilingual default / CJK | whisper-tiny (existing; consider small) | n/a | seconds | 0.104 GB | ## Top risks diff --git a/frontend/src-tauri/Cargo.lock b/frontend/src-tauri/Cargo.lock index 5f1e1839f..5528513c2 100644 --- a/frontend/src-tauri/Cargo.lock +++ b/frontend/src-tauri/Cargo.lock @@ -101,6 +101,7 @@ dependencies = [ "parking_lot", "percent-encoding", "windows-sys 0.60.2", + "wl-clipboard-rs", "x11rb", ] @@ -1040,6 +1041,12 @@ dependencies = [ "tendril", ] +[[package]] +name = "downcast-rs" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75b325c5dbd37f80359721ad39aca5a29fb04c89279657cffdda8736d0c0b9d2" + [[package]] name = "dpi" version = "0.1.2" @@ -1287,6 +1294,12 @@ version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + [[package]] name = "flate2" version = "1.1.9" @@ -2582,6 +2595,15 @@ version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086" +[[package]] +name = "nom" +version = "8.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405" +dependencies = [ + "memchr", +] + [[package]] name = "ntapi" version = "0.4.3" @@ -2687,6 +2709,7 @@ checksum = "d49e936b501e5c5bf01fda3a9452ff86dc3ea98ad5f283e1455153142d97518c" dependencies = [ "bitflags 2.13.0", "block2 0.6.2", + "libc", "objc2 0.6.4", "objc2-core-foundation", "objc2-core-graphics", @@ -2948,8 +2971,11 @@ dependencies = [ "enigo", "fs4", "getrandom 0.3.4", + "gtk", "libc", "log", + "objc2 0.6.4", + "objc2-app-kit 0.3.2", "reqwest", "semver", "serde", @@ -2973,6 +2999,7 @@ dependencies = [ "webview2-com", "windows 0.61.3", "windows-core 0.61.2", + "x11rb", "zbus", "zip 2.4.2", ] @@ -3017,6 +3044,16 @@ dependencies = [ "pin-project-lite", ] +[[package]] +name = "os_pipe" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" +dependencies = [ + "libc", + "windows-sys 0.45.0", +] + [[package]] name = "osakit" version = "0.3.1" @@ -3097,6 +3134,17 @@ version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" +[[package]] +name = "petgraph" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" +dependencies = [ + "fixedbitset", + "hashbrown 0.15.5", + "indexmap 2.14.0", +] + [[package]] name = "phf" version = "0.13.1" @@ -3181,7 +3229,7 @@ checksum = "092791278e026273c1b65bbdcfbba3a300f2994c896bd01ab01da613c29c46f1" dependencies = [ "base64 0.22.1", "indexmap 2.14.0", - "quick-xml", + "quick-xml 0.39.4", "serde", "time", ] @@ -3369,6 +3417,15 @@ dependencies = [ "memchr", ] +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", +] + [[package]] name = "quinn" version = "0.11.9" @@ -5285,6 +5342,17 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "tree_magic_mini" +version = "3.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8765b90061cba6c22b5831f675da109ae5561588290f9fa2317adab2714d5a6" +dependencies = [ + "memchr", + "nom", + "petgraph", +] + [[package]] name = "try-lock" version = "0.2.5" @@ -5633,6 +5701,76 @@ dependencies = [ "semver", ] +[[package]] +name = "wayland-backend" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38a91b4eaddff87b1cd1074985e3713da4af2c49742d1b356b2c01670a67a078" +dependencies = [ + "cc", + "downcast-rs", + "rustix", + "smallvec", + "wayland-sys", +] + +[[package]] +name = "wayland-client" +version = "0.31.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3c36a0f861ad76d0901f2800b46321410d9f73f2ea88aac0650d86c32688073" +dependencies = [ + "bitflags 2.13.0", + "rustix", + "wayland-backend", + "wayland-scanner", +] + +[[package]] +name = "wayland-protocols" +version = "0.32.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23d0c813de3daa2ed6520af85a3bd49b0e722a3078506899aa9686fea58dc4b6" +dependencies = [ + "bitflags 2.13.0", + "wayland-backend", + "wayland-client", + "wayland-scanner", +] + +[[package]] +name = "wayland-protocols-wlr" +version = "0.3.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb04e52f7836d7c7976c78ca0250d61e33873c34156a2a1fc9474828ec268234" +dependencies = [ + "bitflags 2.13.0", + "wayland-backend", + "wayland-client", + "wayland-protocols", + "wayland-scanner", +] + +[[package]] +name = "wayland-scanner" +version = "0.31.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "338e30461b3a2b67d70eb30a6d89f8e0c93a833e07d2ae89085cd070c4a00ac0" +dependencies = [ + "proc-macro2", + "quick-xml 0.41.0", + "quote", +] + +[[package]] +name = "wayland-sys" +version = "0.31.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8eab23fefc9e41f8e841df4a9c707e8a8c4ed26e944ef69297184de2785e3be" +dependencies = [ + "pkg-config", +] + [[package]] name = "web-sys" version = "0.3.102" @@ -6468,6 +6606,24 @@ dependencies = [ "wasmparser", ] +[[package]] +name = "wl-clipboard-rs" +version = "0.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9651471a32e87d96ef3a127715382b2d11cc7c8bb9822ded8a7cc94072eb0a3" +dependencies = [ + "libc", + "log", + "os_pipe", + "rustix", + "thiserror 2.0.18", + "tree_magic_mini", + "wayland-backend", + "wayland-client", + "wayland-protocols", + "wayland-protocols-wlr", +] + [[package]] name = "writeable" version = "0.6.3" diff --git a/frontend/src-tauri/Cargo.toml b/frontend/src-tauri/Cargo.toml index 59ef8cc6a..a33217463 100644 --- a/frontend/src-tauri/Cargo.toml +++ b/frontend/src-tauri/Cargo.toml @@ -67,7 +67,13 @@ dirs-next = "2" # Native (OS-side) clipboard write for dictation auto-paste: the widget window # is unfocused on macOS so the simulated ⌘V reaches the target app, which makes # the WebView clipboard APIs fail silently there (#287) -arboard = "3" +arboard = { version = "3", features = ["wayland-data-control"] } + +[target.'cfg(target_os = "macos")'.dependencies] +# Capture the frontmost application at shortcut-down and reactivate that exact +# process before transcript delivery. +objc2 = "0.6" +objc2-app-kit = { version = "0.3", default-features = false, features = ["std", "libc", "NSRunningApplication", "NSWorkspace"] } [target.'cfg(windows)'.dependencies] zip = { version = "2", default-features = false, features = ["deflate"] } @@ -84,13 +90,15 @@ windows-core = "0.61" # `HWND` type — no second copy of the crate enters the dependency graph. # Win32_System_Registry: check_microphone reads the CapabilityAccessManager # ConsentStore mic toggle (RegGetValueW) for the permissions UX. -windows = { version = "0.61", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_System_Registry"] } +windows = { version = "0.61", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_System_Registry", "Win32_System_Threading"] } [target.'cfg(unix)'.dependencies] libc = "0.2" [target.'cfg(target_os = "linux")'.dependencies] webkit2gtk = "2.0" +gtk = "0.18" +x11rb = "0.13" # Wayland compositors do not expose global keys through X11 grabs. Use the # standard xdg-desktop-portal GlobalShortcuts interface there; zbus is already # present transitively through Tauri's opener/single-instance plugins. diff --git a/frontend/src-tauri/src/commands.rs b/frontend/src-tauri/src/commands.rs index 2d0c98708..d55303487 100644 --- a/frontend/src-tauri/src/commands.rs +++ b/frontend/src-tauri/src/commands.rs @@ -7,13 +7,13 @@ use std::time::{Duration, Instant}; use serde::{Deserialize, Serialize}; use tauri::image::Image; -use tauri::Manager; +use tauri::{Emitter, Manager}; use tauri_plugin_dialog::DialogExt; -use crate::dictation_shortcut::{DictationShortcutManager, ShortcutInfo, update_tray_hint}; +use crate::config::{load_config, save_config}; +use crate::dictation_shortcut::{update_tray_hint, DictationShortcutManager, ShortcutInfo}; use crate::{AppFlags, TrayHandle}; use crate::{TRAY_ICON_DEFAULT, TRAY_ICON_RECORDING}; -use crate::config::{load_config, save_config}; // ── Native host-path authorization ─────────────────────────────────────── @@ -65,10 +65,7 @@ fn remember_reveal_path( Ok(()) } -fn reveal_path_is_authorized( - app: &tauri::AppHandle, - target: &Path, -) -> bool { +fn reveal_path_is_authorized(app: &tauri::AppHandle, target: &Path) -> bool { if let Ok(data_root) = fs::canonicalize( crate::setup::resolved_data_dir(app).unwrap_or_else(crate::setup::default_data_dir), ) { @@ -89,12 +86,7 @@ fn reveal_path_is_authorized( fn validate_host_path(kind: &str, path: PathBuf) -> Result { if !matches!( kind, - "models_dir" - | "ffmpeg" - | "ffprobe" - | "dub_export" - | "soni_input" - | "soni_output_dir" + "models_dir" | "ffmpeg" | "ffprobe" | "dub_export" | "soni_input" | "soni_output_dir" ) { return Err("Unsupported host-path capability".into()); } @@ -216,8 +208,11 @@ pub async fn authorize_host_path( kind, path: validated.to_string_lossy().into_owned(), }; - fs::write(&target, serde_json::to_vec(&payload).map_err(|e| e.to_string())?) - .map_err(|e| format!("Could not authorize path: {e}"))?; + fs::write( + &target, + serde_json::to_vec(&payload).map_err(|e| e.to_string())?, + ) + .map_err(|e| format!("Could not authorize path: {e}"))?; #[cfg(unix)] { use std::os::unix::fs::PermissionsExt; @@ -247,7 +242,10 @@ mod host_path_authorization_tests { #[test] fn empty_models_path_is_the_authorized_default_reset() { - assert_eq!(validate_host_path("models_dir", PathBuf::new()).unwrap(), PathBuf::new()); + assert_eq!( + validate_host_path("models_dir", PathBuf::new()).unwrap(), + PathBuf::new() + ); } #[test] @@ -259,11 +257,9 @@ mod host_path_authorization_tests { destination, ); assert!(validate_host_path("dub_export", PathBuf::from("relative/export.wav")).is_err()); - assert!(validate_host_path( - "dub_export", - parent.join("missing-directory/export.wav"), - ) - .is_err()); + assert!( + validate_host_path("dub_export", parent.join("missing-directory/export.wav"),).is_err() + ); } } @@ -316,12 +312,14 @@ pub fn read_log_tail(source: String, tail: Option) -> LogTailPayload { let path = match source.as_str() { "backend" => backend_runtime_log_path(), "tauri" => tauri_log_path(), - _ => return LogTailPayload { - lines: vec![], - path: String::new(), - exists: false, - total_lines: 0, - }, + _ => { + return LogTailPayload { + lines: vec![], + path: String::new(), + exists: false, + total_lines: 0, + } + } }; let path_str = path.to_string_lossy().to_string(); @@ -363,15 +361,11 @@ fn backend_runtime_log_path() -> PathBuf { let data_dir = if cfg!(target_os = "macos") { dirs_data_dir().join("OmniVoice") } else if cfg!(target_os = "windows") { - PathBuf::from( - std::env::var("APPDATA").unwrap_or_else(|_| ".".to_string()), - ) - .join("OmniVoice") + PathBuf::from(std::env::var("APPDATA").unwrap_or_else(|_| ".".to_string())) + .join("OmniVoice") } else { - PathBuf::from( - std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()), - ) - .join(".omnivoice") + PathBuf::from(std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string())) + .join(".omnivoice") }; data_dir.join("omnivoice.log") } @@ -379,16 +373,12 @@ fn backend_runtime_log_path() -> PathBuf { fn dirs_data_dir() -> PathBuf { #[cfg(target_os = "macos")] { - PathBuf::from( - std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()), - ) - .join("Library/Application Support") + PathBuf::from(std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string())) + .join("Library/Application Support") } #[cfg(not(target_os = "macos"))] { - PathBuf::from( - std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()), - ) + PathBuf::from(std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string())) } } @@ -403,7 +393,10 @@ fn tauri_log_path() -> PathBuf { .join("tauri.log") } else if cfg!(target_os = "windows") { let appdata = std::env::var("APPDATA").unwrap_or_else(|_| home.clone()); - PathBuf::from(appdata).join(bid).join("logs").join("tauri.log") + PathBuf::from(appdata) + .join(bid) + .join("logs") + .join("tauri.log") } else { PathBuf::from(&home) .join(".local/share") @@ -508,22 +501,16 @@ fn hf_hub_cache_dir() -> PathBuf { .join("hub") } -// ── Simulate paste ──────────────────────────────────────────────────────── - -use enigo::{Direction, Enigo, Key, Keyboard, Settings as EnigoSettings}; +// ── Dictation output ───────────────────────────────────────────────────── /// Error-kind builder the dictation widget switches on. Kinds are a plain -/// string prefix ("a11y:" | "clipboard:" | "paste:") so the JS side can do -/// `err.split(':')[0]` without a serde enum crossing the IPC boundary. +/// string prefix ("a11y:" | "clipboard:" | "paste:" | "preflight:") so the +/// JS side can do `err.split(':')[0]` without a serde enum crossing the IPC +/// boundary. fn kind_err(kind: &str, detail: impl std::fmt::Display) -> String { format!("{kind}:{detail}") } -/// How long the transcript must sit on the clipboard before the user's -/// previous clipboard is restored: ~300ms covers slow paste consumers -/// (Electron apps, remote desktops) without being user-noticeable. -const CLIPBOARD_RESTORE_DELAY: Duration = Duration::from_millis(300); - /// macOS Accessibility grant check — CGEvent key synthesis silently no-ops /// without it. Direct FFI against ApplicationServices: one symbol, not worth /// a crate. @@ -705,7 +692,12 @@ pub fn open_microphone_settings() -> Result<(), String> { .arg("x-apple.systempreferences:com.apple.preference.security?Privacy_Microphone") .spawn() .map(|_| ()) - .map_err(|e| kind_err("settings", format!("failed to open microphone settings: {e}"))) + .map_err(|e| { + kind_err( + "settings", + format!("failed to open microphone settings: {e}"), + ) + }) } #[cfg(target_os = "windows")] { @@ -718,7 +710,12 @@ pub fn open_microphone_settings() -> Result<(), String> { .creation_flags(0x0800_0000) // CREATE_NO_WINDOW .spawn() .map(|_| ()) - .map_err(|e| kind_err("settings", format!("failed to open microphone settings: {e}"))) + .map_err(|e| { + kind_err( + "settings", + format!("failed to open microphone settings: {e}"), + ) + }) } #[cfg(not(any(target_os = "macos", target_os = "windows")))] { @@ -744,7 +741,10 @@ pub fn open_input_monitoring_settings() -> Result<(), String> { .spawn() .map(|_| ()) .map_err(|e| { - kind_err("settings", format!("failed to open input monitoring settings: {e}")) + kind_err( + "settings", + format!("failed to open input monitoring settings: {e}"), + ) }) } #[cfg(not(target_os = "macos"))] @@ -757,71 +757,44 @@ pub fn open_input_monitoring_settings() -> Result<(), String> { } #[tauri::command] -pub fn simulate_paste(text: Option) -> Result<(), String> { - // macOS: fail loud BEFORE touching the clipboard if Accessibility isn't - // granted — otherwise the ⌘V below silently goes nowhere and the caller - // can't tell (the old fire-and-forget behavior). +pub async fn simulate_paste( + text: String, + session_id: u64, + flags: tauri::State<'_, AppFlags>, +) -> Result { + // macOS: a revoked/missing Accessibility grant prevents synthesis, but it + // must not discard the result. Keep the full transcript copied and report + // the fallback truthfully. #[cfg(target_os = "macos")] if !accessibility_trusted() { - return Err(kind_err("a11y", "accessibility permission not granted")); - } - - // Write the transcript to the clipboard natively first: the widget window - // is intentionally unfocused on macOS (so the simulated ⌘V reaches the - // target app), which makes the WebView clipboard APIs (navigator.clipboard - // / execCommand('copy')) fail silently there (#287). `text` is optional so - // call sites that already populated the clipboard keep working. - // - // Save what the user had there first (text only — restoring images/files - // isn't worth the platform-specific surface) so dictation doesn't clobber - // their clipboard. - let mut saved: Option = None; - if let Some(t) = text { - let mut cb = arboard::Clipboard::new() - .map_err(|e| kind_err("clipboard", format!("init failed: {e}")))?; - saved = cb.get_text().ok(); - cb.set_text(t) - .map_err(|e| kind_err("clipboard", format!("write failed: {e}")))?; - } - - std::thread::sleep(Duration::from_millis(80)); - - let mut enigo = Enigo::new(&EnigoSettings::default()) - .map_err(|e| kind_err("paste", format!("failed to init keyboard sim: {e}")))?; - - #[cfg(target_os = "macos")] - { - enigo.key(Key::Meta, Direction::Press) - .map_err(|e| kind_err("paste", format!("key press failed: {e}")))?; - enigo.key(Key::Unicode('v'), Direction::Click) - .map_err(|e| kind_err("paste", format!("key click failed: {e}")))?; - enigo.key(Key::Meta, Direction::Release) - .map_err(|e| kind_err("paste", format!("key release failed: {e}")))?; - } - - #[cfg(not(target_os = "macos"))] - { - enigo.key(Key::Control, Direction::Press) - .map_err(|e| kind_err("paste", format!("key press failed: {e}")))?; - enigo.key(Key::Unicode('v'), Direction::Click) - .map_err(|e| kind_err("paste", format!("key click failed: {e}")))?; - enigo.key(Key::Control, Direction::Release) - .map_err(|e| kind_err("paste", format!("key release failed: {e}")))?; - } - - // Best-effort restore of the user's clipboard once the target app has - // consumed the paste. Only on success — on a paste error the transcript - // stays on the clipboard so the user can ⌘V it manually as a fallback. - if let Some(prev) = saved { - std::thread::spawn(move || { - std::thread::sleep(CLIPBOARD_RESTORE_DELAY); - if let Ok(mut cb) = arboard::Clipboard::new() { - let _ = cb.set_text(prev); - } - }); - } + let output = flags.output.clone(); + return tauri::async_runtime::spawn_blocking(move || { + output.copy_for_session(session_id, &text) + }) + .await + .map_err(|error| kind_err("clipboard", format!("output worker failed: {error}")))?; + } + + let output = flags.output.clone(); + tauri::async_runtime::spawn_blocking(move || output.deliver(session_id, &text)) + .await + .map_err(|error| kind_err("paste", format!("output worker failed: {error}")))? +} - Ok(()) +/// Preserve the authoritative transcript without emitting any keyboard input. +/// Used after live typing may have left an unknown prefix in the target: a +/// second insertion would duplicate text, but losing the complete result is +/// not an acceptable fallback. +#[tauri::command] +pub async fn copy_dictation_output_session( + text: String, + session_id: u64, + flags: tauri::State<'_, AppFlags>, +) -> Result { + let output = flags.output.clone(); + tauri::async_runtime::spawn_blocking(move || output.copy_for_session(session_id, &text)) + .await + .map_err(|error| kind_err("clipboard", format!("output worker failed: {error}")))? } // ── Simulate live typing ────────────────────────────────────────────────── @@ -833,18 +806,22 @@ pub fn simulate_paste(text: Option) -> Result<(), String> { /// revised), then `text` is typed. Either may be empty/zero, so a single call /// can correct-then-type in one round trip. /// -/// Cross-platform: `enigo`'s `.text()` synthesizes Unicode key events on macOS -/// (CGEvent), Windows (`SendInput` w/ `KEYEVENTF_UNICODE`), and Linux (X11/ -/// libei). Backspace is a plain virtual-key `Click`, identical on all three. -/// On macOS this reuses the SAME accessibility permission `simulate_paste` -/// already requires (both go through `enigo` → CGEvent); no new grant needed. +/// The session-bound output layer reactivates the destination captured at +/// shortcut-down before emitting input. On Wayland it selects one compatible +/// compositor helper before emission and never retries after a possible +/// partial write. /// /// Returns `Err` if the input layer is unavailable (e.g. accessibility not -/// granted) so the JS caller can fall back to the clipboard+paste path for -/// that segment without double-inserting. Errors carry the same kind -/// prefixes as `simulate_paste` ("a11y:" | "paste:"). +/// granted). Because a failed input call may already have emitted a prefix, +/// the JS caller suppresses later insertion for that session. `preflight:` +/// explicitly means nothing was emitted and a final paste remains safe. #[tauri::command] -pub fn simulate_type(text: Option, backspaces: Option) -> Result<(), String> { +pub async fn simulate_type( + text: String, + backspaces: Option, + session_id: u64, + flags: tauri::State<'_, AppFlags>, +) -> Result { // Same a11y gate as simulate_paste — `.text()`/`.key()` go through the // identical CGEvent path on macOS and would silently no-op without it. #[cfg(target_os = "macos")] @@ -852,24 +829,46 @@ pub fn simulate_type(text: Option, backspaces: Option) -> Result<() return Err(kind_err("a11y", "accessibility permission not granted")); } - let mut enigo = Enigo::new(&EnigoSettings::default()) - .map_err(|e| kind_err("paste", format!("failed to init keyboard sim: {e}")))?; + let output = flags.output.clone(); + tauri::async_runtime::spawn_blocking(move || { + output.type_delta(session_id, &text, backspaces.unwrap_or(0)) + }) + .await + .map_err(|error| kind_err("paste", format!("output worker failed: {error}")))? +} - let n = backspaces.unwrap_or(0); - for _ in 0..n { - enigo - .key(Key::Backspace, Direction::Click) - .map_err(|e| kind_err("paste", format!("backspace failed: {e}")))?; - } +#[tauri::command] +pub async fn activate_dictation_output_session( + session_id: u64, + flags: tauri::State<'_, AppFlags>, +) -> Result<(), String> { + let output = flags.output.clone(); + tauri::async_runtime::spawn_blocking(move || output.activate_session(session_id)) + .await + .map_err(|error| kind_err("paste", format!("output worker failed: {error}")))? +} - if let Some(t) = text { - if !t.is_empty() { - enigo - .text(&t) - .map_err(|e| kind_err("paste", format!("type failed: {e}")))?; - } - } +#[tauri::command] +pub async fn reject_dictation_output_session( + session_id: u64, + flags: tauri::State<'_, AppFlags>, +) -> Result<(), String> { + let output = flags.output.clone(); + tauri::async_runtime::spawn_blocking(move || output.reject_session_candidate(session_id)) + .await + .map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?; + Ok(()) +} +#[tauri::command] +pub async fn finish_dictation_output_session( + session_id: u64, + flags: tauri::State<'_, AppFlags>, +) -> Result<(), String> { + let output = flags.output.clone(); + tauri::async_runtime::spawn_blocking(move || output.finish_session(session_id)) + .await + .map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?; Ok(()) } @@ -889,11 +888,16 @@ pub fn set_tray_recording( // permanently-hidden widget made meaningless.) flags.dictating.store(recording, Ordering::SeqCst); log::info!("Dictation recording state: {recording}"); - let bytes = if recording { TRAY_ICON_RECORDING } else { TRAY_ICON_DEFAULT }; + let bytes = if recording { + TRAY_ICON_RECORDING + } else { + TRAY_ICON_DEFAULT + }; let img = Image::from_bytes(bytes).map_err(|e| format!("decode tray icon: {e}"))?; let lock = tray_handle.tray.lock().map_err(|_| "tray lock poisoned")?; if let Some(ref tray) = *lock { - tray.set_icon(Some(img)).map_err(|e| format!("set_icon: {e}"))?; + tray.set_icon(Some(img)) + .map_err(|e| format!("set_icon: {e}"))?; } update_tray_hint(&app, &shortcuts.info().display, recording); Ok(()) @@ -987,7 +991,10 @@ fn place_dictation_pill(app: &tauri::AppHandle, win: &tauri::WebviewWindow) { } log::info!( "pill: placed at {x},{y} ({}x{} on a {}x{} monitor)", - size.width, size.height, area.width, area.height + size.width, + size.height, + area.width, + area.height ); } @@ -1049,9 +1056,15 @@ pub fn mark_dictation_capture_ready(app: tauri::AppHandle) { return; }; capture.ready = true; - if let Some(action) = capture.pending.take() { - drop(capture); - crate::dispatch_dictation_capture(&app, &action); + let pending = std::mem::take(&mut capture.pending); + drop(capture); + for event in pending { + if let Err(error) = app.emit(event.name, event.payload) { + log::warn!( + "Queued dictation event {} could not emit: {error}", + event.name + ); + } } } @@ -1198,7 +1211,8 @@ pub fn reveal_host_path(app: tauri::AppHandle, path: String) -> Result<(), Strin let folder = if target.is_dir() { target.clone() } else { - target.parent() + target + .parent() .ok_or_else(|| "That path has no containing folder".to_string())? .to_path_buf() }; @@ -1268,7 +1282,10 @@ const CLEAR_WEBVIEW_RETRY_DELAY: Duration = Duration::from_millis(500); /// Windows — because step 2 runs before an `AppHandle` exists. fn webview_cache_paths() -> Option<(PathBuf, PathBuf)> { let base = dirs_next::data_local_dir()?.join(crate::config::BUNDLE_IDENTIFIER); - Some((base.join(CLEAR_WEBVIEW_MARKER), base.join(WEBVIEW_CACHE_DIR))) + Some(( + base.join(CLEAR_WEBVIEW_MARKER), + base.join(WEBVIEW_CACHE_DIR), + )) } #[tauri::command] @@ -1281,8 +1298,11 @@ pub fn clear_webview_cache_and_relaunch(app: tauri::AppHandle) -> Result<(), Str if let Some(parent) = marker.parent() { let _ = fs::create_dir_all(parent); } - fs::write(&marker, b"requested by the splash recovery panel (issue #879)\n") - .map_err(|e| format!("write {}: {e}", marker.display()))?; + fs::write( + &marker, + b"requested by the splash recovery panel (issue #879)\n", + ) + .map_err(|e| format!("write {}: {e}", marker.display()))?; log::warn!( "WebView cache repair requested (#879) — relaunching to clear {}", cache.display() @@ -1301,7 +1321,12 @@ pub fn clear_webview_cache_if_marked() { let Some((marker, cache)) = webview_cache_paths() else { return; }; - clear_webview_cache_at(&marker, &cache, CLEAR_WEBVIEW_ATTEMPTS, CLEAR_WEBVIEW_RETRY_DELAY); + clear_webview_cache_at( + &marker, + &cache, + CLEAR_WEBVIEW_ATTEMPTS, + CLEAR_WEBVIEW_RETRY_DELAY, + ); } /// Filesystem half of [`clear_webview_cache_if_marked`], parameterized over @@ -1403,7 +1428,10 @@ mod webview_cache_repair_tests { let cache = dir.path().join(super::WEBVIEW_CACHE_DIR); fs::write(&marker, b"test").unwrap(); clear_webview_cache_at(&marker, &cache, FEW, NO_WAIT); - assert!(!marker.exists(), "marker consumed even with nothing to clear"); + assert!( + !marker.exists(), + "marker consumed even with nothing to clear" + ); } /// A cache that can't be deleted (Windows: WebView2 file locks; simulated @@ -1419,7 +1447,10 @@ mod webview_cache_repair_tests { // Deny writes on the cache dir so its entries can't be unlinked. fs::set_permissions(&cache, fs::Permissions::from_mode(0o555)).unwrap(); clear_webview_cache_at(&marker, &cache, FEW, NO_WAIT); - assert!(!marker.exists(), "one-shot: marker consumed even on failure"); + assert!( + !marker.exists(), + "one-shot: marker consumed even on failure" + ); assert!(cache.exists(), "a locked cache survives the failed repair"); // Restore permissions so TempDir can clean up. fs::set_permissions(&cache, fs::Permissions::from_mode(0o755)).unwrap(); @@ -1428,7 +1459,7 @@ mod webview_cache_repair_tests { #[cfg(test)] mod paste_error_tests { - use super::{kind_err, CLIPBOARD_RESTORE_DELAY}; + use super::kind_err; #[test] fn kind_err_prefixes_with_kind() { @@ -1450,11 +1481,4 @@ mod paste_error_tests { let e = kind_err("clipboard", "init failed: os error 5"); assert_eq!(e.split_once(':').map(|(k, _)| k), Some("clipboard")); } - - #[test] - fn restore_delay_is_about_300ms() { - // Contract with the widget layer: previous clipboard comes back - // ~300ms after the paste, long enough for slow paste consumers. - assert_eq!(CLIPBOARD_RESTORE_DELAY.as_millis(), 300); - } } diff --git a/frontend/src-tauri/src/dictation_output.rs b/frontend/src-tauri/src/dictation_output.rs new file mode 100644 index 000000000..366ea8872 --- /dev/null +++ b/frontend/src-tauri/src/dictation_output.rs @@ -0,0 +1,1656 @@ +//! Session-bound transcript delivery across desktop platforms. +//! +//! The public surface deliberately talks in sessions and outcomes. Focus +//! discovery, clipboard ownership, and input synthesis stay private so a late +//! transcript can never accidentally target a newer dictation session. + +use std::path::PathBuf; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread; +use std::time::{Duration, Instant}; + +use arboard::{Clipboard, ImageData}; +use enigo::{Direction, Enigo, Key, Keyboard, Settings as EnigoSettings}; +use serde::Serialize; + +const TARGET_SETTLE_DELAY: Duration = Duration::from_millis(60); +const CLIPBOARD_CONSUME_DELAY: Duration = Duration::from_millis(300); +const START_EVENT_DEDUPE_WINDOW: Duration = Duration::from_millis(150); + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum DeliveryOutcome { + Inserted, + Copied, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum CaptureOrigin { + Shortcut, + Tray, +} + +#[derive(Clone, Default)] +pub struct DictationOutput { + inner: Arc, +} + +#[derive(Default)] +struct Inner { + next_session_id: AtomicU64, + operation: Mutex<()>, + state: Mutex, +} + +#[derive(Default)] +struct OutputState { + active: Option, + pending: Option, + clipboard_generation: u64, + tray_target: Option<(PlatformTarget, Instant)>, +} + +struct OutputSession { + id: u64, + started_at: Instant, + target: Option, + clipboard: Option, + clipboard_only: bool, +} + +struct ClipboardLease { + original: ClipboardSnapshot, + generation: u64, + staged: String, + staged_at: Instant, +} + +#[derive(Clone)] +enum ClipboardSnapshot { + Text(String), + Html { + html: String, + alt_text: Option, + }, + Files(Vec), + Image(ImageData<'static>), + // Rich clipboard formats that arboard cannot round-trip must not be + // replaced a second time with a guessed value. + Unsupported, +} + +impl DictationOutput { + /// Remember focus on tray mouse-down, before the menu itself can become + /// foreground. Tauri does not emit tray pointer events on Linux; X11 can + /// still capture `_NET_ACTIVE_WINDOW` at the menu action, while Wayland + /// tray starts intentionally fall back to the clipboard. + pub fn prime_tray_target(&self) { + self.prime_tray_target_with(capture_target); + } + + fn prime_tray_target_with(&self, capture: F) + where + F: FnOnce() -> Option, + { + // Focus discovery must happen at callback entry. Clipboard delivery + // can hold the operation lock while another application becomes + // foreground, so acquiring it first would capture the wrong target. + let captured_at = Instant::now(); + let target = capture().filter(|target| !target.belongs_to_current_process()); + if let Ok(mut state) = self.inner.state.lock() { + state.tray_target = target.map(|target| (target, captured_at)); + } + } + + /// Capture the destination before the capture event can show the pill. + pub fn begin_session(&self, origin: CaptureOrigin) -> u64 { + self.begin_session_with(origin, capture_target, tray_action_capture_supported()) + } + + fn begin_session_with( + &self, + origin: CaptureOrigin, + capture: F, + capture_tray_on_action: bool, + ) -> u64 + where + F: FnOnce() -> Option, + { + // Capture before waiting for another delivery operation. This is the + // shortcut-down target, even when a clipboard/helper call is busy. + let started_at = Instant::now(); + let captured_target = (origin == CaptureOrigin::Shortcut || capture_tray_on_action) + .then(capture) + .flatten() + .filter(|target| { + origin == CaptureOrigin::Shortcut || !target.belongs_to_current_process() + }); + let mut state = self + .inner + .state + .lock() + .unwrap_or_else(|error| error.into_inner()); + // Collapse only duplicate delivery of the same physical press. A + // later press becomes a candidate until the frontend accepts it, so + // a press ignored during transcription cannot invalidate that result. + if let Some(pending) = state.pending.as_ref() { + if start_events_duplicate(pending.started_at, started_at) { + return pending.id; + } + } + state.pending = None; + if let Some(active) = state.active.as_ref() { + if start_events_duplicate(active.started_at, started_at) { + return active.id; + } + } + let primed_tray_target = state + .tray_target + .take() + .filter(|(_, captured_at)| captured_at.elapsed() <= Duration::from_secs(10)) + .map(|(target, _)| target); + let target = choose_session_target(origin, captured_target, primed_tray_target); + let id = self + .inner + .next_session_id + .fetch_add(1, Ordering::Relaxed) + .wrapping_add(1); + let clipboard_only = should_start_clipboard_only( + origin, + target.is_some(), + untargeted_wayland_insert_enabled(), + ); + let session = OutputSession { + id, + started_at, + clipboard_only, + target, + clipboard: None, + }; + if state.active.is_some() { + state.pending = Some(session); + } else { + state.active = Some(session); + } + id + } + + pub fn current_session_id(&self) -> Option { + self.inner.state.lock().ok().and_then(|state| { + state + .pending + .as_ref() + .or(state.active.as_ref()) + .map(|session| session.id) + }) + } + + /// Promote a captured candidate only after the frontend accepts its start + /// event. This keeps an older transcribing session valid when a new press + /// is ignored, and makes late teardown of that older ID harmless. + pub fn activate_session(&self, session_id: u64) -> Result<(), String> { + let _operation = self + .inner + .operation + .lock() + .map_err(|_| kind_err("paste", "output operation lock poisoned"))?; + let replaced_lease = { + let mut state = self + .inner + .state + .lock() + .map_err(|_| kind_err("paste", "output state lock poisoned"))?; + if state.active.as_ref().map(|session| session.id) == Some(session_id) { + return Ok(()); + } + if state.pending.as_ref().map(|session| session.id) != Some(session_id) { + return Err(kind_err("paste", "stale dictation session candidate")); + } + let candidate = state.pending.take().expect("candidate validated above"); + let replaced = state.active.replace(candidate); + replaced.and_then(|session| session.clipboard) + }; + if let Some(lease) = replaced_lease { + self.schedule_finished_restore(lease); + } + Ok(()) + } + + /// Drop an unaccepted candidate without touching the active session. A + /// key repeat can legitimately carry the active ID, so rejection must be + /// a no-op for anything other than the matching pending candidate. + pub fn reject_session_candidate(&self, session_id: u64) { + let _operation = self + .inner + .operation + .lock() + .unwrap_or_else(|error| error.into_inner()); + let Ok(mut state) = self.inner.state.lock() else { + return; + }; + if state.pending.as_ref().map(|session| session.id) == Some(session_id) { + state.pending = None; + } + } + + pub fn deliver(&self, session_id: u64, text: &str) -> Result { + let _operation = self + .inner + .operation + .lock() + .map_err(|_| kind_err("paste", "output operation lock poisoned"))?; + self.require_session(session_id)?; + if text.is_empty() { + return Ok(DeliveryOutcome::Inserted); + } + + if self.session_is_clipboard_only(session_id)? { + self.copy_only(session_id, text)?; + return Ok(DeliveryOutcome::Copied); + } + + #[cfg(target_os = "linux")] + if is_wayland() { + return self.deliver_wayland(session_id, text); + } + + let target = self.session_target(session_id)?; + let Some(target) = target else { + self.copy_only(session_id, text)?; + return Ok(DeliveryOutcome::Copied); + }; + if !activate_target(&target) { + self.copy_only(session_id, text)?; + return Ok(DeliveryOutcome::Copied); + } + + let generation = self.stage_clipboard(session_id, text)?; + if !target_is_active(&target) { + self.latch_staged_copy_only(session_id, generation, text)?; + return Ok(DeliveryOutcome::Copied); + } + if synthesize_paste().is_err() { + // The key sequence may have reached the application. Keep the + // transcript staged, latch clipboard-only, and never attempt a + // second insertion. + self.latch_staged_copy_only(session_id, generation, text)?; + return Ok(DeliveryOutcome::Copied); + } + self.schedule_restore(session_id, generation, text.to_owned()); + Ok(DeliveryOutcome::Inserted) + } + + /// Keep the complete transcript on the clipboard without attempting input + /// synthesis. Used when a platform permission gate fails before emission. + pub fn copy_for_session(&self, session_id: u64, text: &str) -> Result { + let _operation = self + .inner + .operation + .lock() + .map_err(|_| kind_err("clipboard", "output operation lock poisoned"))?; + self.require_session(session_id)?; + self.copy_only(session_id, text)?; + Ok(DeliveryOutcome::Copied) + } + + pub fn type_delta( + &self, + session_id: u64, + text: &str, + backspaces: u32, + ) -> Result { + let _operation = self + .inner + .operation + .lock() + .map_err(|_| kind_err("paste", "output operation lock poisoned"))?; + self.require_session(session_id) + .map_err(|_| kind_err("preflight", "stale dictation session before live typing"))?; + if self.session_is_clipboard_only(session_id)? { + return Err(kind_err("preflight", "dictation session is clipboard-only")); + } + + #[cfg(target_os = "linux")] + if is_wayland() { + return type_delta_wayland(text, backspaces); + } + + let target = self.session_target(session_id)?; + let Some(target) = target else { + return Err(kind_err("preflight", "the original target is unavailable")); + }; + if !activate_target(&target) { + return Err(kind_err( + "preflight", + "the original target could not be activated", + )); + } + if !target_is_active(&target) { + return Err(kind_err( + "preflight", + "the original target lost focus before live typing", + )); + } + synthesize_text(text, backspaces)?; + Ok(DeliveryOutcome::Inserted) + } + + pub fn finish_session(&self, session_id: u64) { + let _operation = self + .inner + .operation + .lock() + .unwrap_or_else(|error| error.into_inner()); + let finished = { + let Ok(mut state) = self.inner.state.lock() else { + return; + }; + if state.pending.as_ref().map(|session| session.id) == Some(session_id) { + state.pending.take() + } else if state.active.as_ref().map(|session| session.id) == Some(session_id) { + state.active.take() + } else { + None + } + }; + if let Some(lease) = finished.and_then(|session| session.clipboard) { + self.schedule_finished_restore(lease); + } + } + + fn require_session(&self, session_id: u64) -> Result<(), String> { + let state = self + .inner + .state + .lock() + .map_err(|_| kind_err("paste", "output state lock poisoned"))?; + if state.active.as_ref().map(|session| session.id) == Some(session_id) { + Ok(()) + } else { + Err(kind_err("paste", "stale dictation session")) + } + } + + fn session_target(&self, session_id: u64) -> Result, String> { + let state = self + .inner + .state + .lock() + .map_err(|_| kind_err("paste", "output state lock poisoned"))?; + let session = state + .active + .as_ref() + .filter(|session| session.id == session_id) + .ok_or_else(|| kind_err("paste", "stale dictation session"))?; + Ok(session.target.clone()) + } + + fn session_is_clipboard_only(&self, session_id: u64) -> Result { + let state = self + .inner + .state + .lock() + .map_err(|_| kind_err("paste", "output state lock poisoned"))?; + state + .active + .as_ref() + .filter(|session| session.id == session_id) + .map(|session| session.clipboard_only) + .ok_or_else(|| kind_err("paste", "stale dictation session")) + } + + fn stage_clipboard(&self, session_id: u64, text: &str) -> Result { + let mut clipboard = open_clipboard()?; + let mut state = self + .inner + .state + .lock() + .map_err(|_| kind_err("clipboard", "output state lock poisoned"))?; + // Overlapping streaming segments share one original snapshot. If the + // user copied something after our prior stage, that becomes the new + // protected value instead of being overwritten by a stale restore. + let current = clipboard.get_text().ok(); + let keep_original = state + .active + .as_ref() + .filter(|session| session.id == session_id) + .ok_or_else(|| kind_err("paste", "stale dictation session"))? + .clipboard + .as_ref() + .is_some_and(|lease| clipboard_still_staged(&lease.staged, current.as_deref())); + let replacement = (!keep_original).then(|| snapshot_clipboard(&mut clipboard)); + + set_clipboard_text(&mut clipboard, text)?; + state.clipboard_generation = state.clipboard_generation.wrapping_add(1); + let generation = state.clipboard_generation; + let session = state + .active + .as_mut() + .filter(|session| session.id == session_id) + .ok_or_else(|| kind_err("paste", "stale dictation session"))?; + if let Some(original) = replacement { + session.clipboard = Some(ClipboardLease { + original, + generation, + staged: String::new(), + staged_at: Instant::now(), + }); + } + let lease = session.clipboard.as_mut().expect("clipboard lease created"); + lease.generation = generation; + lease.staged.clear(); + lease.staged.push_str(text); + lease.staged_at = Instant::now(); + Ok(generation) + } + + fn copy_only(&self, session_id: u64, text: &str) -> Result<(), String> { + let mut state = self + .inner + .state + .lock() + .map_err(|_| kind_err("clipboard", "output state lock poisoned"))?; + state + .active + .as_ref() + .filter(|session| session.id == session_id) + .ok_or_else(|| kind_err("paste", "stale dictation session"))?; + let mut clipboard = open_clipboard()?; + set_clipboard_text(&mut clipboard, text)?; + state.clipboard_generation = state.clipboard_generation.wrapping_add(1); + let session = state.active.as_mut().expect("session validated"); + session.clipboard = None; + session.clipboard_only = true; + Ok(()) + } + + fn latch_staged_copy_only( + &self, + session_id: u64, + generation: u64, + staged: &str, + ) -> Result<(), String> { + let mut state = self + .inner + .state + .lock() + .map_err(|_| kind_err("clipboard", "output state lock poisoned"))?; + let is_current_stage = state.clipboard_generation == generation + && state + .active + .as_ref() + .filter(|session| session.id == session_id) + .and_then(|session| session.clipboard.as_ref()) + .is_some_and(|lease| lease.generation == generation && lease.staged == staged); + if !is_current_stage { + return Err(kind_err("paste", "stale staged clipboard")); + } + + // The transcript is already on the clipboard. Invalidate every + // scheduled restore without reopening a clipboard that may be + // transiently unavailable, then keep this session copy-only. + state.clipboard_generation = state.clipboard_generation.wrapping_add(1); + let session = state.active.as_mut().expect("session validated above"); + session.clipboard = None; + session.clipboard_only = true; + Ok(()) + } + + fn schedule_restore(&self, session_id: u64, generation: u64, staged: String) { + let output = self.clone(); + thread::spawn(move || { + thread::sleep(CLIPBOARD_CONSUME_DELAY); + output.restore_if_current(session_id, generation, &staged); + }); + } + + fn restore_if_current(&self, session_id: u64, generation: u64, staged: &str) { + let _operation = self + .inner + .operation + .lock() + .unwrap_or_else(|error| error.into_inner()); + let mut state = match self.inner.state.lock() { + Ok(state) => state, + Err(_) => return, + }; + if state.clipboard_generation != generation { + return; + } + let Some(session) = state + .active + .as_mut() + .filter(|session| session.id == session_id) + else { + return; + }; + let Some(lease) = session.clipboard.as_ref() else { + return; + }; + if lease.generation != generation || lease.staged != staged { + return; + } + let original = lease.original.clone(); + let Ok(mut clipboard) = open_clipboard() else { + return; + }; + let current = clipboard.get_text().ok(); + if !clipboard_still_staged(staged, current.as_deref()) { + session.clipboard = None; + return; + } + if restore_clipboard(&mut clipboard, &original).is_ok() { + session.clipboard = None; + } + } + + fn schedule_finished_restore(&self, lease: ClipboardLease) { + let output = self.clone(); + let delay = CLIPBOARD_CONSUME_DELAY.saturating_sub(lease.staged_at.elapsed()); + thread::spawn(move || { + thread::sleep(delay); + output.restore_finished_lease(lease); + }); + } + + fn restore_finished_lease(&self, lease: ClipboardLease) { + let _operation = self + .inner + .operation + .lock() + .unwrap_or_else(|error| error.into_inner()); + let Ok(state) = self.inner.state.lock() else { + return; + }; + if state.clipboard_generation != lease.generation { + return; + } + let Ok(mut clipboard) = open_clipboard() else { + return; + }; + let current = clipboard.get_text().ok(); + if clipboard_still_staged(&lease.staged, current.as_deref()) { + let _ = restore_clipboard(&mut clipboard, &lease.original); + } + } +} + +fn open_clipboard() -> Result { + let attempts = if cfg!(target_os = "windows") { 4 } else { 1 }; + let mut last_error = None; + for attempt in 0..attempts { + match Clipboard::new() { + Ok(clipboard) => return Ok(clipboard), + Err(error) => last_error = Some(error), + } + if attempt + 1 < attempts { + thread::sleep(Duration::from_millis(25)); + } + } + Err(kind_err( + "clipboard", + format!("init failed: {}", last_error.expect("at least one attempt")), + )) +} + +fn set_clipboard_text(clipboard: &mut Clipboard, text: &str) -> Result<(), String> { + let attempts = if cfg!(target_os = "windows") { 4 } else { 1 }; + let mut last_error = None; + for attempt in 0..attempts { + match clipboard.set_text(text.to_owned()) { + Ok(()) => return Ok(()), + Err(error) => last_error = Some(error), + } + if attempt + 1 < attempts { + thread::sleep(Duration::from_millis(25)); + } + } + Err(kind_err( + "clipboard", + format!( + "write failed: {}", + last_error.expect("at least one attempt") + ), + )) +} + +fn snapshot_clipboard(clipboard: &mut Clipboard) -> ClipboardSnapshot { + // Rich formats commonly expose a plain-text fallback. Probe them first so + // restoring the clipboard does not silently flatten HTML, images, or file + // selections into that fallback. + let files = clipboard.get().file_list().ok(); + let html = clipboard.get().html().ok(); + let image = clipboard.get_image().ok(); + let text = clipboard.get_text().ok(); + snapshot_from_formats(files, html, image, text) +} + +fn snapshot_from_formats( + files: Option>, + html: Option, + image: Option>, + text: Option, +) -> ClipboardSnapshot { + if let Some(files) = files.filter(|paths| !paths.is_empty()) { + ClipboardSnapshot::Files(files) + } else if let Some(html) = html { + ClipboardSnapshot::Html { + html, + alt_text: text, + } + } else if let Some(image) = image { + ClipboardSnapshot::Image(image) + } else if let Some(text) = text { + ClipboardSnapshot::Text(text) + } else { + ClipboardSnapshot::Unsupported + } +} + +fn restore_clipboard( + clipboard: &mut Clipboard, + snapshot: &ClipboardSnapshot, +) -> Result<(), String> { + match snapshot { + ClipboardSnapshot::Text(text) => set_clipboard_text(clipboard, text), + ClipboardSnapshot::Html { html, alt_text } => clipboard + .set() + .html(html.clone(), alt_text.clone()) + .map_err(|error| kind_err("clipboard", format!("HTML restore failed: {error}"))), + ClipboardSnapshot::Files(paths) => clipboard + .set() + .file_list(paths) + .map_err(|error| kind_err("clipboard", format!("file-list restore failed: {error}"))), + ClipboardSnapshot::Image(image) => clipboard + .set_image(image.clone()) + .map_err(|error| kind_err("clipboard", format!("image restore failed: {error}"))), + ClipboardSnapshot::Unsupported => Ok(()), + } +} + +fn clipboard_still_staged(staged: &str, current: Option<&str>) -> bool { + current == Some(staged) +} + +fn choose_session_target( + origin: CaptureOrigin, + captured_at_action: Option, + primed_tray_target: Option, +) -> Option { + match origin { + CaptureOrigin::Shortcut => captured_at_action, + CaptureOrigin::Tray => primed_tray_target.or(captured_at_action), + } +} + +fn start_events_duplicate(existing: Instant, incoming: Instant) -> bool { + let distance = if incoming >= existing { + incoming.duration_since(existing) + } else { + existing.duration_since(incoming) + }; + distance <= START_EVENT_DEDUPE_WINDOW +} + +#[cfg(target_os = "linux")] +fn tray_action_capture_supported() -> bool { + !is_wayland() +} + +#[cfg(not(target_os = "linux"))] +fn tray_action_capture_supported() -> bool { + false +} + +fn should_start_clipboard_only( + origin: CaptureOrigin, + target_available: bool, + untargeted_wayland_opt_in: bool, +) -> bool { + !target_available && !(origin == CaptureOrigin::Shortcut && untargeted_wayland_opt_in) +} + +#[cfg(target_os = "linux")] +fn untargeted_wayland_insert_enabled() -> bool { + is_wayland() + && std::env::var("VOICESTUDIO_WAYLAND_UNTARGETED_INSERT").is_ok_and(|value| { + matches!( + value.trim().to_ascii_lowercase().as_str(), + "1" | "true" | "yes" + ) + }) +} + +#[cfg(not(target_os = "linux"))] +fn untargeted_wayland_insert_enabled() -> bool { + false +} + +fn synthesize_paste() -> Result<(), String> { + let mut enigo = Enigo::new(&EnigoSettings::default()) + .map_err(|error| kind_err("paste", format!("keyboard init failed: {error}")))?; + #[cfg(target_os = "macos")] + let modifier = Key::Meta; + #[cfg(not(target_os = "macos"))] + let modifier = Key::Control; + + enigo + .key(modifier, Direction::Press) + .map_err(|error| kind_err("paste", format!("modifier press failed: {error}")))?; + let click = enigo.key(Key::Unicode('v'), Direction::Click); + let release = enigo.key(modifier, Direction::Release); + if let Err(error) = click { + return Err(kind_err("paste", format!("paste key failed: {error}"))); + } + release.map_err(|error| kind_err("paste", format!("modifier release failed: {error}"))) +} + +fn synthesize_text(text: &str, backspaces: u32) -> Result<(), String> { + let mut enigo = Enigo::new(&EnigoSettings::default()) + .map_err(|error| kind_err("paste", format!("keyboard init failed: {error}")))?; + for _ in 0..backspaces { + enigo + .key(Key::Backspace, Direction::Click) + .map_err(|error| kind_err("paste", format!("backspace failed: {error}")))?; + } + if !text.is_empty() { + enigo + .text(text) + .map_err(|error| kind_err("paste", format!("type failed: {error}")))?; + } + Ok(()) +} + +fn kind_err(kind: &str, detail: impl std::fmt::Display) -> String { + format!("{kind}:{detail}") +} + +// ── Destination capture and reactivation ──────────────────────────────── + +#[cfg(target_os = "macos")] +#[derive(Clone)] +struct PlatformTarget { + app: objc2::rc::Retained, + pid: i32, +} + +#[cfg(target_os = "macos")] +impl PlatformTarget { + fn belongs_to_current_process(&self) -> bool { + self.pid == std::process::id() as i32 + } +} + +#[cfg(target_os = "macos")] +fn capture_target() -> Option { + let app = objc2_app_kit::NSWorkspace::sharedWorkspace().frontmostApplication()?; + let pid = app.processIdentifier(); + Some(PlatformTarget { app, pid }) +} + +#[cfg(target_os = "macos")] +fn activate_target(target: &PlatformTarget) -> bool { + use objc2_app_kit::NSApplicationActivationOptions; + + if target.app.processIdentifier() != target.pid + || !target + .app + .activateWithOptions(NSApplicationActivationOptions::empty()) + { + return false; + } + thread::sleep(TARGET_SETTLE_DELAY); + target.app.isActive() && target.app.processIdentifier() == target.pid +} + +#[cfg(target_os = "macos")] +fn target_is_active(target: &PlatformTarget) -> bool { + target.app.processIdentifier() == target.pid && target.app.isActive() +} + +#[cfg(target_os = "windows")] +#[derive(Clone)] +struct PlatformTarget { + hwnd: isize, + pid: u32, +} + +#[cfg(target_os = "windows")] +impl PlatformTarget { + fn belongs_to_current_process(&self) -> bool { + self.pid == std::process::id() + } +} + +#[cfg(target_os = "windows")] +fn capture_target() -> Option { + use windows::Win32::UI::WindowsAndMessaging::{GetForegroundWindow, GetWindowThreadProcessId}; + + unsafe { + let hwnd = GetForegroundWindow(); + if hwnd.0.is_null() { + return None; + } + let mut pid = 0; + GetWindowThreadProcessId(hwnd, Some(&mut pid)); + if pid == 0 { + return None; + } + Some(PlatformTarget { + hwnd: hwnd.0 as isize, + pid, + }) + } +} + +#[cfg(target_os = "windows")] +fn activate_target(target: &PlatformTarget) -> bool { + use windows::Win32::Foundation::HWND; + use windows::Win32::System::Threading::{AttachThreadInput, GetCurrentThreadId}; + use windows::Win32::UI::WindowsAndMessaging::{ + BringWindowToTop, GetForegroundWindow, GetWindowThreadProcessId, IsWindow, + SetForegroundWindow, + }; + + unsafe { + let hwnd = HWND(target.hwnd as *mut _); + if !IsWindow(Some(hwnd)).as_bool() { + return false; + } + let mut pid = 0; + let target_thread = GetWindowThreadProcessId(hwnd, Some(&mut pid)); + if pid != target.pid || target_thread == 0 { + return false; + } + let current_thread = GetCurrentThreadId(); + let attached = target_thread != current_thread + && AttachThreadInput(current_thread, target_thread, true).as_bool(); + let _ = BringWindowToTop(hwnd); + let requested = SetForegroundWindow(hwnd).as_bool(); + if attached { + let _ = AttachThreadInput(current_thread, target_thread, false); + } + if !requested { + return false; + } + thread::sleep(TARGET_SETTLE_DELAY); + GetForegroundWindow() == hwnd + } +} + +#[cfg(target_os = "windows")] +fn target_is_active(target: &PlatformTarget) -> bool { + use windows::Win32::Foundation::HWND; + use windows::Win32::UI::WindowsAndMessaging::{ + GetForegroundWindow, GetWindowThreadProcessId, IsWindow, + }; + + unsafe { + let hwnd = HWND(target.hwnd as *mut _); + if !IsWindow(Some(hwnd)).as_bool() || GetForegroundWindow() != hwnd { + return false; + } + let mut pid = 0; + GetWindowThreadProcessId(hwnd, Some(&mut pid)); + pid == target.pid + } +} + +#[cfg(target_os = "linux")] +#[derive(Clone)] +struct PlatformTarget { + window: u32, + pid: Option, +} + +#[cfg(target_os = "linux")] +impl PlatformTarget { + fn belongs_to_current_process(&self) -> bool { + self.pid == Some(std::process::id()) + } +} + +#[cfg(target_os = "linux")] +fn capture_target() -> Option { + use x11rb::connection::Connection; + + if is_wayland() { + return None; + } + let (connection, screen_num) = x11rb::connect(None).ok()?; + let root = connection.setup().roots.get(screen_num)?.root; + let window = x11_active_window(&connection, root)?; + // A window id alone can be reused after its client exits. Without the + // standard EWMH PID we cannot prove this is still the shortcut-down + // target at delivery time, so degrade to clipboard-only instead. + let pid = x11_window_pid(&connection, window)?; + Some(PlatformTarget { + window, + pid: Some(pid), + }) +} + +#[cfg(target_os = "linux")] +fn activate_target(target: &PlatformTarget) -> bool { + use x11rb::connection::Connection; + use x11rb::protocol::xproto::{ + ClientMessageData, ClientMessageEvent, ConnectionExt, EventMask, + }; + + let Ok((connection, screen_num)) = x11rb::connect(None) else { + return false; + }; + let Some(root) = connection + .setup() + .roots + .get(screen_num) + .map(|screen| screen.root) + else { + return false; + }; + let Ok(attributes) = connection.get_window_attributes(target.window) else { + return false; + }; + if attributes.reply().is_err() { + return false; + } + if target.pid.is_some() && x11_window_pid(&connection, target.window) != target.pid { + return false; + } + let Ok(cookie) = connection.intern_atom(false, b"_NET_ACTIVE_WINDOW") else { + return false; + }; + let Ok(reply) = cookie.reply() else { + return false; + }; + let event = ClientMessageEvent::new( + 32, + target.window, + reply.atom, + ClientMessageData::from([1, 0, 0, 0, 0]), + ); + if connection + .send_event( + false, + root, + EventMask::SUBSTRUCTURE_REDIRECT | EventMask::SUBSTRUCTURE_NOTIFY, + event, + ) + .is_err() + || connection.flush().is_err() + { + return false; + } + thread::sleep(TARGET_SETTLE_DELAY); + x11_active_window(&connection, root) == Some(target.window) +} + +#[cfg(target_os = "linux")] +fn target_is_active(target: &PlatformTarget) -> bool { + use x11rb::connection::Connection; + + let Ok((connection, screen_num)) = x11rb::connect(None) else { + return false; + }; + let Some(root) = connection + .setup() + .roots + .get(screen_num) + .map(|screen| screen.root) + else { + return false; + }; + x11_active_window(&connection, root) == Some(target.window) + && (target.pid.is_none() || x11_window_pid(&connection, target.window) == target.pid) +} + +#[cfg(target_os = "linux")] +fn x11_active_window(connection: &C, root: u32) -> Option { + use x11rb::protocol::xproto::{AtomEnum, ConnectionExt}; + + let atom = connection + .intern_atom(false, b"_NET_ACTIVE_WINDOW") + .ok()? + .reply() + .ok()? + .atom; + connection + .get_property(false, root, atom, AtomEnum::WINDOW, 0, 1) + .ok()? + .reply() + .ok()? + .value32()? + .next() +} + +#[cfg(target_os = "linux")] +fn x11_window_pid(connection: &C, window: u32) -> Option { + use x11rb::protocol::xproto::{AtomEnum, ConnectionExt}; + + let atom = connection + .intern_atom(false, b"_NET_WM_PID") + .ok()? + .reply() + .ok()? + .atom; + connection + .get_property(false, window, atom, AtomEnum::CARDINAL, 0, 1) + .ok()? + .reply() + .ok()? + .value32()? + .next() +} + +#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))] +#[derive(Clone)] +struct PlatformTarget; + +#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))] +impl PlatformTarget { + fn belongs_to_current_process(&self) -> bool { + false + } +} + +#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))] +fn capture_target() -> Option { + None +} + +#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))] +fn activate_target(_: &PlatformTarget) -> bool { + false +} + +#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))] +fn target_is_active(_: &PlatformTarget) -> bool { + false +} + +// ── Wayland insertion adapters ────────────────────────────────────────── + +#[cfg(any(target_os = "linux", test))] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum LinuxTool { + Wtype, + Dotool, + Ydotool, +} + +#[cfg(any(target_os = "linux", test))] +fn linux_tool_plan(wayland: bool, desktop: &str) -> Vec { + if !wayland { + return Vec::new(); + } + let tokens: Vec = desktop + .split([':', ';', ',', ' ']) + .filter(|token| !token.is_empty()) + .map(str::to_ascii_lowercase) + .collect(); + let kde = tokens + .iter() + .any(|token| token.contains("kde") || token.contains("plasma")); + let gnome = tokens.iter().any(|token| token.contains("gnome")); + if kde { + vec![LinuxTool::Dotool, LinuxTool::Ydotool] + } else if gnome { + vec![LinuxTool::Dotool, LinuxTool::Ydotool] + } else { + vec![LinuxTool::Wtype, LinuxTool::Dotool, LinuxTool::Ydotool] + } +} + +#[cfg(any(target_os = "linux", test))] +fn wayland_live_type_tool(desktop: &str) -> Option { + linux_tool_plan(true, desktop) + .into_iter() + // Live revisions need both arbitrary Unicode and backspace support. + .find(|tool| *tool == LinuxTool::Wtype) +} + +#[cfg(target_os = "linux")] +fn is_wayland() -> bool { + std::env::var("XDG_SESSION_TYPE").is_ok_and(|value| value.eq_ignore_ascii_case("wayland")) + || std::env::var_os("WAYLAND_DISPLAY").is_some() +} + +#[cfg(target_os = "linux")] +impl DictationOutput { + fn deliver_wayland(&self, session_id: u64, text: &str) -> Result { + let desktop = std::env::var("XDG_CURRENT_DESKTOP").unwrap_or_default(); + for tool in linux_tool_plan(true, &desktop) { + if !linux_helper_ready(tool) { + continue; + } + match tool { + // This accepts arbitrary Unicode. Only one started helper + // is ever tried, because a failing process may have inserted a + // prefix and retrying would duplicate it. + LinuxTool::Wtype => { + if run_linux_helper("wtype", &["-"], Some(text.as_bytes())).is_err() { + self.copy_only(session_id, text)?; + return Ok(DeliveryOutcome::Copied); + } + return Ok(DeliveryOutcome::Inserted); + } + // dotool/ydotool text modes are layout-limited. Paste a + // clipboard payload with fixed key commands instead. + LinuxTool::Dotool => { + let generation = self.stage_clipboard(session_id, text)?; + if run_linux_helper("dotool", &[], Some(b"key ctrl+v\n")).is_err() { + self.latch_staged_copy_only(session_id, generation, text)?; + return Ok(DeliveryOutcome::Copied); + } + self.schedule_restore(session_id, generation, text.to_owned()); + return Ok(DeliveryOutcome::Inserted); + } + LinuxTool::Ydotool => { + let generation = self.stage_clipboard(session_id, text)?; + if run_linux_helper("ydotool", &["key", "29:1", "47:1", "47:0", "29:0"], None) + .is_err() + { + self.latch_staged_copy_only(session_id, generation, text)?; + return Ok(DeliveryOutcome::Copied); + } + self.schedule_restore(session_id, generation, text.to_owned()); + return Ok(DeliveryOutcome::Inserted); + } + } + } + self.copy_only(session_id, text)?; + Ok(DeliveryOutcome::Copied) + } +} + +#[cfg(target_os = "linux")] +impl LinuxTool { + fn command(self) -> &'static str { + match self { + Self::Wtype => "wtype", + Self::Dotool => "dotool", + Self::Ydotool => "ydotool", + } + } +} + +#[cfg(any(target_os = "linux", test))] +fn linux_helper_ready_with( + tool: LinuxTool, + exists: E, + probe: P, + uinput_writable: U, +) -> bool +where + E: FnOnce(&str) -> bool, + P: FnOnce(&str, &[&str]) -> bool, + U: FnOnce() -> bool, +{ + let command = match tool { + LinuxTool::Wtype => "wtype", + LinuxTool::Dotool => "dotool", + LinuxTool::Ydotool => "ydotool", + }; + if !exists(command) { + return false; + } + match tool { + // dotool opens uinput itself. Opening and immediately closing the + // device verifies permissions without creating a virtual keyboard or + // emitting input. + LinuxTool::Dotool => uinput_writable(), + // Since 1.0, ydotool is only a client for ydotoold. `debug` verifies + // the socket and daemon without emitting input. + LinuxTool::Ydotool => probe(command, &["debug"]), + LinuxTool::Wtype => true, + } +} + +#[cfg(target_os = "linux")] +fn linux_helper_ready(tool: LinuxTool) -> bool { + linux_helper_ready_with( + tool, + linux_helper_exists, + |command, args| run_linux_helper(command, args, None).is_ok(), + linux_uinput_writable, + ) +} + +#[cfg(target_os = "linux")] +fn linux_uinput_writable() -> bool { + ["/dev/uinput", "/dev/input/uinput"] + .iter() + .any(|path| std::fs::OpenOptions::new().write(true).open(path).is_ok()) +} + +#[cfg(target_os = "linux")] +fn type_delta_wayland(text: &str, backspaces: u32) -> Result { + let desktop = std::env::var("XDG_CURRENT_DESKTOP").unwrap_or_default(); + if wayland_live_type_tool(&desktop) == Some(LinuxTool::Wtype) && linux_helper_exists("wtype") { + let mut owned = Vec::with_capacity(backspaces as usize * 4 + 1); + for _ in 0..backspaces { + owned.extend(["-P".to_owned(), "BackSpace".to_owned()]); + owned.extend(["-p".to_owned(), "BackSpace".to_owned()]); + } + owned.push("-".to_owned()); + let args: Vec<&str> = owned.iter().map(String::as_str).collect(); + run_linux_helper("wtype", &args, Some(text.as_bytes()))?; + return Ok(DeliveryOutcome::Inserted); + } + Err(kind_err( + "preflight", + "live typing is unavailable on this Wayland compositor", + )) +} + +#[cfg(target_os = "linux")] +fn linux_helper_exists(name: &str) -> bool { + use std::os::unix::fs::PermissionsExt; + + std::env::var_os("PATH").is_some_and(|path| { + std::env::split_paths(&path).any(|dir| { + let path = dir.join(name); + path.metadata().is_ok_and(|metadata| { + metadata.is_file() && metadata.permissions().mode() & 0o111 != 0 + }) + }) + }) +} + +#[cfg(target_os = "linux")] +fn run_linux_helper(name: &str, args: &[&str], stdin: Option<&[u8]>) -> Result<(), String> { + use std::io::{Read, Write}; + use std::process::{Command, Stdio}; + use std::time::Instant; + + let mut command = Command::new(name); + command + .args(args) + .stdout(Stdio::null()) + .stderr(Stdio::piped()) + .stdin(if stdin.is_some() { + Stdio::piped() + } else { + Stdio::null() + }); + // AppImages bundle ABI-sensitive libraries. Host desktop helpers must use + // the host loader while retaining DISPLAY/WAYLAND_DISPLAY and the runtime + // directory from the user session. + if std::env::var_os("APPIMAGE").is_some() || std::env::var_os("APPDIR").is_some() { + command + .env_remove("LD_LIBRARY_PATH") + .env_remove("LD_PRELOAD"); + } + let started = Instant::now(); + let mut child = command + .spawn() + .map_err(|error| kind_err("paste", format!("{name} could not start: {error}")))?; + let writer = if let Some(bytes) = stdin { + let Some(mut pipe) = child.stdin.take() else { + let _ = child.kill(); + let _ = child.wait(); + return Err(kind_err("paste", format!("{name} stdin unavailable"))); + }; + let bytes = bytes.to_owned(); + Some(thread::spawn(move || pipe.write_all(&bytes))) + } else { + None + }; + let stderr_drain = child.stderr.take().map(|mut stderr| { + thread::spawn(move || { + let mut buffer = [0_u8; 1024]; + while let Ok(read) = stderr.read(&mut buffer) { + if read == 0 { + break; + } + } + }) + }); + + let status = loop { + match child.try_wait() { + Ok(Some(status)) => break Ok(status), + Ok(None) if started.elapsed() < Duration::from_secs(2) => { + thread::sleep(Duration::from_millis(10)); + } + Ok(None) => { + let _ = child.kill(); + let _ = child.wait(); + break Err(kind_err("paste", format!("{name} timed out"))); + } + Err(error) => { + let _ = child.kill(); + let _ = child.wait(); + break Err(kind_err("paste", format!("{name} wait failed: {error}"))); + } + } + }; + let write_result = writer.map(|writer| writer.join()); + if let Some(stderr_drain) = stderr_drain { + let _ = stderr_drain.join(); + } + let status = status?; + if let Some(write_result) = write_result { + write_result + .map_err(|_| kind_err("paste", format!("{name} input worker failed")))? + .map_err(|error| kind_err("paste", format!("{name} input failed: {error}")))?; + } + if status.success() { + return Ok(()); + } + Err(kind_err( + "paste", + format!("{name} exited with status {status}"), + )) +} + +#[cfg(test)] +mod tests { + use super::{ + clipboard_still_staged, linux_helper_ready_with, linux_tool_plan, + should_start_clipboard_only, wayland_live_type_tool, CaptureOrigin, ClipboardLease, + ClipboardSnapshot, DeliveryOutcome, DictationOutput, LinuxTool, OutputSession, + START_EVENT_DEDUPE_WINDOW, + }; + use std::sync::mpsc; + use std::thread; + use std::time::{Duration, Instant}; + + #[test] + fn clipboard_restore_never_overwrites_a_new_user_copy() { + assert!(clipboard_still_staged( + "dictated text", + Some("dictated text") + )); + assert!(!clipboard_still_staged( + "dictated text", + Some("new user copy") + )); + assert!(!clipboard_still_staged("dictated text", None)); + } + + #[test] + fn rich_clipboard_formats_win_over_plain_text_fallbacks() { + let html = super::snapshot_from_formats( + None, + Some("formatted".to_owned()), + None, + Some("formatted".to_owned()), + ); + assert!(matches!( + html, + ClipboardSnapshot::Html { alt_text: Some(ref text), .. } if text == "formatted" + )); + + let files = super::snapshot_from_formats( + Some(vec![std::path::PathBuf::from("/tmp/example.wav")]), + None, + None, + Some("file:///tmp/example.wav".to_owned()), + ); + assert!(matches!(files, ClipboardSnapshot::Files(ref paths) if paths.len() == 1)); + } + + #[test] + fn post_stage_failure_keeps_the_transcript_without_another_clipboard_write() { + let output = DictationOutput::default(); + { + let mut state = output.inner.state.lock().unwrap(); + state.clipboard_generation = 11; + state.active = Some(OutputSession { + id: 7, + started_at: Instant::now(), + target: None, + clipboard_only: false, + clipboard: Some(ClipboardLease { + original: ClipboardSnapshot::Text("original".to_owned()), + generation: 11, + staged: "transcript".to_owned(), + staged_at: Instant::now(), + }), + }); + } + + output.latch_staged_copy_only(7, 11, "transcript").unwrap(); + + let state = output.inner.state.lock().unwrap(); + let session = state.active.as_ref().unwrap(); + assert!(session.clipboard_only); + assert!(session.clipboard.is_none()); + assert_eq!(state.clipboard_generation, 12); + } + + #[test] + fn clipboard_only_live_type_failure_is_known_to_precede_emission() { + let output = DictationOutput::default(); + output.inner.state.lock().unwrap().active = Some(OutputSession { + id: 9, + started_at: Instant::now(), + target: None, + clipboard_only: true, + clipboard: None, + }); + + let error = output.type_delta(9, "not emitted", 0).unwrap_err(); + + assert!(error.starts_with("preflight:"), "{error}"); + } + + #[test] + fn wayland_tool_plan_matches_compositor_capabilities() { + assert_eq!( + linux_tool_plan(true, "KDE"), + vec![LinuxTool::Dotool, LinuxTool::Ydotool] + ); + assert_eq!( + linux_tool_plan(true, "ubuntu:GNOME"), + vec![LinuxTool::Dotool, LinuxTool::Ydotool] + ); + assert_eq!( + linux_tool_plan(true, "sway"), + vec![LinuxTool::Wtype, LinuxTool::Dotool, LinuxTool::Ydotool] + ); + assert_eq!( + linux_tool_plan(true, "plasma;KDE"), + vec![LinuxTool::Dotool, LinuxTool::Ydotool] + ); + assert!(linux_tool_plan(false, "KDE").is_empty()); + } + + #[test] + fn wayland_live_typing_requires_a_revision_capable_backend() { + assert_eq!(wayland_live_type_tool("KDE"), None); + assert_eq!(wayland_live_type_tool("ubuntu:GNOME"), None); + assert_eq!(wayland_live_type_tool("sway"), Some(LinuxTool::Wtype)); + } + + #[test] + fn ydotool_requires_a_reachable_daemon() { + assert!(!linux_helper_ready_with( + LinuxTool::Ydotool, + |command| command == "ydotool", + |command, args| command == "ydotool" && args == ["debug"] && false, + || true, + )); + assert!(linux_helper_ready_with( + LinuxTool::Ydotool, + |_| true, + |command, args| command == "ydotool" && args == ["debug"], + || true, + )); + assert!(linux_helper_ready_with( + LinuxTool::Wtype, + |_| true, + |_, _| panic!("non-daemon helper should not be probed"), + || panic!("compositor helper should not probe uinput"), + )); + } + + #[test] + fn dotool_requires_writable_uinput_before_selection() { + assert!(!linux_helper_ready_with( + LinuxTool::Dotool, + |_| true, + |_, _| panic!("dotool should not run a command probe"), + || false, + )); + assert!(linux_helper_ready_with( + LinuxTool::Dotool, + |_| true, + |_, _| panic!("dotool should not run a command probe"), + || true, + )); + } + + #[test] + fn delivery_outcome_has_a_stable_wire_shape() { + assert_eq!( + serde_json::to_string(&DeliveryOutcome::Inserted).unwrap(), + "\"inserted\"" + ); + assert_eq!( + serde_json::to_string(&DeliveryOutcome::Copied).unwrap(), + "\"copied\"" + ); + } + + #[test] + fn missing_target_is_copy_only_unless_wayland_insertion_is_explicitly_enabled() { + assert!(should_start_clipboard_only( + CaptureOrigin::Shortcut, + false, + false + )); + assert!(!should_start_clipboard_only( + CaptureOrigin::Shortcut, + false, + true + )); + assert!(should_start_clipboard_only( + CaptureOrigin::Tray, + false, + true + )); + assert!(!should_start_clipboard_only( + CaptureOrigin::Shortcut, + true, + false + )); + } + + #[test] + fn accepted_restart_survives_late_finish_of_the_previous_session() { + let output = DictationOutput::default(); + let first = output.begin_session(CaptureOrigin::Shortcut); + let duplicate = output.begin_session(CaptureOrigin::Shortcut); + assert_eq!(duplicate, first); + + output + .inner + .state + .lock() + .unwrap() + .active + .as_mut() + .unwrap() + .started_at = Instant::now() - START_EVENT_DEDUPE_WINDOW - Duration::from_millis(1); + let restarted = output.begin_session(CaptureOrigin::Shortcut); + assert_ne!(restarted, first); + output.activate_session(restarted).unwrap(); + output.finish_session(first); + assert_eq!(output.current_session_id(), Some(restarted)); + } + + #[test] + fn rejected_restart_candidate_does_not_invalidate_the_active_session() { + let output = DictationOutput::default(); + let active = output.begin_session(CaptureOrigin::Shortcut); + output + .inner + .state + .lock() + .unwrap() + .active + .as_mut() + .unwrap() + .started_at = Instant::now() - START_EVENT_DEDUPE_WINDOW - Duration::from_millis(1); + + let rejected = output.begin_session(CaptureOrigin::Shortcut); + output.reject_session_candidate(rejected); + + assert_eq!(output.current_session_id(), Some(active)); + assert!(output.require_session(active).is_ok()); + output.reject_session_candidate(active); + assert!(output.require_session(active).is_ok()); + } + + #[test] + fn busy_output_does_not_delay_capture_or_split_duplicate_start_events() { + let output = DictationOutput::default(); + let operation = output.inner.operation.lock().unwrap(); + let (captured_tx, captured_rx) = mpsc::channel(); + let (finished_tx, finished_rx) = mpsc::channel(); + let workers: Vec<_> = (0..2) + .map(|_| { + let worker_output = output.clone(); + let captured_tx = captured_tx.clone(); + let finished_tx = finished_tx.clone(); + thread::spawn(move || { + let id = worker_output.begin_session_with( + CaptureOrigin::Shortcut, + || { + captured_tx.send(()).unwrap(); + None + }, + false, + ); + finished_tx.send(id).unwrap(); + id + }) + }) + .collect(); + + for _ in 0..2 { + captured_rx + .recv_timeout(Duration::from_millis(100)) + .expect("focus capture waited behind the output operation lock"); + } + let ids: Vec<_> = (0..2) + .map(|_| { + finished_rx + .recv_timeout(Duration::from_millis(100)) + .expect("session reservation waited behind the output operation lock") + }) + .collect(); + assert_eq!(ids[0], ids[1]); + drop(operation); + + for worker in workers { + assert!(worker.join().is_ok()); + } + } + + #[test] + fn duplicate_start_uses_event_time_not_lock_admission_time() { + let first = Instant::now() - START_EVENT_DEDUPE_WINDOW - Duration::from_millis(20); + let duplicate = first + Duration::from_millis(5); + assert!(super::start_events_duplicate(first, duplicate)); + } + + #[test] + fn tray_target_prefers_pointer_prime_then_x11_action_capture() { + assert_eq!( + super::choose_session_target(CaptureOrigin::Tray, Some("action"), Some("primed")), + Some("primed"), + ); + assert_eq!( + super::choose_session_target(CaptureOrigin::Tray, Some("x11-action"), None), + Some("x11-action"), + ); + assert_eq!( + super::choose_session_target(CaptureOrigin::Tray, None::<&str>, None), + None, + ); + } +} diff --git a/frontend/src-tauri/src/lib.rs b/frontend/src-tauri/src/lib.rs index 80c8d9f41..39c2bad62 100644 --- a/frontend/src-tauri/src/lib.rs +++ b/frontend/src-tauri/src/lib.rs @@ -7,33 +7,36 @@ //! backend – spawn backend process, port probing, log paths //! commands – Tauri IPC commands (sysinfo, logs, HF cache, paste, tray, dictation) -pub mod config; -pub mod setup; -pub mod bootstrap; -pub mod tools; pub mod backend; +pub mod blank_guard; +pub mod bootstrap; pub mod commands; -pub mod dictation_shortcut; +pub mod config; pub mod crash; +pub mod dictation_output; +pub mod dictation_shortcut; pub mod reset; +pub mod setup; +pub mod tools; pub mod uninstall; pub mod updater_channel; -pub mod blank_guard; #[cfg(target_os = "linux")] pub mod wayland_shortcut; +use std::collections::VecDeque; use std::process::Child; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; use std::time::Duration; -use tauri::{Emitter, Manager}; use tauri::menu::{MenuBuilder, MenuItemBuilder}; use tauri::tray::TrayIconBuilder; +use tauri::{Emitter, Manager}; use tauri_plugin_positioner::{Position, WindowExt}; -use crate::bootstrap::{BootstrapStage, BootstrapState, set_stage}; +use crate::bootstrap::{set_stage, BootstrapStage, BootstrapState}; use crate::config::load_config; +use crate::dictation_output::CaptureOrigin; use crate::dictation_shortcut::DictationShortcutManager; // ── Port ────────────────────────────────────────────────────────────────── @@ -63,11 +66,32 @@ pub struct AppFlags { /// the tray icon), so that same call keeps this in step. pub dictating: AtomicBool, pub capture: Mutex, + pub output: dictation_output::DictationOutput, } pub struct CaptureDispatchState { - pub ready: bool, - pub pending: Option, + pub(crate) ready: bool, + pub(crate) pending: VecDeque, +} + +impl Default for CaptureDispatchState { + fn default() -> Self { + Self { + ready: false, + pending: VecDeque::new(), + } + } +} + +#[derive(Clone, serde::Serialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct DictationCapturePayload { + pub(crate) session_id: u64, +} + +pub(crate) struct CaptureEvent { + pub(crate) name: &'static str, + pub(crate) payload: DictationCapturePayload, } pub struct TrayHandle { @@ -84,8 +108,24 @@ fn dictation_capture_event(action: &str, dictating: bool) -> &'static str { } pub fn dispatch_dictation_capture(app: &tauri::AppHandle, action: &str) { + dispatch_dictation_capture_from(app, action, CaptureOrigin::Shortcut); +} + +fn dispatch_dictation_capture_from(app: &tauri::AppHandle, action: &str, origin: CaptureOrigin) { let flags = app.state::(); let event = dictation_capture_event(action, flags.dictating.load(Ordering::SeqCst)); + let session_id = if event == "tray-dictate" { + flags.output.begin_session(origin) + } else if let Some(session_id) = flags.output.current_session_id() { + session_id + } else { + log::warn!("Dictation capture '{action}' ignored — no active output session"); + return; + }; + let capture_event = CaptureEvent { + name: event, + payload: DictationCapturePayload { session_id }, + }; let Ok(mut capture) = flags.capture.lock() else { log::warn!("Dictation capture state lock poisoned"); return; @@ -94,7 +134,7 @@ pub fn dispatch_dictation_capture(app: &tauri::AppHandle, action: &str) { // A press that reaches Rust but produces no recording is otherwise // indistinguishable from one the compositor never delivered, so say // which side of the handshake the press left on. - if let Err(error) = app.emit(event, ()) { + if let Err(error) = app.emit(event, capture_event.payload) { log::warn!("Dictation capture '{action}' could not emit {event}: {error}"); } else { log::info!("Dictation capture '{action}' emitted as {event}"); @@ -103,21 +143,35 @@ pub fn dispatch_dictation_capture(app: &tauri::AppHandle, action: &str) { log::warn!( "Dictation capture '{action}' queued — the capture window has not registered yet" ); - capture.pending = Some(action.to_owned()); + capture.pending.push_back(capture_event); } } #[cfg(test)] mod dictation_capture_tests { - use super::dictation_capture_event; + use super::{ + dictation_capture_event, CaptureDispatchState, CaptureEvent, DictationCapturePayload, + }; #[test] fn toggle_starts_when_idle_and_stops_when_recording() { assert_eq!(dictation_capture_event("toggle", false), "tray-dictate"); - assert_eq!( - dictation_capture_event("toggle", true), - "tray-dictate-stop" - ); + assert_eq!(dictation_capture_event("toggle", true), "tray-dictate-stop"); + } + + #[test] + fn readiness_queue_preserves_press_then_release() { + let mut state = CaptureDispatchState::default(); + state.pending.push_back(CaptureEvent { + name: "tray-dictate", + payload: DictationCapturePayload { session_id: 7 }, + }); + state.pending.push_back(CaptureEvent { + name: "tray-dictate-stop", + payload: DictationCapturePayload { session_id: 7 }, + }); + let names: Vec<_> = state.pending.into_iter().map(|event| event.name).collect(); + assert_eq!(names, ["tray-dictate", "tray-dictate-stop"]); } } @@ -374,12 +428,19 @@ mod pill_noactivate_tests { WS_EX_NOACTIVATE_BIT, "NOACTIVATE bit must be set" ); - assert_eq!(updated & topmost, topmost, "pre-existing style bits must survive"); + assert_eq!( + updated & topmost, + topmost, + "pre-existing style bits must survive" + ); } #[test] fn idempotent_if_already_noactivate() { - assert_eq!(with_noactivate_style(WS_EX_NOACTIVATE_BIT), WS_EX_NOACTIVATE_BIT); + assert_eq!( + with_noactivate_style(WS_EX_NOACTIVATE_BIT), + WS_EX_NOACTIVATE_BIT + ); } #[test] @@ -408,7 +469,11 @@ pub fn run() { if pill_mode { log::info!( "Starting in pill (dictation-only) mode (source: {})", - if cli_pill { "--pill flag" } else { "config.launch_as_widget" } + if cli_pill { + "--pill flag" + } else { + "config.launch_as_widget" + } ); // On macOS, hide the Dock icon in pill mode so only the tray shows. // This is handled after the app builds via set_activation_policy. @@ -476,7 +541,11 @@ pub fn run() { commands::read_log_tail, commands::hf_cache_scan, commands::simulate_paste, + commands::copy_dictation_output_session, commands::simulate_type, + commands::activate_dictation_output_session, + commands::reject_dictation_output_session, + commands::finish_dictation_output_session, commands::check_accessibility, commands::open_accessibility_settings, commands::check_microphone, @@ -562,6 +631,7 @@ pub fn run() { .decorations(false) .always_on_top(true) .visible(false) + .focused(false) .skip_taskbar(true) .center() // Stamp the window's identity BEFORE any app script runs. @@ -586,15 +656,23 @@ pub fn run() { if let Ok(win) = &result { mark_pill_noactivate(win); } + // Wayland cannot reactivate an arbitrary foreign client. The + // GTK toplevel therefore must never accept focus when mapped. + #[cfg(target_os = "linux")] + if let Ok(win) = &result { + use gtk::prelude::GtkWindowExt; + if let Ok(gtk_window) = win.gtk_window() { + gtk_window.set_accept_focus(false); + gtk_window.set_focus_on_map(false); + } + } } app.manage(AppFlags { quitting: AtomicBool::new(false), dictating: AtomicBool::new(false), - capture: Mutex::new(CaptureDispatchState { - ready: false, - pending: None, - }), + capture: Mutex::new(CaptureDispatchState::default()), + output: dictation_output::DictationOutput::default(), }); app.manage(TrayHandle { tray: Mutex::new(None), @@ -700,6 +778,20 @@ pub fn run() { .icon(app.default_window_icon().unwrap().clone()) .menu(&tray_menu) .tooltip(if pill_mode_tray { "VoiceStudio Dictation" } else { "VoiceStudio" }) + .on_tray_icon_event(|tray, event| { + if matches!( + event, + tauri::tray::TrayIconEvent::Click { + button_state: tauri::tray::MouseButtonState::Down, + .. + } + ) { + tray.app_handle() + .state::() + .output + .prime_tray_target(); + } + }) .on_menu_event(move |app, event| { match event.id().as_ref() { "show" => { @@ -760,9 +852,9 @@ pub fn run() { // current by the frontend's existing // `set_tray_recording` call on every start and stop. if app.state::().dictating.load(Ordering::SeqCst) { - dispatch_dictation_capture(app, "stop"); + dispatch_dictation_capture_from(app, "stop", CaptureOrigin::Tray); } else { - dispatch_dictation_capture(app, "start"); + dispatch_dictation_capture_from(app, "start", CaptureOrigin::Tray); } } "settings" => { diff --git a/frontend/src-tauri/tests/backend_lifecycle.rs b/frontend/src-tauri/tests/backend_lifecycle.rs index 8c7ef1a62..2f56b0860 100644 --- a/frontend/src-tauri/tests/backend_lifecycle.rs +++ b/frontend/src-tauri/tests/backend_lifecycle.rs @@ -217,7 +217,8 @@ impl TestApp { app.manage(AppFlags { quitting: AtomicBool::new(false), dictating: AtomicBool::new(false), - capture: Mutex::new(CaptureDispatchState { ready: false, pending: None }), + capture: Mutex::new(CaptureDispatchState::default()), + output: app_lib::dictation_output::DictationOutput::default(), }); let stage = Arc::new(Mutex::new(BootstrapStage::Checking)); let logs: Arc>> = Arc::new(Mutex::new(Vec::new())); diff --git a/frontend/src/components/CaptureWidget.jsx b/frontend/src/components/CaptureWidget.jsx index c802873dc..4791c4b4e 100644 --- a/frontend/src/components/CaptureWidget.jsx +++ b/frontend/src/components/CaptureWidget.jsx @@ -4,6 +4,7 @@ import { X, Loader } from 'lucide-react'; import { toast } from 'react-hot-toast'; import { useAppStore } from '../store'; import { useTranslation } from 'react-i18next'; +import { invoke as tauriInvoke } from '@tauri-apps/api/core'; import { API, apiFetch } from '../api/client'; import { authenticatedWsUrl } from '../api/authSession'; @@ -129,22 +130,30 @@ export function isSherpaModel(id) { return typeof id === 'string' && id.startsWith('sherpa-'); } -/** - * Classify a sherpa `final` message against the utterances committed so far. - * Pure + exported for unit testing the live-streaming state machine. - * • 'summary' — the authoritative EOF summary (text === the committed - * join): finalise, don't re-paste. - * • 'utterance' — a new per-utterance commit: paste it live + append. - * • 'terminator' — an empty no-speech EOF final with nothing committed: - * finalise (resolve the pill). - * • 'ignore' — empty final but utterances exist (covered by the summary). - */ -export function classifySherpaFinal(segText, committed) { - const text = (segText || '').trim(); - const joined = (committed || []).join(' ').trim(); - if (text && text === joined && joined !== '') return 'summary'; - if (!text) return committed && committed.length ? 'ignore' : 'terminator'; - return 'utterance'; +/** Classify the backend's explicit sherpa final-frame contract. */ +export function classifySherpaFinal(message) { + const text = typeof message?.text === 'string' ? message.text.trim() : ''; + if (message?.final_kind === 'summary') return text ? 'summary' : 'terminator'; + if (message?.final_kind === 'utterance') return text ? 'utterance' : 'ignore'; + return 'ignore'; +} + +/** Return the EOF-summary suffix that has not already been committed live. */ +export function sherpaSummaryTail(summaryText, committed) { + const summary = (summaryText || '').trim(); + const delivered = (committed || []).join(' ').trim(); + if (!delivered) return summary; + if (summary === delivered) return ''; + const prefix = `${delivered} `; + return summary.startsWith(prefix) ? summary.slice(prefix.length).trim() : ''; +} + +/** Combine delivery outcomes without ever hiding a clipboard-only fallback. */ +export function aggregateDeliveryKind(current, next) { + const priority = { noop: 0, pasted: 1, inserted: 2, copied: 3 }; + if (!current) return next || null; + if (!next) return current; + return (priority[next] || 0) > (priority[current] || 0) ? next : current; } /** @@ -184,79 +193,145 @@ export function computeTypeDelta(prevTyped, nextText) { * `{ kind, message }`. The Rust command prefixes its Err strings with the * failing layer — "a11y:" (macOS Accessibility not granted; the pill offers * open_accessibility_settings), "clipboard:" (couldn't write/restore the user - * clipboard) or "paste:" (the synthetic ⌘V/Ctrl+V itself failed). Pure + - * exported for unit testing. + * clipboard), "preflight:" (input was rejected before any key could be emitted) + * or "paste:" (the synthetic ⌘V/Ctrl+V itself failed). Pure + exported for + * unit testing. */ export function parsePasteError(err) { const raw = typeof err === 'string' ? err : (err && err.message) || String(err ?? ''); - for (const kind of ['a11y', 'clipboard', 'paste']) { + for (const kind of ['a11y', 'clipboard', 'paste', 'preflight']) { if (raw.startsWith(`${kind}:`)) return { kind, message: raw.slice(kind.length + 1).trim() }; } return { kind: 'paste', message: raw }; } -// Deliver a transcript to the user: best-effort WebView clipboard copy (works -// in browser mode; in Tauri the unfocused widget window can't always reach the -// WebView clipboard — #287) then, inside Tauri, the native simulate_paste -// (which saves the user clipboard, writes + sends ⌘V/Ctrl+V, then restores). -// Returns { ok: true, kind: 'pasted' | 'copied' } or { ok: false, error }. -// The caller renders the TRUE outcome — "Pasted" is never shown unless the -// invoke actually resolved Ok. -async function deliverText(text) { - let copyErr = null; - try { - await copyText(text); - } catch (err) { - // Only fatal when there is no native path below to write it instead. - copyErr = err; - } +// Native sessions own clipboard preservation and the captured focus target, +// so the widget must never pre-write the WebView clipboard in Tauri. +async function deliverText(text, sessionId = null) { if (!inTauri()) { + let copyErr = null; + try { + await copyText(text); + } catch (err) { + copyErr = err; + } if (copyErr) { return { ok: false, error: { kind: 'clipboard', message: String(copyErr?.message || copyErr) }, }; } - return { ok: true, kind: 'copied' }; + return { ok: true, kind: 'copied', copySource: 'webview' }; + } + if (!sessionId) { + return { + ok: false, + error: { kind: 'paste', message: 'native output session unavailable' }, + }; } try { const { invoke } = await import('@tauri-apps/api/core'); - await invoke('simulate_paste', { text }); - return { ok: true, kind: 'pasted' }; + const outcome = await invoke('simulate_paste', { text, sessionId }); + return { + ok: true, + kind: outcome === 'copied' || outcome === 'inserted' ? outcome : 'pasted', + copySource: outcome === 'copied' ? 'native' : null, + }; } catch (err) { - return { ok: false, error: parsePasteError(err) }; + const nativeError = parsePasteError(err); + if (nativeError.kind === 'clipboard') { + try { + // Linux WebKit and the native clipboard backend do not always support + // the same display/session combinations. Try the WebView path only + // after native delivery has failed, preserving the captured target. + await copyText(text); + return { ok: true, kind: 'copied', copySource: 'webview' }; + } catch (copyErr) { + return { + ok: false, + error: { kind: 'clipboard', message: String(copyErr?.message || copyErr) }, + }; + } + } + return { ok: false, error: nativeError }; } } -// Live paste of a committed utterance into whatever app has focus. Same -// clipboard+⌘V/Ctrl+V path as the session final, so each silence-endpoint -// utterance lands in the target field as the user pauses — that's what makes -// streaming dictation feel live. Returns the deliverText outcome so the +// Preserve an authoritative transcript without emitting keyboard input. This +// is the only safe rescue after live synthesis may have left a partial prefix. +async function copySessionText(text, sessionId) { + if (!inTauri() || !sessionId) { + return { + ok: false, + error: { kind: 'clipboard', message: 'native output session unavailable' }, + }; + } + try { + const { invoke } = await import('@tauri-apps/api/core'); + const outcome = await invoke('copy_dictation_output_session', { text, sessionId }); + if (outcome !== 'copied') { + return { + ok: false, + error: { kind: 'clipboard', message: 'native clipboard delivery was not confirmed' }, + }; + } + return { ok: true, kind: 'copied', copySource: 'native' }; + } catch (err) { + const nativeError = parsePasteError(err); + if (nativeError.kind === 'clipboard') { + try { + await copyText(text); + return { ok: true, kind: 'copied', copySource: 'webview' }; + } catch (copyErr) { + return { + ok: false, + error: { kind: 'clipboard', message: String(copyErr?.message || copyErr) }, + }; + } + } + return { ok: false, error: nativeError }; + } +} + +// Deliver a committed utterance to the session's captured target so each +// silence endpoint lands as the user pauses. Returns the native outcome so the // session can surface a failed segment instead of pretending it landed. -async function pasteSegment(text) { +async function pasteSegment(text, sessionId) { if (!text) return { ok: true, kind: 'noop' }; - return deliverText(text); + return deliverText(text, sessionId); } -// Live, word-by-word typing of the in-flight utterance into whatever app has -// focus — the native-dictation experience (words appear AS you speak, not only -// on pauses). Given the delta vs what we last typed, it backspaces any revised -// tail then types the corrected suffix via the `simulate_type` Tauri command -// (one round trip). Returns true on success, false if the input layer was -// unavailable (not in Tauri, or accessibility not granted) so the caller can -// fall back to the paste path for that segment without double-inserting. -async function typeDelta({ backspaces, text }) { - if (!backspaces && !text) return true; - if (!inTauri()) return false; +// Live, word-by-word typing of the in-flight utterance into the session's +// captured target (words appear AS you speak, not only on pauses). Given the +// delta vs what we last typed, it backspaces any revised tail then types the +// corrected suffix via the `simulate_type` Tauri command (one round trip). The +// structured outcome lets the caller latch failures; native synthesis can +// partially emit before returning Err, so retry-pasting the whole utterance +// would not be safe. +async function typeDelta({ backspaces, text }, sessionId) { + if (!backspaces && !text) return { ok: true, kind: 'noop', mayHaveEmitted: false }; + if (!inTauri() || !sessionId) { + return { + ok: false, + error: { kind: 'paste', message: 'native output session unavailable' }, + mayHaveEmitted: false, + }; + } try { const { invoke } = await import('@tauri-apps/api/core'); - await invoke('simulate_type', { text, backspaces }); - return true; + await invoke('simulate_type', { text, backspaces, sessionId }); + return { ok: true, kind: 'inserted', mayHaveEmitted: true }; } catch (err) { - // Caller latches live typing off and pastes instead — the failure is not - // silent (a failing paste then raises the pill's error state). console.warn('simulate_type failed:', err); - return false; + const error = parsePasteError(err); + return { + ok: false, + error, + // Native marks failures detected before synthesis separately. Retrying + // those via committed delivery is safe; all other failures may have + // emitted a partial delta before the helper returned Err. + mayHaveEmitted: error.kind !== 'preflight' && error.kind !== 'a11y', + }; } } @@ -276,6 +351,7 @@ function errorLabel(t, info) { case 'clipboard': return t('capture.clipboard_error'); case 'paste': + case 'preflight': return t('capture.paste_error'); case 'mic': return t('capture.mic_denied'); @@ -289,10 +365,10 @@ function errorLabel(t, info) { * * Minimal status-only UI: live waveform (or status dot) + label + timer. * All interaction via global hotkey (hold-to-talk); Esc cancels anywhere. - * Records → transcribes → auto-pastes → auto-dismisses — and every state the - * pill shows is TRUE: "Pasted" only after simulate_paste resolved Ok, model - * download/load progress straight from the backend's status frames, and an - * actionable setup state when macOS Accessibility hasn't been granted yet. + * Records → transcribes → delivers → auto-dismisses — and every state the pill + * shows is TRUE: "Inserted" only after native synthesis succeeds, "Copied" + * after clipboard fallback, model progress from backend status frames, and an + * actionable setup state when macOS Accessibility has not been granted yet. */ export default function CaptureWidget({ onDismiss }) { const { t } = useTranslation(); @@ -303,9 +379,9 @@ export default function CaptureWidget({ onDismiss }) { const [, setLastEngine] = useState(''); const [, setLastTime] = useState(0); const [partialText, setPartialText] = useState(''); - // How the finished transcript actually reached the user: 'pasted' (native - // simulate_paste Ok) or 'copied' (clipboard only — browser mode). Drives the - // done label so the pill never claims a paste that didn't happen. + // How the finished transcript actually reached the user: 'inserted' after + // confirmed native synthesis, 'copied' for clipboard-only delivery, or the + // legacy browser 'pasted' value. Drives the done label truthfully. const [doneKind, setDoneKind] = useState(null); // { kind, message } for the error state (mic / a11y / clipboard / paste / // transcription / server). The a11y kind renders the Open-Settings action. @@ -329,12 +405,37 @@ export default function CaptureWidget({ onDismiss }) { // re-subscribing on every pref change. const modeRef = useRef(dictationMode); const enabledRef = useRef(dictationEnabled); + const prefsHydrationRef = useRef(null); useEffect(() => { modeRef.current = dictationMode; }, [dictationMode]); useEffect(() => { enabledRef.current = dictationEnabled; }, [dictationEnabled]); + const ensureDictationPrefsHydrated = useCallback(() => { + if (!prefsHydrationRef.current) { + prefsHydrationRef.current = Promise.resolve() + .then(() => loadDictationPrefs()) + .catch((err) => { + // The store keeps its cross-platform seeds when the backend is not + // ready. Readiness must still resolve so the native hotkey can work. + console.warn('dictation prefs hydration failed:', err); + }) + .then(() => { + // Zustand updates before loadDictationPrefs resolves, but React's + // selector effects may render later. Synchronise the long-lived + // native listener refs now so its first event cannot use seed prefs. + const prefs = useAppStore.getState(); + if (typeof prefs.dictationEnabled === 'boolean') { + enabledRef.current = prefs.dictationEnabled; + } + if (prefs.dictationMode === 'toggle' || prefs.dictationMode === 'hold') { + modeRef.current = prefs.dictationMode; + } + }); + } + return prefsHydrationRef.current; + }, [loadDictationPrefs]); // `state` follows the same rule, and for a sharper reason than the prefs do. // The tray listener used to depend on [state], so every single state change // tore the Tauri listener down and re-attached it through an `await import()` @@ -358,9 +459,9 @@ export default function CaptureWidget({ onDismiss }) { // still pending. Preserve that release so the completed start cannot leave // an orphaned recording behind. const holdStartRef = useRef(null); - // Linux can deliver the same shortcut through both the native global-hotkey - // plugin and the focused main-window fallback. Collapse that pair into one - // logical action without slowing intentional toggle-mode presses. + const holdStartSequenceRef = useRef(0); + // Some native accelerator stacks can deliver duplicate events. Collapse a + // near-simultaneous pair without slowing intentional toggle-mode presses. const nativeEventAtRef = useRef({ start: 0, stop: 0 }); // Sherpa live-streaming session refs. `sherpaModeRef` flips on at start when a @@ -373,11 +474,14 @@ export default function CaptureWidget({ onDismiss }) { // left alone — we never backspace across an utterance boundary). It resets to // '' each time an utterance is committed. `liveTypingRef` is seeded from the // LS_LIVE_TYPING pref at session start (default OFF — commit-only insert, no - // visible backspace storms) and latches off if a simulate_type call fails so - // the rest of the session uses the paste fallback instead of typing-then- - // also-pasting (which would double-insert). + // visible backspace storms) and latches off if a simulate_type call fails. + // A failure after possible emission is terminal; a zero-emission preflight + // safely downgrades the rest of the session to committed delivery. const typedRef = useRef(''); const liveTypingRef = useRef(false); + // A native type command may emit only part of its delta before returning Err. + // Once that happens, pasting the whole final could duplicate unknown text. + const typingFailedRef = useRef(false); // Set after an utterance commits: the next utterance's first typed delta is // prefixed with a single separating space (so we don't trail a space after the // final utterance, and words across utterances don't run together). @@ -393,6 +497,23 @@ export default function CaptureWidget({ onDismiss }) { // First delivery failure of a live session (per-utterance paste). Checked at // finalise so the pill reports the truth instead of a green "Pasted". const segmentErrorRef = useRef(null); + // Rust captures the target app before emitting the start event. Every native + // delivery in this recording carries that same lease id until finish. + const outputSessionIdRef = useRef(null); + const captureGenerationRef = useRef(0); + const deliveryKindRef = useRef(null); + // A native `copied` outcome latches Rust into clipboard-only mode. A WebView + // fallback does not, so its final summary must also use a non-inserting copy + // path rather than retrying native paste and risking duplicate insertion. + const deliveryCopySourceRef = useRef(null); + const startInFlightRef = useRef(false); + const finishInFlightRef = useRef(null); + const nativeActivationChainRef = useRef(Promise.resolve()); + const nativeStartSequenceRef = useRef(0); + // A candidate accepted while the previous microphone graph is still + // starting normally adopts that graph. If the graph instead terminates + // before startup unwinds, replay this already-activated candidate once. + const pendingNativeStartRef = useRef(null); const mediaRecorderRef = useRef(null); const chunksRef = useRef([]); @@ -418,6 +539,33 @@ export default function CaptureWidget({ onDismiss }) { const aecStopRef = useRef(null); // async teardown of the mic worklet graph const farEndUnsubRef = useRef(null); // unsubscribe from the far-end bus + const finishOutputSession = useCallback((requestedId = null) => { + const sessionId = requestedId || outputSessionIdRef.current; + if (!sessionId || !inTauri()) return Promise.resolve(true); + const previous = finishInFlightRef.current; + if (previous?.sessionId === sessionId) return previous.promise; + + const operation = { sessionId, promise: null }; + operation.promise = (async () => { + if (previous) await previous.promise; + let released = false; + try { + await tauriInvoke('finish_dictation_output_session', { sessionId }); + released = true; + } catch (err) { + console.warn('finish dictation output session failed:', err); + } finally { + if (released && outputSessionIdRef.current === sessionId) { + outputSessionIdRef.current = null; + } + if (finishInFlightRef.current === operation) finishInFlightRef.current = null; + } + return released; + })(); + finishInFlightRef.current = operation; + return operation.promise; + }, []); + const teardownAec = useCallback(async () => { try { farEndUnsubRef.current?.(); @@ -476,12 +624,12 @@ export default function CaptureWidget({ onDismiss }) { // window), so it loads the prefs itself rather than relying on the Settings // window having loaded them. useEffect(() => { - loadDictationPrefs(); - }, [loadDictationPrefs]); + void ensureDictationPrefsHydrated(); + }, [ensureDictationPrefsHydrated]); - // First-run truthfulness: without the macOS Accessibility grant neither - // simulate_paste nor simulate_type can deliver a single character — so probe - // up front and show a one-time setup pill instead of pretending to work. + // First-run truthfulness: without the macOS Accessibility grant native text + // insertion is unavailable, though the completed transcript can still fall + // back to the clipboard. Probe up front so the setup pill exposes the grant. // (Resolves true on Windows/Linux and outside Tauri.) useEffect(() => { let stale = false; @@ -539,41 +687,114 @@ export default function CaptureWidget({ onDismiss }) { (async () => { try { const { listen } = await import('@tauri-apps/api/event'); - unlistenStart = await listen('tray-dictate', () => { + unlistenStart = await listen('tray-dictate', async (event) => { const now = Date.now(); if (now - nativeEventAtRef.current.start < 150) return; nativeEventAtRef.current.start = now; + const sessionId = event?.payload?.sessionId; + if (!sessionId) { + hideWidgetWindow(); + return; + } + await ensureDictationPrefsHydrated(); if (!enabledRef.current) { // The hotkey is inert, but Rust has already shown the window. // Put it back rather than leaving an empty capsule on screen. hideWidgetWindow(); + finishOutputSession(sessionId); return; } + const sequence = ++nativeStartSequenceRef.current; const s = stateRef.current; + const startupWasInFlight = startInFlightRef.current; + const restartable = s === 'idle' || s === 'done' || s === 'error' || s === 'setup'; + if (!restartable && !startupWasInFlight) { + if (modeRef.current === 'toggle' && s === 'recording') { + stopRecordingRef.current?.(); + } + try { + await tauriInvoke('reject_dictation_output_session', { sessionId }); + } catch (err) { + console.warn('reject dictation output session failed:', err); + } + return; + } + const trackHold = modeRef.current === 'hold'; + if (trackHold) { + holdStartSequenceRef.current = sequence; + holdStartRef.current = 'starting'; + } + const clearPendingHold = () => { + if (holdStartSequenceRef.current !== sequence) return; + holdStartRef.current = null; + holdStartSequenceRef.current = 0; + }; + + const activation = nativeActivationChainRef.current.then(() => + tauriInvoke('activate_dictation_output_session', { sessionId }), + ); + nativeActivationChainRef.current = activation.catch(() => {}); + try { + await activation; + } catch (err) { + console.warn('activate dictation output session failed:', err); + clearPendingHold(); + await finishOutputSession(sessionId); + hideWidgetWindow(); + return; + } + if (cancelled || sequence !== nativeStartSequenceRef.current || !enabledRef.current) { + clearPendingHold(); + await finishOutputSession(sessionId); + return; + } + if (startupWasInFlight) { + const current = stateRef.current; + if (startInFlightRef.current) { + outputSessionIdRef.current = sessionId; + pendingNativeStartRef.current = { sessionId, trackHold, sequence }; + } else if (current === 'recording' || current === 'transcribing') { + outputSessionIdRef.current = sessionId; + } else if ( + current === 'idle' || + current === 'done' || + current === 'error' || + current === 'setup' + ) { + outputSessionIdRef.current = sessionId; + startRecordingRef.current?.(trackHold, sessionId); + } else { + clearPendingHold(); + await finishOutputSession(sessionId); + } + return; + } if (s === 'setup') { // Re-probe on each press — the user may have just granted access - // in System Settings; if so, flow straight into recording. - if (modeRef.current === 'hold') holdStartRef.current = 'starting'; - checkAccessibility().then((ok) => { - if (ok) startRecordingRef.current?.(modeRef.current === 'hold'); - else holdStartRef.current = null; + // in System Settings. A missing grant no longer blocks capture: + // native delivery can truthfully fall back to clipboard-only. + outputSessionIdRef.current = sessionId; + checkAccessibility().then(() => { + if (outputSessionIdRef.current !== sessionId) return; + startRecordingRef.current?.(modeRef.current === 'hold', sessionId); }); return; } const idle = s === 'idle' || s === 'done' || s === 'error'; if (modeRef.current === 'toggle') { // Press once to start, again to stop. - if (idle) startRecordingRef.current?.(); + if (idle) startRecordingRef.current?.(false, sessionId); else if (s === 'recording') stopRecordingRef.current?.(); } else if (idle) { // Hold mode: keydown → start. - startRecordingRef.current?.(true); + startRecordingRef.current?.(true, sessionId); } }); - unlistenStop = await listen('tray-dictate-stop', () => { + unlistenStop = await listen('tray-dictate-stop', async () => { const now = Date.now(); if (now - nativeEventAtRef.current.stop < 150) return; nativeEventAtRef.current.stop = now; + await ensureDictationPrefsHydrated(); // Only hold mode acts on release; toggle ignores it. if (modeRef.current === 'hold' && stateRef.current === 'recording') { stopRecordingRef.current?.(); @@ -581,6 +802,7 @@ export default function CaptureWidget({ onDismiss }) { holdStartRef.current = 'released'; } }); + await ensureDictationPrefsHydrated(); const { invoke } = await import('@tauri-apps/api/core'); await invoke('mark_dictation_capture_ready'); // Unmounted while the dynamic import was in flight — drop the @@ -597,6 +819,7 @@ export default function CaptureWidget({ onDismiss }) { })(); return () => { cancelled = true; + nativeStartSequenceRef.current += 1; if (unlistenStart) unlistenStart(); if (unlistenStop) unlistenStop(); }; @@ -611,6 +834,7 @@ export default function CaptureWidget({ onDismiss }) { // keyup stops. The Ctrl/Cmd+Shift+Space combo matches the documented default // shortcut; the desktop app's user-rebindable accelerator is a Tauri concern. useEffect(() => { + if (inTauri()) return; const isCombo = (e) => (e.metaKey || e.ctrlKey) && e.shiftKey && e.code === 'Space'; const onKeyDown = (e) => { if (!isCombo(e)) return; @@ -665,9 +889,17 @@ export default function CaptureWidget({ onDismiss }) { setModelStatus(null); setErrorInfo(null); setDoneKind(null); + await finishOutputSession(); await hideWidgetWindow(); if (onDismiss) onDismiss(); - }, [teardownAec, onDismiss]); + }, [finishOutputSession, teardownAec, onDismiss]); + + useEffect( + () => () => { + void finishOutputSession(); + }, + [finishOutputSession], + ); // Auto-dismiss after a beat, tracked in a ref so Esc or a fresh session can // cancel it (a stale timer must never hide a newly-started recording). @@ -700,6 +932,8 @@ export default function CaptureWidget({ onDismiss }) { // Esc = abort. Stops capture, discards the audio and any in-flight result // (nothing is pasted), closes the socket and hides the pill. const cancelSession = useCallback(() => { + captureGenerationRef.current += 1; + pendingNativeStartRef.current = null; wsHadFinalRef.current = true; // any late final/fallback result is discarded if (fallbackTimerRef.current) { clearTimeout(fallbackTimerRef.current); @@ -710,6 +944,7 @@ export default function CaptureWidget({ onDismiss }) { if (ws) ws.close(); stopCaptureGraph(); committedRef.current = []; + typingFailedRef.current = false; typeChainRef.current = Promise.resolve(); pasteChainRef.current = Promise.resolve(); setTrayRecording(false); @@ -740,7 +975,9 @@ export default function CaptureWidget({ onDismiss }) { // path, including any added later, so no single call site can reintroduce it. useEffect(() => { if (state !== 'error' || transcript) return; - const t = setTimeout(() => dismiss(), ERROR_AUTO_DISMISS_MS); + const t = setTimeout(() => { + if (!startInFlightRef.current) dismiss(); + }, ERROR_AUTO_DISMISS_MS); return () => clearTimeout(t); }, [state, transcript, dismiss]); @@ -748,7 +985,18 @@ export default function CaptureWidget({ onDismiss }) { // → auto-dismiss on success. A failed delivery is an error state (with the // Accessibility action when that's the fix), never a fake "Pasted". const applyResult = useCallback( - async (data) => { + async ( + data, + sessionId = outputSessionIdRef.current, + generation = captureGenerationRef.current, + ) => { + const isCurrent = () => + generation === captureGenerationRef.current && + (!sessionId || outputSessionIdRef.current === sessionId); + if (!isCurrent()) { + await finishOutputSession(sessionId); + return; + } // Wave 2.1: the backend may attach an LLM-refined version of the final // text (filler words removed, self-corrections applied). Paste/show the // refined text when present; the raw text is kept in history alongside. @@ -764,39 +1012,134 @@ export default function CaptureWidget({ onDismiss }) { if (!finalText) { // No speech — brief notice, then auto-dismiss. + await finishOutputSession(sessionId); + if (captureGenerationRef.current !== generation) return; setDoneKind(null); setState('done'); scheduleDismiss(2500); return; } - const res = await deliverText(finalText); + const res = await deliverText(finalText, sessionId); + if (!isCurrent()) { + await finishOutputSession(sessionId); + return; + } + await finishOutputSession(sessionId); + if (captureGenerationRef.current !== generation) return; if (res.ok) { setDoneKind(res.kind); setState('done'); scheduleDismiss(1500); } else { - // The transcript did NOT land. deliverText copied it to the clipboard - // first when it could, but the pill must say what failed — and stay - // up until the user acts (no auto-dismiss on errors). + // The transcript did NOT land. Keep the pill up until the user acts; + // native delivery owns clipboard fallback and reports that as success. setErrorInfo(res.error); setState('error'); } }, - [scheduleDismiss], + [finishOutputSession, scheduleDismiss], ); + const queueSegmentPaste = useCallback((text) => { + const sessionId = outputSessionIdRef.current; + const generation = captureGenerationRef.current; + const typeBarrier = typeChainRef.current; + const isCurrent = () => + generation === captureGenerationRef.current && + (!sessionId || outputSessionIdRef.current === sessionId); + const run = async () => { + // A summary can race the last live-type invoke. Wait for that command's + // outcome before deciding whether paste is safe. + await typeBarrier; + if (!isCurrent()) return; + if (typingFailedRef.current) return; + // Once native delivery falls back to clipboard-only, wait for the + // authoritative summary instead of repeatedly replacing it with pieces. + if (deliveryKindRef.current === 'copied') return; + const result = await pasteSegment(text, sessionId); + if (!isCurrent()) return; + if (result.ok) { + deliveryKindRef.current = aggregateDeliveryKind(deliveryKindRef.current, result.kind); + if (result.copySource) deliveryCopySourceRef.current = result.copySource; + } else if (!segmentErrorRef.current) { + segmentErrorRef.current = result.error; + } + }; + const previous = pasteChainRef.current; + pasteChainRef.current = previous.then(run, run); + return pasteChainRef.current; + }, []); + // Finalise a sherpa LIVE-streaming session. The per-utterance finals were // already delivered into the focused field as the user paused, so this does // NOT re-paste — it shows the authoritative full transcript in the pill, // reports any segment that failed to land, and auto-dismisses on success. // The EOF-summary `final` (or an early socket close) drives this. const finalizeSession = useCallback( - async (data) => { + async ( + data, + sessionId = outputSessionIdRef.current, + generation = captureGenerationRef.current, + ) => { + const isCurrent = () => + generation === captureGenerationRef.current && + (!sessionId || outputSessionIdRef.current === sessionId); // Wait for in-flight per-utterance deliveries first — the outcome the // pill reports must be the settled one, not a hopeful guess. - await Promise.all([pasteChainRef.current, typeChainRef.current]); + // A pending type preflight can downgrade an utterance and enqueue its + // committed delivery as the type chain settles. Read pasteChain only + // after that point so finalisation cannot race the newly queued paste. + await typeChainRef.current; + if (!isCurrent()) { + await finishOutputSession(sessionId); + return; + } + await pasteChainRef.current; + if (!isCurrent()) { + await finishOutputSession(sessionId); + return; + } const fullText = data.refined_text || data.text || ''; + if (typingFailedRef.current && fullText) { + const rescue = await copySessionText(fullText, sessionId); + if (!isCurrent()) { + await finishOutputSession(sessionId); + return; + } + if (rescue.ok) { + deliveryKindRef.current = aggregateDeliveryKind(deliveryKindRef.current, rescue.kind); + if (rescue.copySource) deliveryCopySourceRef.current = rescue.copySource; + segmentErrorRef.current = null; + } else { + segmentErrorRef.current = rescue.error; + } + } else if (deliveryKindRef.current === 'copied' && fullText) { + // Only a native `copied` outcome latches Rust clipboard-only. If native + // delivery failed and WebKit supplied the clipboard fallback, retrying + // simulate_paste could recover and duplicate already inserted segments. + const refresh = + deliveryCopySourceRef.current === 'webview' + ? await copyText(fullText) + .then(() => ({ ok: true, kind: 'copied', copySource: 'webview' })) + .catch((err) => ({ + ok: false, + error: { kind: 'clipboard', message: String(err?.message || err) }, + })) + : await deliverText(fullText, sessionId); + if (!isCurrent()) { + await finishOutputSession(sessionId); + return; + } + if (refresh.ok) { + deliveryKindRef.current = aggregateDeliveryKind(deliveryKindRef.current, refresh.kind); + if (refresh.copySource) deliveryCopySourceRef.current = refresh.copySource; + } else if (!segmentErrorRef.current) { + segmentErrorRef.current = refresh.error; + } + } + await finishOutputSession(sessionId); + if (captureGenerationRef.current !== generation) return; setTranscript(fullText); setLastEngine(data.engine || 'sherpa-onnx-asr'); setLastTime(data.transcription_time_s || 0); @@ -806,29 +1149,38 @@ export default function CaptureWidget({ onDismiss }) { // re-record — that would duplicate the session. setPartialText(''); committedRef.current = []; - if (segmentErrorRef.current && fullText) { + typingFailedRef.current = false; + const deliveryKind = deliveryKindRef.current; + deliveryKindRef.current = null; + deliveryCopySourceRef.current = null; + if (segmentErrorRef.current) { // At least one utterance never reached the target app — the truthful // outcome is an error (with the a11y action when relevant). setErrorInfo(segmentErrorRef.current); setState('error'); return; } - setDoneKind(fullText ? (inTauri() ? 'pasted' : 'copied') : null); + setDoneKind(fullText ? deliveryKind : null); setState('done'); scheduleDismiss(fullText ? 1500 : 2500); }, - [scheduleDismiss], + [finishOutputSession, scheduleDismiss], ); // Type the recognizer's latest revision of the in-flight utterance into the // focused field, reconciling against what we typed before via a prefix diff. // Serialised on `typeChainRef` so concurrent partials can't interleave. If the - // delta typing fails (no Tauri / no a11y grant), latch live-typing off and let - // the per-utterance paste fallback carry the text instead — never both. + // delta typing fails, latch live typing off. Zero-emission preflight failures + // downgrade to committed delivery; a possibly partial emission is terminal. const liveType = useCallback((nextText) => { if (!liveTypingRef.current) return typeChainRef.current; + const generation = captureGenerationRef.current; + const sessionId = outputSessionIdRef.current; + const isCurrent = () => + generation === captureGenerationRef.current && + (!sessionId || outputSessionIdRef.current === sessionId); const run = async () => { - if (!liveTypingRef.current) return; + if (!isCurrent() || !liveTypingRef.current) return; // Prefix the first delta of a new (non-first) utterance with a separator, // tracked inside typedRef so the diff stays self-consistent. let target = nextText || ''; @@ -838,452 +1190,582 @@ export default function CaptureWidget({ onDismiss }) { } const delta = computeTypeDelta(typedRef.current, target); if (delta.noop) return; - const ok = await typeDelta(delta); - if (ok) { + const result = await typeDelta(delta, sessionId); + if (!isCurrent()) return; + if (result.ok) { typedRef.current = target; + deliveryKindRef.current = aggregateDeliveryKind(deliveryKindRef.current, result.kind); } else { - // Input layer unavailable — stop typing for the rest of the session so - // we don't half-type. The paste path (pasteSegment on finals) takes over. + // Stop live typing for the rest of the session. A native preflight or + // Accessibility failure emits nothing, so committed paste/copy remains + // safe only when no earlier delta for this utterance landed. Synthesis + // failures and an already-inserted prefix both make a whole retry unsafe. liveTypingRef.current = false; + if (result.mayHaveEmitted || typedRef.current !== '') { + typingFailedRef.current = true; + if (!segmentErrorRef.current) segmentErrorRef.current = result.error; + } } }; - typeChainRef.current = typeChainRef.current.then(run, run); + const previous = typeChainRef.current; + typeChainRef.current = previous.then(run, run); return typeChainRef.current; }, []); - const startRecording = useCallback(async () => { - // Pre-flight: when the OS itself reports the mic grant as DENIED, - // getUserMedia can only throw an opaque NotAllowedError — skip it and - // show the guided path (per-OS hint + Open Settings deep-link) instead. - // 'prompt'/'granted'/'unknown' proceed exactly as before (getUserMedia - // raises the OS prompt; micError.js stays the reactive fallback), and - // outside Tauri checkMicrophone() is always 'unknown' → unchanged. - if ((await checkMicrophone()) === 'denied') { - holdStartRef.current = null; - showMicDeniedGuide(t); - setTrayRecording(false); - setErrorInfo({ - kind: 'mic', - message: t(micHintKey(detectPlatform())), - deniedByOs: true, - }); - setState('error'); - return; - } - try { - const stream = await navigator.mediaDevices.getUserMedia({ - audio: { echoCancellation: true, noiseSuppression: true, sampleRate: 16000 }, - }); - streamRef.current = stream; - chunksRef.current = []; - recordingFormatRef.current = { mimeType: 'audio/webm', extension: 'webm' }; - wsPendingRef.current = []; - wsHadFinalRef.current = false; - committedRef.current = []; - segmentErrorRef.current = null; - typedRef.current = ''; - // Live retract-retype is OPT-IN (visible backspace storms in the target - // app unnerved users): default sessions insert only committed finals via - // the paste path; the pref re-enables word-by-word typing. - liveTypingRef.current = localStorage.getItem(LS_LIVE_TYPING) === '1'; - pendingSepRef.current = false; - typeChainRef.current = Promise.resolve(); - pasteChainRef.current = Promise.resolve(); - waveRef.current.reset(); - if (fallbackTimerRef.current) { - clearTimeout(fallbackTimerRef.current); - fallbackTimerRef.current = null; - } - if (dismissTimerRef.current) { - clearTimeout(dismissTimerRef.current); - dismissTimerRef.current = null; + const startRecordingImpl = useCallback( + async (generation, startupSessionId = outputSessionIdRef.current) => { + // A newer native session can adopt this microphone graph while setup is + // still awaiting permissions/worklets. If the old attempt then fails, + // release only its original lease; finishing the adopted lease would + // make the replay below stale before it can start a fresh graph. + const finishAttemptOutputSession = () => { + const pending = pendingNativeStartRef.current; + const replacementOwnsCurrent = + pending?.sessionId === outputSessionIdRef.current && + startupSessionId && + startupSessionId !== pending.sessionId; + return finishOutputSession( + replacementOwnsCurrent ? startupSessionId : outputSessionIdRef.current, + ); + }; + // Pre-flight: when the OS itself reports the mic grant as DENIED, + // getUserMedia can only throw an opaque NotAllowedError — skip it and + // show the guided path (per-OS hint + Open Settings deep-link) instead. + // 'prompt'/'granted'/'unknown' proceed exactly as before (getUserMedia + // raises the OS prompt; micError.js stays the reactive fallback), and + // outside Tauri checkMicrophone() is always 'unknown' → unchanged. + if ((await checkMicrophone()) === 'denied') { + holdStartRef.current = null; + showMicDeniedGuide(t); + setTrayRecording(false); + setErrorInfo({ + kind: 'mic', + message: t(micHintKey(detectPlatform())), + deniedByOs: true, + }); + setState('error'); + await finishAttemptOutputSession(); + return; } - - // Read prefs at start time (avoids stale closures). AEC is opt-in; the - // sherpa live engine is selected when the persisted dictation model is a - // sherpa-onnx model — that path streams raw int16 PCM and emits live - // partials + a `final` per spoken utterance (committed on silence). - const aecOn = useAppStore.getState().aecEnabled === true; - const modelId = useAppStore.getState().dictationModelId; - const sherpaOn = isSherpaModel(modelId); - const supportedRecorder = - aecOn || sherpaOn - ? null - : startSupportedMediaRecorder(stream, { - onData: (e) => { - if (e.data.size === 0) return; - if (e.data.type) recordingFormatRef.current = audioFormatForMimeType(e.data.type); - chunksRef.current.push(e.data); - void e.data.arrayBuffer().then((buf) => { - const ws = wsRef.current; - if (ws && ws.readyState === WebSocket.OPEN) ws.send(buf); - else wsPendingRef.current.push(buf); - }); - }, - onStop: () => {}, - }); - const pcmFallback = !aecOn && !sherpaOn && supportedRecorder === null; - if (supportedRecorder) mediaRecorderRef.current = supportedRecorder.recorder; - aecModeRef.current = aecOn; - sherpaModeRef.current = sherpaOn; - pcmModeRef.current = pcmFallback; - // Raw-PCM transport is used whenever AEC or the sherpa live engine is on. - const pcmMode = aecOn || sherpaOn || pcmFallback; - - // Open WebSocket BEFORE starting capture. try { - // Scheme + host derive from the API base (window.location lies inside - // the Tauri webview). A remote bearer session is converted to a fresh, - // path-bound WebSocket ticket; neither the master nor session token is - // ever placed in this URL. - // • sherpa → ?model=&sr=16000 (raw int16 PCM, live partials) - // • AEC → ?aec=1&sr=16000 (tagged raw PCM, NLMS canceller) - // • both → ?model=&aec=1&sr=16000 - // • no recorder → ?pcm=1&sr=16000 (WebKitGTK fallback) - // • otherwise → /ws/transcribe (negotiated media container) - const params = []; - if (sherpaOn) params.push(`model=${encodeURIComponent(modelId)}`); - if (aecOn) params.push('aec=1'); - if (pcmFallback) params.push('pcm=1'); - if (pcmMode) params.push('sr=16000'); - const wsPath = params.length ? `/ws/transcribe?${params.join('&')}` : '/ws/transcribe'; - const endpoint = await authenticatedWsUrl(wsPath, { apiBase: API }); - const ws = new WebSocket(endpoint); - ws.binaryType = 'arraybuffer'; - const failRawPcmSession = () => { - if ( - wsHadFinalRef.current || - !(sherpaModeRef.current || aecModeRef.current || pcmModeRef.current) - ) { - return false; - } - wsHadFinalRef.current = true; - stopCaptureGraph(); - setTrayRecording(false); - setModelStatus(null); - setErrorInfo({ kind: 'server', message: '' }); - setState('error'); - return true; - }; - ws.onopen = () => { - for (const buf of wsPendingRef.current) { + const stream = await navigator.mediaDevices.getUserMedia({ + audio: { echoCancellation: true, noiseSuppression: true, sampleRate: 16000 }, + }); + streamRef.current = stream; + chunksRef.current = []; + recordingFormatRef.current = { mimeType: 'audio/webm', extension: 'webm' }; + wsPendingRef.current = []; + wsHadFinalRef.current = false; + committedRef.current = []; + segmentErrorRef.current = null; + deliveryKindRef.current = null; + deliveryCopySourceRef.current = null; + typedRef.current = ''; + typingFailedRef.current = false; + // Live retract-retype is OPT-IN (visible backspace storms in the target + // app unnerved users): default sessions insert only committed finals via + // the paste path; the pref re-enables word-by-word typing. + liveTypingRef.current = localStorage.getItem(LS_LIVE_TYPING) === '1'; + pendingSepRef.current = false; + typeChainRef.current = Promise.resolve(); + pasteChainRef.current = Promise.resolve(); + waveRef.current.reset(); + if (fallbackTimerRef.current) { + clearTimeout(fallbackTimerRef.current); + fallbackTimerRef.current = null; + } + if (dismissTimerRef.current) { + clearTimeout(dismissTimerRef.current); + dismissTimerRef.current = null; + } + + // Read prefs at start time (avoids stale closures). AEC is opt-in; the + // sherpa live engine is selected when the persisted dictation model is a + // sherpa-onnx model — that path streams raw int16 PCM and emits live + // partials + a `final` per spoken utterance (committed on silence). + const aecOn = useAppStore.getState().aecEnabled === true; + const modelId = useAppStore.getState().dictationModelId; + const sherpaOn = isSherpaModel(modelId); + const supportedRecorder = + aecOn || sherpaOn + ? null + : startSupportedMediaRecorder(stream, { + onData: (e) => { + if (e.data.size === 0) return; + if (e.data.type) recordingFormatRef.current = audioFormatForMimeType(e.data.type); + chunksRef.current.push(e.data); + void e.data.arrayBuffer().then((buf) => { + const ws = wsRef.current; + if (ws && ws.readyState === WebSocket.OPEN) ws.send(buf); + else wsPendingRef.current.push(buf); + }); + }, + onStop: () => {}, + }); + const pcmFallback = !aecOn && !sherpaOn && supportedRecorder === null; + if (supportedRecorder) mediaRecorderRef.current = supportedRecorder.recorder; + aecModeRef.current = aecOn; + sherpaModeRef.current = sherpaOn; + pcmModeRef.current = pcmFallback; + // Raw-PCM transport is used whenever AEC or the sherpa live engine is on. + const pcmMode = aecOn || sherpaOn || pcmFallback; + + // Open WebSocket BEFORE starting capture. + try { + // Scheme + host derive from the API base (window.location lies inside + // the Tauri webview). A remote bearer session is converted to a fresh, + // path-bound WebSocket ticket; neither the master nor session token is + // ever placed in this URL. + // • sherpa → ?model=&sr=16000 (raw int16 PCM, live partials) + // • AEC → ?aec=1&sr=16000 (tagged raw PCM, NLMS canceller) + // • both → ?model=&aec=1&sr=16000 + // • no recorder → ?pcm=1&sr=16000 (WebKitGTK fallback) + // • otherwise → /ws/transcribe (negotiated media container) + const params = []; + if (sherpaOn) params.push(`model=${encodeURIComponent(modelId)}`); + if (aecOn) params.push('aec=1'); + if (pcmFallback) params.push('pcm=1'); + if (pcmMode) params.push('sr=16000'); + const wsPath = params.length ? `/ws/transcribe?${params.join('&')}` : '/ws/transcribe'; + const endpoint = await authenticatedWsUrl(wsPath, { apiBase: API }); + const ws = new WebSocket(endpoint); + ws.binaryType = 'arraybuffer'; + const failRawPcmSession = () => { + if ( + wsHadFinalRef.current || + !(sherpaModeRef.current || aecModeRef.current || pcmModeRef.current) + ) { + return false; + } + wsHadFinalRef.current = true; + stopCaptureGraph(); + setTrayRecording(false); + setModelStatus(null); + setErrorInfo({ kind: 'server', message: '' }); + setState('error'); + void finishAttemptOutputSession(); + return true; + }; + ws.onopen = () => { + for (const buf of wsPendingRef.current) { + try { + ws.send(buf); + } catch (err) { + // Socket died mid-flush — onclose/onerror handles recovery. + console.warn('ws flush failed:', err); + break; + } + } + wsPendingRef.current = []; + }; + ws.onmessage = async (evt) => { + if (wsRef.current !== ws) return; + let msg; try { - ws.send(buf); + msg = JSON.parse(evt.data); } catch (err) { - // Socket died mid-flush — onclose/onerror handles recovery. - console.warn('ws flush failed:', err); - break; + console.warn('unparseable /ws/transcribe frame:', err); + return; } - } - wsPendingRef.current = []; - }; - ws.onmessage = (evt) => { - let msg; - try { - msg = JSON.parse(evt.data); - } catch (err) { - console.warn('unparseable /ws/transcribe frame:', err); - return; - } - if (msg.type === 'status') { - // Model lifecycle truthfulness: while the backend fetches/loads - // the ASR model it streams {stage:"downloading",progress} / - // {stage:"loading"} / {stage:"ready"} so the pill can say what is - // actually happening instead of a generic "Listening…". - setModelStatus( - msg.stage === 'ready' - ? null - : { - stage: msg.stage, - progress: typeof msg.progress === 'number' ? msg.progress : null, - }, - ); - } else if (msg.type === 'partial') { - // Live interim text — show the running transcript so far plus the - // in-flight partial, so the pill reads as continuous speech. - const committed = committedRef.current.join(' '); - const live = [committed, msg.text || ''].filter(Boolean).join(' '); - setPartialText(live); - // …and (opt-in) type the revised in-flight utterance into the - // focused field word-by-word. The diff handles recognizer - // self-corrections via backspaces; committed utterances are - // untouched. Only sherpa live partials drive typing — the legacy - // WebM path has no partials — and liveType no-ops unless the - // LS_LIVE_TYPING pref opted in. - if (sherpaModeRef.current) liveType(msg.text || ''); - } else if (msg.type === 'final') { - if (sherpaModeRef.current) { - // Two sherpa `final` shapes: - // • STREAMING models emit a `final` per spoken utterance (on - // each silence endpoint) THEN a session-summary `final` on - // EOF whose text is the join of every utterance. - // • OFFLINE models (incl. the default Parakeet v3) emit live - // partials then exactly ONE `final` (the whole transcript) - // on EOF. - // Rule: a `final` whose text equals what we've already committed - // is the authoritative EOF SUMMARY → finalise without re-pasting - // (its pieces already landed live). Any other `final` is a NEW - // utterance → paste it live and append. The single offline final - // is "new" (nothing committed yet) so it pastes once; the socket - // close then finalises from the committed text. - // Classify on the RAW text: the EOF summary's `text` is exactly - // the join of the committed utterances, but its optional LLM - // `refined_text` is not — classifying on the refined string - // would misread the summary as a new utterance and re-paste the - // whole transcript (double insert). Delivery still prefers the - // refined text where one applies (the single offline final). - const segText = msg.refined_text || msg.text || ''; - const cls = classifySherpaFinal(msg.text || '', committedRef.current); - if (cls === 'summary' || cls === 'terminator') { - // Authoritative EOF (summary text already pasted live, or an - // empty no-speech terminator) → finalise so the pill resolves. + if (msg.type === 'status') { + // Model lifecycle truthfulness: while the backend fetches/loads + // the ASR model it streams {stage:"downloading",progress} / + // {stage:"loading"} / {stage:"ready"} so the pill can say what is + // actually happening instead of a generic "Listening…". + setModelStatus( + msg.stage === 'ready' + ? null + : { + stage: msg.stage, + progress: typeof msg.progress === 'number' ? msg.progress : null, + }, + ); + } else if (msg.type === 'partial') { + // Live interim text — show the running transcript so far plus the + // in-flight partial, so the pill reads as continuous speech. + const committed = committedRef.current.join(' '); + const live = [committed, msg.text || ''].filter(Boolean).join(' '); + setPartialText(live); + // …and (opt-in) type the revised in-flight utterance into the + // focused field word-by-word. The diff handles recognizer + // self-corrections via backspaces; committed utterances are + // untouched. Only sherpa live partials drive typing — the legacy + // WebM path has no partials — and liveType no-ops unless the + // LS_LIVE_TYPING pref opted in. + if (sherpaModeRef.current) liveType(msg.text || ''); + } else if (msg.type === 'final') { + if (sherpaModeRef.current) { + if (msg.model_silent && !(msg.text || '').trim()) { + // Speech reached the selected model, but it produced no text. + // This is a broken-model result, not a successful quiet session. + wsHadFinalRef.current = true; + if (fallbackTimerRef.current) { + clearTimeout(fallbackTimerRef.current); + fallbackTimerRef.current = null; + } + stopCaptureGraph(); + setTrayRecording(false); + setModelStatus(null); + setTranscript(''); + setErrorInfo({ kind: 'transcription', message: '' }); + setState('error'); + await finishAttemptOutputSession(); + ws.close(); + return; + } + // Never infer the frame kind from text equality: two utterances + // may be identical, while EOF may contain an uncommitted tail. + const segText = msg.refined_text || msg.text || ''; + const cls = classifySherpaFinal(msg); + if (cls === 'summary' || cls === 'terminator') { + if (cls === 'summary') { + const tail = sherpaSummaryTail(msg.text || '', committedRef.current); + if (tail) { + const deliveryText = committedRef.current.length + ? ` ${tail}` + : msg.refined_text || tail; + if (!typingFailedRef.current) queueSegmentPaste(deliveryText); + if (!committedRef.current.length && msg.text) addTranscription(msg); + } + } + wsHadFinalRef.current = true; + if (fallbackTimerRef.current) { + clearTimeout(fallbackTimerRef.current); + fallbackTimerRef.current = null; + } + finalizeSession(msg, outputSessionIdRef.current, generation); + ws.close(); + } else if (cls === 'utterance') { + // A per-utterance commit. Reconcile the focused field to the + // recognizer's AUTHORITATIVE final for this utterance (it can + // differ from the last partial — e.g. final punctuation / a + // late self-correction), then FREEZE it: reset typedRef so the + // next utterance's partials diff from empty. We never backspace + // across this boundary. In the default (live typing off) the + // committed final is pasted instead — never both (no + // double-insert) — and a failed paste is recorded so the + // session resolves truthfully. + const needsSeparator = committedRef.current.length > 0; + committedRef.current.push(segText); + setPartialText(committedRef.current.join(' ')); + if (msg.text) addTranscription(msg); + if (liveTypingRef.current) { + const commitGeneration = captureGenerationRef.current; + const commitSessionId = outputSessionIdRef.current; + liveType(segText); + typeChainRef.current = typeChainRef.current.then(() => { + if ( + commitGeneration !== captureGenerationRef.current || + (commitSessionId && outputSessionIdRef.current !== commitSessionId) + ) { + return; + } + const commitAfterSafeDowngrade = + !liveTypingRef.current && + !typingFailedRef.current && + typedRef.current === ''; + typedRef.current = ''; + // Seed the next utterance's typed-state with a separating + // space (matching the ' '.join used by the pill/history) so + // its first delta types " word" — words never run together, + // and there is no trailing space after the LAST utterance. + pendingSepRef.current = true; + if (commitAfterSafeDowngrade) { + queueSegmentPaste(needsSeparator ? ` ${segText}` : segText); + } + }); + } else if (typingFailedRef.current) { + // A failed native delta may have partially landed even when + // typedRef has no confirmed prefix. Never risk duplicating it. + typedRef.current = ''; + } else { + queueSegmentPaste(needsSeparator ? ` ${segText}` : segText); + } + } + } else { + // Legacy single-final path (Whisper/WebM) — unchanged. wsHadFinalRef.current = true; if (fallbackTimerRef.current) { clearTimeout(fallbackTimerRef.current); fallbackTimerRef.current = null; } - finalizeSession(msg); + applyResult(msg, outputSessionIdRef.current, generation); ws.close(); - } else if (cls === 'utterance') { - // A per-utterance commit. Reconcile the focused field to the - // recognizer's AUTHORITATIVE final for this utterance (it can - // differ from the last partial — e.g. final punctuation / a - // late self-correction), then FREEZE it: reset typedRef so the - // next utterance's partials diff from empty. We never backspace - // across this boundary. In the default (live typing off) the - // committed final is pasted instead — never both (no - // double-insert) — and a failed paste is recorded so the - // session resolves truthfully. - committedRef.current.push(segText); - setPartialText(committedRef.current.join(' ')); - if (msg.text) addTranscription(msg); - if (liveTypingRef.current) { - liveType(segText); - typeChainRef.current = typeChainRef.current.then(() => { - typedRef.current = ''; - // Seed the next utterance's typed-state with a separating - // space (matching the ' '.join used by the pill/history) so - // its first delta types " word" — words never run together, - // and there is no trailing space after the LAST utterance. - pendingSepRef.current = true; - }); - } else { - pasteChainRef.current = pasteChainRef.current - .then(() => pasteSegment(segText)) - .then((res) => { - if (!res.ok && !segmentErrorRef.current) { - segmentErrorRef.current = res.error; - } - }); - } } - } else { - // Legacy single-final path (Whisper/WebM) — unchanged. - wsHadFinalRef.current = true; + } else if (msg.type === 'error') { if (fallbackTimerRef.current) { clearTimeout(fallbackTimerRef.current); fallbackTimerRef.current = null; } - applyResult(msg); ws.close(); + wsRef.current = null; + if (asrMissingPayload(msg)) { + // Typed preflight: no ASR model installed. The POST fallback + // would hit the same 409, so don't re-send — render the + // download CTA and resolve the pill into its error state. + wsHadFinalRef.current = true; + stopCaptureGraph(); + setTrayRecording(false); + setModelStatus(null); + toastAsrModelMissing(asrMissingPayload(msg)); + setErrorInfo({ kind: 'transcription', message: t('asr_missing.message') }); + setState('error'); + void finishAttemptOutputSession(); + } else if (sherpaModeRef.current || aecModeRef.current || pcmModeRef.current) { + // Raw-PCM paths have no WebM blob to re-POST — surface the + // backend's error instead of leaving the pill wedged in + // "Transcribing…" forever. + wsHadFinalRef.current = true; + stopCaptureGraph(); + setTrayRecording(false); + setModelStatus(null); + setErrorInfo({ kind: msg.kind || 'server', message: msg.message || '' }); + setState('error'); + void finishAttemptOutputSession(); + } else if (!wsHadFinalRef.current) { + sendForTranscription(outputSessionIdRef.current, generation); + } } - } else if (msg.type === 'error') { - if (fallbackTimerRef.current) { - clearTimeout(fallbackTimerRef.current); - fallbackTimerRef.current = null; - } - ws.close(); + }; + ws.onerror = () => { + if (wsRef.current !== ws) return; wsRef.current = null; - if (asrMissingPayload(msg)) { - // Typed preflight: no ASR model installed. The POST fallback - // would hit the same 409, so don't re-send — render the - // download CTA and resolve the pill into its error state. - wsHadFinalRef.current = true; - stopCaptureGraph(); - setTrayRecording(false); - setModelStatus(null); - toastAsrModelMissing(asrMissingPayload(msg)); - setErrorInfo({ kind: 'transcription', message: t('asr_missing.message') }); - setState('error'); - } else if (sherpaModeRef.current || aecModeRef.current || pcmModeRef.current) { - // Raw-PCM paths have no WebM blob to re-POST — surface the - // backend's error instead of leaving the pill wedged in - // "Transcribing…" forever. - wsHadFinalRef.current = true; - stopCaptureGraph(); - setTrayRecording(false); - setModelStatus(null); - setErrorInfo({ kind: msg.kind || 'server', message: msg.message || '' }); - setState('error'); - } else if (!wsHadFinalRef.current) { - sendForTranscription(); + failRawPcmSession(); + }; + ws.onclose = () => { + // A terminal path can await native session release while a new + // candidate is activated and starts its own socket. The old close + // must never clear or finalise that newer recording. + if (wsRef.current !== ws) return; + wsRef.current = null; + if (sherpaModeRef.current) { + // Sherpa: nothing to POST (no WebM blob). If the socket dropped + // before the EOF summary but we committed utterances live, close out + // the session from what we have so the pill resolves. + if (!wsHadFinalRef.current && committedRef.current.length) { + wsHadFinalRef.current = true; + finalizeSession( + { text: committedRef.current.join(' '), engine: 'sherpa-onnx-asr' }, + outputSessionIdRef.current, + generation, + ); + } else if (!wsHadFinalRef.current) { + wsHadFinalRef.current = true; + stopCaptureGraph(); + setTrayRecording(false); + setErrorInfo({ kind: 'server', message: '' }); + setState('error'); + void finishAttemptOutputSession(); + } + return; } - } - }; - ws.onerror = () => { - wsRef.current = null; - failRawPcmSession(); - }; - ws.onclose = () => { - wsRef.current = null; - if (sherpaModeRef.current) { - // Sherpa: nothing to POST (no WebM blob). If the socket dropped - // before the EOF summary but we committed utterances live, close out - // the session from what we have so the pill resolves. - if (!wsHadFinalRef.current && committedRef.current.length) { - wsHadFinalRef.current = true; - finalizeSession({ text: committedRef.current.join(' '), engine: 'sherpa-onnx-asr' }); + if (failRawPcmSession()) return; + if ( + !wsHadFinalRef.current && + mediaRecorderRef.current && + mediaRecorderRef.current.state === 'inactive' + ) { + if (fallbackTimerRef.current) { + clearTimeout(fallbackTimerRef.current); + fallbackTimerRef.current = null; + } + sendForTranscription(outputSessionIdRef.current, generation); } + }; + wsRef.current = ws; + } catch { + wsRef.current = null; + if (pcmMode) { + // Raw-PCM has no POST fallback — a socket that can't even be + // constructed is fatal to the session, so say so instead of + // recording into the void. + stream.getTracks().forEach((tr) => tr.stop()); + streamRef.current = null; + setErrorInfo({ kind: 'server', message: '' }); + setState('error'); + await finishAttemptOutputSession(); return; } - if (failRawPcmSession()) return; - if ( - !wsHadFinalRef.current && - mediaRecorderRef.current && - mediaRecorderRef.current.state === 'inactive' - ) { - if (fallbackTimerRef.current) { - clearTimeout(fallbackTimerRef.current); - fallbackTimerRef.current = null; - } - sendForTranscription(); - } - }; - wsRef.current = ws; - } catch { - wsRef.current = null; - if (pcmMode) { - // Raw-PCM has no POST fallback — a socket that can't even be - // constructed is fatal to the session, so say so instead of - // recording into the void. - stream.getTracks().forEach((tr) => tr.stop()); - streamRef.current = null; - setErrorInfo({ kind: 'server', message: '' }); - setState('error'); - return; + // Legacy path continues below: the recorder still buffers chunks and + // the POST /transcribe fallback delivers the result on stop. + console.warn('ws open failed — will fall back to POST /transcribe'); } - // Legacy path continues below: the recorder still buffers chunks and - // the POST /transcribe fallback delivers the result on stop. - console.warn('ws open failed — will fall back to POST /transcribe'); - } - if (pcmMode) { - // Raw-PCM path: stream int16 mono frames at 16 kHz via the AudioWorklet - // (no MediaRecorder, no WebM POST fallback — the WS is the only channel). - // • sherpa live engine → UNTAGGED int16 frames (the non-AEC sherpa - // handler reads plain PCM); the far-end bus is NOT subscribed. - // • AEC on → frames are 1-byte tagged (0x00 mic / 0x01 far-end) and the - // audio player's output is subscribed as the echo reference. - // Every mic frame also feeds the waveform ring buffer — the pill's - // bars are computed client-side from the SAME worklet frames (no - // second audio pipeline). - const [{ startMicCapture }, { frameFromFloat, floatToInt16, AEC_NEAR, AEC_FAR }] = - await Promise.all([import('../utils/aec/micCapture'), import('../utils/aec/pcm')]); - const sendBuf = (buf) => { - const ws = wsRef.current; - if (ws && ws.readyState === WebSocket.OPEN) { - try { - ws.send(buf); - } catch (err) { - // Socket is going down — onclose finalises/recovers the session. - console.warn('ws send failed:', err); + if (pcmMode) { + // Raw-PCM path: stream int16 mono frames at 16 kHz via the AudioWorklet + // (no MediaRecorder, no WebM POST fallback — the WS is the only channel). + // • sherpa live engine → UNTAGGED int16 frames (the non-AEC sherpa + // handler reads plain PCM); the far-end bus is NOT subscribed. + // • AEC on → frames are 1-byte tagged (0x00 mic / 0x01 far-end) and the + // audio player's output is subscribed as the echo reference. + // Every mic frame also feeds the waveform ring buffer — the pill's + // bars are computed client-side from the SAME worklet frames (no + // second audio pipeline). + const [{ startMicCapture }, { frameFromFloat, floatToInt16, AEC_NEAR, AEC_FAR }] = + await Promise.all([import('../utils/aec/micCapture'), import('../utils/aec/pcm')]); + const sendBuf = (buf) => { + const ws = wsRef.current; + if (ws && ws.readyState === WebSocket.OPEN) { + try { + ws.send(buf); + } catch (err) { + // Socket is going down — onclose finalises/recovers the session. + console.warn('ws send failed:', err); + } + } else { + wsPendingRef.current.push(buf); } + }; + if (aecOn) { + // Tagged frames + far-end reference (echo cancellation). Works for the + // sherpa+AEC combo too — the backend demuxes the tag before the + // sherpa handler sees the cleaned near-end PCM. + const { subscribeFarEnd } = await import('../utils/aec/farEndBus'); + const sendTagged = (float32, kind) => sendBuf(frameFromFloat(float32, kind)); + aecStopRef.current = await startMicCapture( + stream, + (f) => { + waveRef.current.push(f); + sendTagged(f, AEC_NEAR); + }, + { sampleRate: 16000 }, + ); + farEndUnsubRef.current = subscribeFarEnd((f) => sendTagged(f, AEC_FAR)); } else { - wsPendingRef.current.push(buf); + // Untagged int16 frames for the plain sherpa live path. Send the + // Int16Array's underlying buffer verbatim (little-endian on every + // target platform = numpy's native int16 read on the server). + aecStopRef.current = await startMicCapture( + stream, + (f) => { + waveRef.current.push(f); + const i16 = floatToInt16(f); + sendBuf(i16.buffer.slice(i16.byteOffset, i16.byteOffset + i16.byteLength)); + }, + { sampleRate: 16000 }, + ); } - }; - if (aecOn) { - // Tagged frames + far-end reference (echo cancellation). Works for the - // sherpa+AEC combo too — the backend demuxes the tag before the - // sherpa handler sees the cleaned near-end PCM. - const { subscribeFarEnd } = await import('../utils/aec/farEndBus'); - const sendTagged = (float32, kind) => sendBuf(frameFromFloat(float32, kind)); - aecStopRef.current = await startMicCapture( - stream, - (f) => { - waveRef.current.push(f); - sendTagged(f, AEC_NEAR); - }, - { sampleRate: 16000 }, - ); - farEndUnsubRef.current = subscribeFarEnd((f) => sendTagged(f, AEC_FAR)); + mediaRecorderRef.current = null; } else { - // Untagged int16 frames for the plain sherpa live path. Send the - // Int16Array's underlying buffer verbatim (little-endian on every - // target platform = numpy's native int16 read on the server). - aecStopRef.current = await startMicCapture( - stream, - (f) => { - waveRef.current.push(f); - const i16 = floatToInt16(f); - sendBuf(i16.buffer.slice(i16.byteOffset, i16.byteOffset + i16.byteLength)); - }, - { sampleRate: 16000 }, - ); + const { recorder, mimeType, extension } = supportedRecorder; + recordingFormatRef.current = { mimeType, extension }; + mediaRecorderRef.current = recorder; + } + // The session may already have RESOLVED while the mic graph was being + // set up (the awaits above): a connect-time WS error frame (e.g. the + // typed asr_model_missing preflight) or an Esc-cancel sets + // wsHadFinalRef and renders the truthful terminal state. Entering + // 'recording' now would clobber that state and — with the socket gone — + // strand the next Stop on "Transcribing…" forever. Release the capture + // inputs and leave the pill alone. + if (wsHadFinalRef.current) { + stopCaptureGraph(); + return; + } + startTimeRef.current = Date.now(); + setTrayRecording(true); + setWaveOn(pcmMode); + setBars(Array.from({ length: WAVE_BARS }, () => 0)); + setState('recording'); + setTranscript(''); + setPartialText(''); + setModelStatus(null); + setErrorInfo(null); + setDoneKind(null); + setDuration(0); + stateRef.current = 'recording'; + if (holdStartRef.current === 'released') { + holdStartRef.current = null; + stopRecordingRef.current?.(); + } else { + holdStartRef.current = null; + } + } catch (err) { + holdStartRef.current = null; + // Same guard as the success path above (#1175 review): the session may + // already have RESOLVED while setup was failing — a connect-time WS + // error frame (e.g. the typed asr_model_missing preflight) or an + // Esc-cancel set wsHadFinalRef and rendered the truthful terminal + // state. A late mic error must not clobber it. + if (wsHadFinalRef.current) { + stopCaptureGraph(); + return; } - mediaRecorderRef.current = null; - } else { - const { recorder, mimeType, extension } = supportedRecorder; - recordingFormatRef.current = { mimeType, extension }; - mediaRecorderRef.current = recorder; - } - // The session may already have RESOLVED while the mic graph was being - // set up (the awaits above): a connect-time WS error frame (e.g. the - // typed asr_model_missing preflight) or an Esc-cancel sets - // wsHadFinalRef and renders the truthful terminal state. Entering - // 'recording' now would clobber that state and — with the socket gone — - // strand the next Stop on "Transcribing…" forever. Release the capture - // inputs and leave the pill alone. - if (wsHadFinalRef.current) { stopCaptureGraph(); - return; + // Distinguish "permission denied" (→ per-OS settings hint) from + // "no device" / "device busy" / anything else (#323). + toast.error(micErrorMessage(t, err), { duration: 6000 }); + setTrayRecording(false); + setErrorInfo({ + kind: 'mic', + message: String(err?.message || err), + // Permission-denied errors (describeMicError sets a hintKey only for + // those) get the pill's Open-Settings action inside Tauri. + deniedByOs: !!describeMicError(err).hintKey, + }); + setState('error'); + await finishAttemptOutputSession(); } - startTimeRef.current = Date.now(); - setTrayRecording(true); - setWaveOn(pcmMode); - setBars(Array.from({ length: WAVE_BARS }, () => 0)); - setState('recording'); - setTranscript(''); - setPartialText(''); - setModelStatus(null); - setErrorInfo(null); - setDoneKind(null); - setDuration(0); - stateRef.current = 'recording'; - if (holdStartRef.current === 'released') { - holdStartRef.current = null; - stopRecordingRef.current?.(); - } else { - holdStartRef.current = null; + }, + [ + applyResult, + finalizeSession, + finishOutputSession, + liveType, + queueSegmentPaste, + stopCaptureGraph, + t, + ], + ); + + const startRecording = useCallback( + async (sessionId = null) => { + if (startInFlightRef.current) { + if (sessionId && sessionId !== outputSessionIdRef.current) { + // Duplicate native events normally carry one id. Defensively adopt a + // newer non-empty lease without launching a second microphone graph. + outputSessionIdRef.current = sessionId; + } + return; } - } catch (err) { - holdStartRef.current = null; - // Same guard as the success path above (#1175 review): the session may - // already have RESOLVED while setup was failing — a connect-time WS - // error frame (e.g. the typed asr_model_missing preflight) or an - // Esc-cancel set wsHadFinalRef and rendered the truthful terminal - // state. A late mic error must not clobber it. - if (wsHadFinalRef.current) { - stopCaptureGraph(); + if (inTauri() && !sessionId) { + hideWidgetWindow(); return; } - stopCaptureGraph(); - // Distinguish "permission denied" (→ per-OS settings hint) from - // "no device" / "device busy" / anything else (#323). - toast.error(micErrorMessage(t, err), { duration: 6000 }); - setTrayRecording(false); - setErrorInfo({ - kind: 'mic', - message: String(err?.message || err), - // Permission-denied errors (describeMicError sets a hintKey only for - // those) get the pill's Open-Settings action inside Tauri. - deniedByOs: !!describeMicError(err).hintKey, - }); - setState('error'); - } - }, [applyResult, finalizeSession, liveType, stopCaptureGraph, t]); + startInFlightRef.current = true; + if (sessionId) outputSessionIdRef.current = sessionId; + const generation = ++captureGenerationRef.current; + if (dismissTimerRef.current) { + clearTimeout(dismissTimerRef.current); + dismissTimerRef.current = null; + } + try { + await startRecordingImpl(generation, sessionId); + } finally { + startInFlightRef.current = false; + const pending = pendingNativeStartRef.current; + if (pending) { + if ( + pending.sequence !== nativeStartSequenceRef.current || + outputSessionIdRef.current !== pending.sessionId + ) { + pendingNativeStartRef.current = null; + } else { + const current = stateRef.current; + pendingNativeStartRef.current = null; + if (current !== 'recording' && current !== 'transcribing') { + await startRecordingRef.current?.(pending.trackHold, pending.sessionId); + } + } + } + } + }, + [startRecordingImpl], + ); const stopRecording = useCallback(() => { + const generation = captureGenerationRef.current; + const sessionId = outputSessionIdRef.current; stopCaptureGraph(); // Signal EOF to WebSocket const ws = wsRef.current; @@ -1310,7 +1792,7 @@ export default function CaptureWidget({ onDismiss }) { if (!wsHadFinalRef.current) { wsRef.current?.close(); wsRef.current = null; - sendForTranscription(); + sendForTranscription(sessionId, generation); } }, ms); } @@ -1318,53 +1800,63 @@ export default function CaptureWidget({ onDismiss }) { setState('transcribing'); }, [stopCaptureGraph]); - const sendForTranscription = useCallback(async () => { - if (wsHadFinalRef.current) return; - // No encoded blob exists on a raw-PCM path — the WS is the only result - // channel there. - if (aecModeRef.current || sherpaModeRef.current || pcmModeRef.current) return; - - const { mimeType, extension } = recordingFormatRef.current; - const blob = new Blob(chunksRef.current, { type: mimeType }); - const formData = new FormData(); - formData.append('audio', blob, `capture.${extension}`); - formData.append('mode', captureMode); + const sendForTranscription = useCallback( + async (sessionId = outputSessionIdRef.current, generation = captureGenerationRef.current) => { + const isCurrent = () => + generation === captureGenerationRef.current && + (!sessionId || outputSessionIdRef.current === sessionId); + if (!isCurrent() || wsHadFinalRef.current) return; + // No encoded blob exists on a raw-PCM path — the WS is the only result + // channel there. + if (aecModeRef.current || sherpaModeRef.current || pcmModeRef.current) return; + + const { mimeType, extension } = recordingFormatRef.current; + const blob = new Blob(chunksRef.current, { type: mimeType }); + const formData = new FormData(); + formData.append('audio', blob, `capture.${extension}`); + formData.append('mode', captureMode); - try { - // apiFetch attaches the PIN / remote API key headers (Wave 2.3) - // and throws on non-2xx with the server's detail message. - const res = await apiFetch('/transcribe', { - method: 'POST', - body: formData, - }); - const data = await res.json(); - if (wsHadFinalRef.current) return; - await applyResult(data); - } catch (err) { - if (wsHadFinalRef.current) return; - const missing = asrMissingPayload(err); - if (missing) { - // Typed 409: no ASR model installed → download CTA, not a dead end. - toastAsrModelMissing(missing); - setErrorInfo({ kind: 'transcription', message: t('asr_missing.message') }); + try { + // apiFetch attaches the PIN / remote API key headers (Wave 2.3) + // and throws on non-2xx with the server's detail message. + const res = await apiFetch('/transcribe', { + method: 'POST', + body: formData, + }); + if (!isCurrent()) return; + const data = await res.json(); + if (!isCurrent() || wsHadFinalRef.current) return; + await applyResult(data, sessionId, generation); + } catch (err) { + if (!isCurrent() || wsHadFinalRef.current) return; + const missing = asrMissingPayload(err); + if (missing) { + // Typed 409: no ASR model installed → download CTA, not a dead end. + toastAsrModelMissing(missing); + setErrorInfo({ kind: 'transcription', message: t('asr_missing.message') }); + setState('error'); + setTranscript(''); + await finishOutputSession(sessionId); + if (captureGenerationRef.current !== generation) return; + return; + } + toast.error(t('capture.transcription_failed', { message: err.message })); + setErrorInfo({ kind: 'transcription', message: err.message }); setState('error'); setTranscript(''); - return; + await finishOutputSession(sessionId); } - toast.error(t('capture.transcription_failed', { message: err.message })); - setErrorInfo({ kind: 'transcription', message: err.message }); - setState('error'); - setTranscript(''); - } - }, [captureMode, applyResult, t]); + }, + [captureMode, applyResult, finishOutputSession, t], + ); // Keep the trigger refs pointing at the current callbacks. No dep array: it // must run after every render so the once-attached tray listener above never // calls into a stale closure. useEffect(() => { - startRecordingRef.current = (trackHold = false) => { + startRecordingRef.current = (trackHold = false, sessionId = null) => { if (trackHold && holdStartRef.current !== 'released') holdStartRef.current = 'starting'; - return startRecording(); + return startRecording(sessionId); }; stopRecordingRef.current = stopRecording; }); @@ -1508,7 +2000,12 @@ export default function CaptureWidget({ onDismiss }) { label = partialText || t('capture.transcribing_label'); } else if (state === 'done' && transcript) { emoji = '✅'; - label = doneKind === 'copied' ? t('capture.copied') : t('capture.pasted'); + label = + doneKind === 'copied' + ? t('capture.copied') + : doneKind === 'inserted' + ? t('capture.inserted') + : t('capture.pasted'); } else if (state === 'done' && !transcript) { emoji = '⚠️'; label = t('capture.no_speech'); diff --git a/frontend/src/components/CaptureWidget.test.jsx b/frontend/src/components/CaptureWidget.test.jsx index 33004c395..1c7a19fa2 100644 --- a/frontend/src/components/CaptureWidget.test.jsx +++ b/frontend/src/components/CaptureWidget.test.jsx @@ -2,7 +2,7 @@ * CaptureWidget pill behaviour — mocked WS + Tauri invoke. * * Covers the truthfulness rebuild: model status frames render real - * download/load progress, "Pasted" only appears after simulate_paste resolves + * download/load progress, "Inserted" only appears after native delivery resolves * Ok, an "a11y:"-prefixed paste failure renders the actionable Accessibility * error, Esc aborts without pasting, live retract-retype is opt-in (default * sessions never call simulate_type), the missing-Accessibility setup state @@ -14,6 +14,11 @@ import { render, screen, waitFor, fireEvent, act } from '@testing-library/react' import { I18nextProvider } from 'react-i18next'; import i18n from '../i18n'; +const captureLocales = import.meta.glob('../i18n/locales/*.json', { + eager: true, + import: 'default', +}); + // ── Hoisted mock state (vi.mock factories may only reference vi.hoisted vars) ── const mocks = vi.hoisted(() => { const state = { @@ -26,10 +31,20 @@ const mocks = vi.hoisted(() => { const holder = { // Per-test knobs for the Tauri invoke mock. a11y: true, - paste: async () => undefined, + paste: async () => 'inserted', + copy: async () => 'copied', + type: async () => 'inserted', + activate: async () => undefined, + reject: async () => undefined, + finish: async () => undefined, calls: [], + handlers: {}, // Captured micCapture frame callback (the worklet feed). onFrame: null, + startMic: async (_stream, onFrame) => { + holder.onFrame = onFrame; + return async () => {}; + }, // Stable spy for getCurrentWindow().hide — a fresh vi.fn() per call would // make the native hide unassertable. hideWindow: vi.fn(async () => {}), @@ -38,10 +53,17 @@ const mocks = vi.hoisted(() => { state, holder, authenticatedWsUrl: vi.fn(async (path) => `ws://test${path}&ws_ticket=one-use`), + apiFetch: vi.fn(async () => ({ json: async () => ({}) })), + copyText: vi.fn(async () => {}), invoke: async (cmd, args) => { holder.calls.push([cmd, args]); if (cmd === 'check_accessibility') return holder.a11y; - if (cmd === 'simulate_paste') return holder.paste(); + if (cmd === 'simulate_paste') return holder.paste(cmd, args); + if (cmd === 'copy_dictation_output_session') return holder.copy(cmd, args); + if (cmd === 'simulate_type') return holder.type(cmd, args); + if (cmd === 'activate_dictation_output_session') return holder.activate(cmd, args); + if (cmd === 'reject_dictation_output_session') return holder.reject(cmd, args); + if (cmd === 'finish_dictation_output_session') return holder.finish(cmd, args); return undefined; }, }; @@ -53,25 +75,25 @@ vi.mock('../store', () => ({ vi.mock('../api/client', () => ({ API: 'http://test', wsUrl: (p) => `ws://test${p}`, - apiFetch: vi.fn(async () => ({ json: async () => ({}) })), + apiFetch: mocks.apiFetch, })); vi.mock('../api/authSession', () => ({ authenticatedWsUrl: mocks.authenticatedWsUrl })); vi.mock('../pages/Transcriptions', () => ({ addTranscription: vi.fn() })); -vi.mock('../utils/copyText', () => ({ copyText: vi.fn(async () => {}) })); +vi.mock('../utils/copyText', () => ({ copyText: mocks.copyText })); vi.mock('react-hot-toast', () => ({ toast: { error: vi.fn() } })); vi.mock('@tauri-apps/api/core', () => ({ invoke: mocks.invoke })); vi.mock('@tauri-apps/api/event', () => ({ emit: vi.fn(async () => {}), - listen: vi.fn(async () => () => {}), + listen: vi.fn(async (name, handler) => { + mocks.holder.handlers[name] = handler; + return () => delete mocks.holder.handlers[name]; + }), })); vi.mock('@tauri-apps/api/window', () => ({ getCurrentWindow: () => ({ hide: mocks.holder.hideWindow }), })); vi.mock('../utils/aec/micCapture', () => ({ - startMicCapture: async (stream, onFrame) => { - mocks.holder.onFrame = onFrame; - return async () => {}; - }, + startMicCapture: (...args) => mocks.holder.startMic(...args), })); import CaptureWidget from './CaptureWidget'; @@ -105,6 +127,9 @@ class FakeWebSocket { msg(obj) { this.onmessage?.({ data: JSON.stringify(obj) }); } + msgAsync(obj) { + return this.onmessage?.({ data: JSON.stringify(obj) }); + } } class FakeMediaRecorder { @@ -126,9 +151,16 @@ function withI18n(node) { return {node}; } -// Start a session via the in-page shortcut and wait for the live socket. +// Start through the native event that carries Rust's captured output target. async function startSession() { - fireEvent.keyDown(window, { code: 'Space', ctrlKey: true, shiftKey: true }); + return startNativeSession(); +} + +async function startNativeSession(sessionId = 'capture-session-1') { + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ payload: { sessionId } }); + }); await waitFor(() => expect(FakeWebSocket.instances.length).toBe(1)); await screen.findByText(/Listening/); return FakeWebSocket.instances[0]; @@ -138,16 +170,32 @@ describe('CaptureWidget', () => { beforeEach(() => { window.__TAURI_INTERNALS__ = {}; mocks.holder.a11y = true; - mocks.holder.paste = async () => undefined; + mocks.holder.paste = async () => 'inserted'; + mocks.holder.copy = async () => 'copied'; + mocks.holder.type = async () => 'inserted'; + mocks.holder.activate = async () => undefined; + mocks.holder.reject = async () => undefined; + mocks.holder.finish = async () => undefined; mocks.holder.calls = []; + mocks.holder.handlers = {}; mocks.holder.onFrame = null; + mocks.holder.startMic = async (_stream, onFrame) => { + mocks.holder.onFrame = onFrame; + return async () => {}; + }; mocks.holder.hideWindow.mockClear(); + mocks.copyText.mockReset(); + mocks.copyText.mockResolvedValue(undefined); + mocks.apiFetch.mockReset(); + mocks.apiFetch.mockImplementation(async () => ({ json: async () => ({}) })); mocks.authenticatedWsUrl.mockClear(); mocks.authenticatedWsUrl.mockImplementation( async (path) => `ws://test${path}${path.includes('?') ? '&' : '?'}ws_ticket=one-use`, ); mocks.state.dictationMode = 'toggle'; + mocks.state.dictationEnabled = true; mocks.state.dictationModelId = 'sherpa-parakeet-tdt-v3'; + mocks.state.loadDictationPrefs = async () => {}; FakeWebSocket.instances = []; global.WebSocket = FakeWebSocket; global.MediaRecorder = FakeMediaRecorder; @@ -159,14 +207,93 @@ describe('CaptureWidget', () => { }); afterEach(() => { + vi.restoreAllMocks(); delete window.__TAURI_INTERNALS__; delete global.WebSocket; delete global.MediaRecorder; }); const pasteCalls = () => mocks.holder.calls.filter(([c]) => c === 'simulate_paste'); + const copyCalls = () => mocks.holder.calls.filter(([c]) => c === 'copy_dictation_output_session'); const typeCalls = () => mocks.holder.calls.filter(([c]) => c === 'simulate_type'); + it('provides a translated Inserted outcome in every supported locale', () => { + expect(Object.keys(captureLocales)).toHaveLength(21); + for (const [path, locale] of Object.entries(captureLocales)) { + expect(locale.capture.inserted, path).toBeTruthy(); + if (!path.endsWith('/en.json')) expect(locale.capture.inserted, path).not.toBe('Inserted'); + } + }); + + it('attaches the native listener but waits for persisted prefs before ready and start', async () => { + let resolveHydration; + mocks.state.loadDictationPrefs = () => + new Promise((resolve) => { + resolveHydration = () => { + mocks.state.dictationMode = 'hold'; + mocks.state.dictationModelId = 'sherpa-whisper-tiny'; + resolve(); + }; + }); + const getUserMedia = vi.fn(async () => ({ getTracks: () => [{ stop() {} }] })); + Object.defineProperty(navigator, 'mediaDevices', { + configurable: true, + value: { getUserMedia }, + }); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + expect(mocks.holder.calls.some(([command]) => command === 'mark_dictation_capture_ready')).toBe( + false, + ); + + let earlyStart; + act(() => { + earlyStart = mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'hydrated-session' }, + }); + }); + await act(async () => Promise.resolve()); + expect(getUserMedia).not.toHaveBeenCalled(); + expect(mocks.holder.calls).not.toContainEqual([ + 'activate_dictation_output_session', + { sessionId: 'hydrated-session' }, + ]); + + await act(async () => { + resolveHydration(); + await earlyStart; + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1)); + await screen.findByText(/Listening/); + expect(FakeWebSocket.instances[0].url).toContain('model=sherpa-whisper-tiny'); + expect(mocks.holder.calls.some(([command]) => command === 'mark_dictation_capture_ready')).toBe( + true, + ); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate-stop'](); + }); + await screen.findByText(/Transcribing/); + }); + + it('marks native capture ready with safe seeds when prefs hydration fails', async () => { + mocks.state.loadDictationPrefs = async () => { + throw new Error('prefs backend unavailable'); + }; + vi.spyOn(console, 'warn').mockImplementation(() => {}); + render(withI18n()); + + await waitFor(() => + expect( + mocks.holder.calls.some(([command]) => command === 'mark_dictation_capture_ready'), + ).toBe(true), + ); + const ws = await startNativeSession('seed-session'); + expect(ws.url).toContain('model=sherpa-parakeet-tdt-v3'); + }); + it('honors a hold-mode release while microphone startup is pending', async () => { mocks.state.dictationMode = 'hold'; let resolveMicrophone; @@ -182,17 +309,328 @@ describe('CaptureWidget', () => { }); render(withI18n()); - fireEvent.keyDown(window, { code: 'Space', ctrlKey: true, shiftKey: true }); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'hold-session' }, + }); + }); + await waitFor(() => expect(getUserMedia).toHaveBeenCalledOnce()); + await act(async () => { + await mocks.holder.handlers['tray-dictate-stop'](); + }); + await act(async () => { + resolveMicrophone({ getTracks: () => [{ stop() {} }] }); + }); + + expect(await screen.findByText(/Transcribing/, {}, { timeout: 3000 })).toBeInTheDocument(); + expect(screen.queryByText(/Listening/)).not.toBeInTheDocument(); + }); + + it('keeps recording for a new hold press after the previous hold was released during startup', async () => { + mocks.state.dictationMode = 'hold'; + let resolveMicrophone; + const getUserMedia = vi.fn( + () => + new Promise((resolve) => { + resolveMicrophone = resolve; + }), + ); + Object.defineProperty(navigator, 'mediaDevices', { + configurable: true, + value: { getUserMedia }, + }); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'first-hold-session' }, + }); + }); await waitFor(() => expect(getUserMedia).toHaveBeenCalledOnce()); - fireEvent.keyUp(window, { code: 'Space' }); + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate-stop'](); + }); + now.mockReturnValue(1400); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'second-hold-session' }, + }); + }); + await act(async () => { resolveMicrophone({ getTracks: () => [{ stop() {} }] }); }); - expect(await screen.findByText(/Transcribing/)).toBeInTheDocument(); + expect(await screen.findByText(/Listening/, {}, { timeout: 3000 })).toBeInTheDocument(); + expect(screen.queryByText(/Transcribing/)).not.toBeInTheDocument(); + expect(getUserMedia).toHaveBeenCalledOnce(); + }); + + it('honors a hold-mode release while native session activation is pending', async () => { + mocks.state.dictationMode = 'hold'; + let resolveActivation; + mocks.holder.activate = () => + new Promise((resolve) => { + resolveActivation = resolve; + }); + const getUserMedia = vi.fn(async () => ({ getTracks: () => [{ stop() {} }] })); + Object.defineProperty(navigator, 'mediaDevices', { + configurable: true, + value: { getUserMedia }, + }); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + + let startEvent; + act(() => { + startEvent = mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'activating-hold-session' }, + }); + }); + await waitFor(() => expect(resolveActivation).toBeTypeOf('function')); + expect(getUserMedia).not.toHaveBeenCalled(); + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate-stop'](); + }); + await act(async () => { + resolveActivation(); + await startEvent; + }); + + expect(await screen.findByText(/Transcribing/, {}, { timeout: 3000 })).toBeInTheDocument(); expect(screen.queryByText(/Listening/)).not.toBeInTheDocument(); }); + it('allows one microphone startup and adopts the newest native output session', async () => { + let resolveMicrophone; + const getUserMedia = vi.fn( + () => + new Promise((resolve) => { + resolveMicrophone = resolve; + }), + ); + Object.defineProperty(navigator, 'mediaDevices', { + configurable: true, + value: { getUserMedia }, + }); + const now = vi.spyOn(Date, 'now'); + render(withI18n()); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + + now.mockReturnValue(1000); + act(() => { + mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'session-first' } }); + }); + now.mockReturnValue(1200); + act(() => { + mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'session-competing' } }); + }); + await waitFor(() => expect(getUserMedia).toHaveBeenCalled()); + + expect(getUserMedia).toHaveBeenCalledOnce(); + expect( + mocks.holder.calls + .filter(([command]) => command === 'activate_dictation_output_session') + .map(([, args]) => args.sessionId), + ).toEqual(['session-first', 'session-competing']); + + await act(async () => { + resolveMicrophone({ getTracks: () => [{ stop() {} }] }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1)); + const ws = FakeWebSocket.instances[0]; + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'latest target' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'latest target' })); + await screen.findByText(/Inserted/); + + expect(pasteCalls()[0][1].sessionId).toBe('session-competing'); + expect(mocks.holder.calls).toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'session-competing' }, + ]); + now.mockRestore(); + }); + + it('replays an activated start after the previous startup terminates while unwinding', async () => { + let resolveMicSetup; + let micSetupAttempt = 0; + mocks.holder.startMic = (_stream, onFrame) => { + mocks.holder.onFrame = onFrame; + micSetupAttempt += 1; + if (micSetupAttempt > 1) return Promise.resolve(async () => {}); + return new Promise((resolve) => { + resolveMicSetup = () => resolve(async () => {}); + }); + }; + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + + act(() => { + void mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'failed-startup' } }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1)); + await waitFor(() => expect(resolveMicSetup).toBeTypeOf('function')); + await act(async () => { + await FakeWebSocket.instances[0].msgAsync({ + type: 'final', + final_kind: 'summary', + text: '', + model_silent: 'sherpa-parakeet-tdt-v3', + }); + }); + await screen.findByText(/Transcription failed/); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'replayed-startup' } }); + }); + await act(async () => { + resolveMicSetup(); + await Promise.resolve(); + await Promise.resolve(); + }); + + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + await screen.findByText(/Listening/); + expect(mocks.holder.calls).toContainEqual([ + 'activate_dictation_output_session', + { sessionId: 'replayed-startup' }, + ]); + }); + + it('does not finish a replacement session when the adopted startup then terminates', async () => { + let resolveMicSetup; + let micSetupAttempt = 0; + mocks.holder.startMic = (_stream, onFrame) => { + mocks.holder.onFrame = onFrame; + micSetupAttempt += 1; + if (micSetupAttempt > 1) return Promise.resolve(async () => {}); + return new Promise((resolve) => { + resolveMicSetup = () => resolve(async () => {}); + }); + }; + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + await waitFor(() => expect(mocks.holder.handlers['tray-dictate']).toBeTypeOf('function')); + + act(() => { + void mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'old-startup' } }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1)); + await waitFor(() => expect(resolveMicSetup).toBeTypeOf('function')); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'replacement-startup' }, + }); + }); + await act(async () => { + await FakeWebSocket.instances[0].msgAsync({ + type: 'final', + final_kind: 'summary', + text: '', + model_silent: 'sherpa-parakeet-tdt-v3', + }); + }); + await act(async () => { + resolveMicSetup(); + await Promise.resolve(); + await Promise.resolve(); + }); + + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + expect(mocks.holder.calls).not.toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'replacement-startup' }, + ]); + now.mockRestore(); + }); + + it('activates a rapid restart before a late old finish resolves', async () => { + let resolveFinish; + const finishPending = new Promise((resolve) => { + resolveFinish = resolve; + }); + mocks.holder.finish = () => finishPending; + const getUserMedia = vi.fn(async () => ({ getTracks: () => [{ stop() {} }] })); + Object.defineProperty(navigator, 'mediaDevices', { + configurable: true, + value: { getUserMedia }, + }); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + const ws = await startNativeSession('finishing-session'); + + act(() => + ws.msg({ + type: 'final', + final_kind: 'summary', + text: '', + model_silent: 'sherpa-parakeet-tdt-v3', + }), + ); + await screen.findByText(/Transcription failed/); + now.mockReturnValue(1200); + act(() => { + mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'fresh-session' } }); + }); + await waitFor(() => + expect(mocks.holder.calls).toContainEqual([ + 'activate_dictation_output_session', + { sessionId: 'fresh-session' }, + ]), + ); + await waitFor(() => expect(getUserMedia).toHaveBeenCalledTimes(2)); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + await act(async () => resolveFinish()); + const freshWs = FakeWebSocket.instances[1]; + act(() => freshWs.msg({ type: 'final', final_kind: 'utterance', text: 'fresh words' })); + act(() => freshWs.msg({ type: 'final', final_kind: 'summary', text: 'fresh words' })); + + await screen.findByText(/Inserted/); + expect(pasteCalls().at(-1)[1].sessionId).toBe('fresh-session'); + }); + + it('rejects an ignored native candidate while an active session is transcribing', async () => { + mocks.state.dictationMode = 'hold'; + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + render(withI18n()); + await startNativeSession('active-session'); + + now.mockReturnValue(1100); + await act(async () => { + await mocks.holder.handlers['tray-dictate-stop'](); + }); + await screen.findByText(/Transcribing/); + now.mockReturnValue(1300); + act(() => { + mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'ignored-candidate' } }); + }); + + await waitFor(() => + expect(mocks.holder.calls).toContainEqual([ + 'reject_dictation_output_session', + { sessionId: 'ignored-candidate' }, + ]), + ); + expect(mocks.holder.calls).not.toContainEqual([ + 'activate_dictation_output_session', + { sessionId: 'ignored-candidate' }, + ]); + expect(mocks.holder.calls).not.toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'ignored-candidate' }, + ]); + }); + it('falls back to raw PCM when MediaRecorder cannot be constructed', async () => { mocks.state.dictationModelId = null; delete global.MediaRecorder; @@ -225,31 +663,179 @@ describe('CaptureWidget', () => { expect(screen.getByText(/Listening/)).toBeInTheDocument(); }); - it('shows "Pasted" only after simulate_paste resolved Ok', async () => { + it('surfaces a silent selected model as an error instead of successful no-speech', async () => { + render(withI18n()); + const ws = await startSession(); + + await act( + async () => + await ws.msgAsync({ + type: 'final', + final_kind: 'summary', + text: '', + model_silent: 'sherpa-parakeet-tdt-v3', + warning: 'The selected dictation model produced no text.', + }), + ); + + await screen.findByText(/Transcription failed/); + expect(screen.queryByText(/No speech detected/)).not.toBeInTheDocument(); + expect(pasteCalls()).toEqual([]); + await waitFor(() => + expect(mocks.holder.calls).toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'capture-session-1' }, + ]), + ); + }); + + it('shows "Inserted" only after simulate_paste resolved Ok', async () => { render(withI18n()); const ws = await startSession(); // Offline-model shape: one utterance final, then the EOF summary. - act(() => ws.msg({ type: 'final', text: 'hello world' })); - act(() => ws.msg({ type: 'final', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); - await screen.findByText(/Pasted/); + await screen.findByText(/Inserted/); expect(pasteCalls().length).toBeGreaterThan(0); - expect(pasteCalls()[0][1]).toEqual({ text: 'hello world' }); + expect(pasteCalls()[0][1]).toEqual({ + text: 'hello world', + sessionId: 'capture-session-1', + }); + }); + + it('keeps native delivery session-bound and never pre-writes the WebView clipboard', async () => { + render(withI18n()); + const ws = await startNativeSession('native-session-7'); + + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); + + await screen.findByText(/Inserted/); + expect(mocks.copyText).not.toHaveBeenCalled(); + expect(pasteCalls()).toEqual([ + ['simulate_paste', { text: 'hello world', sessionId: 'native-session-7' }], + ]); + expect(mocks.holder.calls).toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'native-session-7' }, + ]); + }); + + it('uses the WebView clipboard only after native clipboard delivery fails', async () => { + mocks.state.dictationModelId = null; + const order = []; + mocks.holder.paste = async () => { + order.push('native'); + throw 'clipboard: native clipboard unavailable'; + }; + mocks.copyText.mockImplementation(async (text) => { + order.push('webview'); + expect(text).toBe('fallback words'); + }); + render(withI18n()); + const ws = await startNativeSession('wayland-session'); + + act(() => ws.msg({ type: 'final', text: 'fallback words' })); + + await screen.findByText(/Copied/); + expect(order).toEqual(['native', 'webview']); + expect(pasteCalls()).toEqual([ + ['simulate_paste', { text: 'fallback words', sessionId: 'wayland-session' }], + ]); + }); + + it('never lets an old legacy delivery finish a restarted native session', async () => { + mocks.state.dictationModelId = null; + let resolveOldPaste; + mocks.holder.paste = () => + new Promise((resolve) => { + resolveOldPaste = resolve; + }); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + const { container } = render(withI18n()); + const oldWs = await startNativeSession('old-legacy-session'); + + act(() => oldWs.msg({ type: 'final', text: 'old legacy words' })); + await waitFor(() => expect(pasteCalls()).toHaveLength(1)); + fireEvent.keyDown(window, { key: 'Escape' }); + await waitFor(() => expect(container.querySelector('.capture-pill')).toBeNull()); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'new-legacy-session' }, + }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + await screen.findByText(/Listening/); + + await act(async () => { + resolveOldPaste('inserted'); + await Promise.resolve(); + await Promise.resolve(); + }); + + expect(screen.getByText(/Listening/)).toBeInTheDocument(); + expect(mocks.holder.calls).not.toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'new-legacy-session' }, + ]); + }); + + it('discards an old deferred POST result after cancel and restart', async () => { + mocks.state.dictationModelId = null; + let resolveOldPost; + mocks.apiFetch.mockImplementation( + () => + new Promise((resolve) => { + resolveOldPost = resolve; + }), + ); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + const { container } = render(withI18n()); + const oldWs = await startNativeSession('old-post-session'); + + act(() => oldWs.msg({ type: 'error', kind: 'server', message: 'socket failed' })); + await waitFor(() => expect(mocks.apiFetch).toHaveBeenCalledOnce()); + fireEvent.keyDown(window, { key: 'Escape' }); + await waitFor(() => expect(container.querySelector('.capture-pill')).toBeNull()); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'new-post-session' } }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + await screen.findByText(/Listening/); + + await act(async () => { + resolveOldPost({ json: async () => ({ text: 'old post words' }) }); + await Promise.resolve(); + await Promise.resolve(); + }); + + expect(screen.getByText(/Listening/)).toBeInTheDocument(); + expect(pasteCalls()).toEqual([]); + expect(mocks.holder.calls).not.toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'new-post-session' }, + ]); }); - it('an "a11y:" paste rejection renders the actionable error, never "Pasted"', async () => { + it('an "a11y:" paste rejection renders the actionable error, never "Inserted"', async () => { mocks.holder.paste = async () => { throw 'a11y: process is not trusted'; }; render(withI18n()); const ws = await startSession(); - act(() => ws.msg({ type: 'final', text: 'hello world' })); - act(() => ws.msg({ type: 'final', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); await screen.findByText(/Accessibility access needed/); - expect(screen.queryByText(/Pasted/)).not.toBeInTheDocument(); + expect(screen.queryByText(/Inserted/)).not.toBeInTheDocument(); + expect(mocks.copyText).not.toHaveBeenCalled(); // The action button opens the OS Accessibility pane. fireEvent.click(screen.getByText('Open Settings')); @@ -274,10 +860,10 @@ describe('CaptureWidget', () => { act(() => ws.msg({ type: 'partial', text: 'hel' })); act(() => ws.msg({ type: 'partial', text: 'hello wor' })); - act(() => ws.msg({ type: 'final', text: 'hello world' })); - act(() => ws.msg({ type: 'final', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); - await screen.findByText(/Pasted/); + await screen.findByText(/Inserted/); // Committed final went through the paste path; no keystroke storms. expect(typeCalls()).toEqual([]); expect(pasteCalls().length).toBeGreaterThan(0); @@ -290,7 +876,233 @@ describe('CaptureWidget', () => { act(() => ws.msg({ type: 'partial', text: 'hello' })); await waitFor(() => expect(typeCalls().length).toBeGreaterThan(0)); - expect(typeCalls()[0][1]).toEqual({ text: 'hello', backspaces: 0 }); + expect(typeCalls()[0][1]).toEqual({ + text: 'hello', + backspaces: 0, + sessionId: 'capture-session-1', + }); + }); + + it('never runs an old queued live-type delta against a restarted native session', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + let resolveFirstType; + mocks.holder.type = vi.fn( + () => + new Promise((resolve) => { + resolveFirstType = resolve; + }), + ); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + const { container } = render(withI18n()); + const oldWs = await startNativeSession('old-type-session'); + + act(() => oldWs.msg({ type: 'partial', text: 'hello' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + act(() => oldWs.msg({ type: 'partial', text: 'hello world' })); + fireEvent.keyDown(window, { key: 'Escape' }); + await waitFor(() => expect(container.querySelector('.capture-pill')).toBeNull()); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'new-type-session' }, + }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + await screen.findByText(/Listening/); + + await act(async () => { + resolveFirstType('inserted'); + await Promise.resolve(); + await Promise.resolve(); + await Promise.resolve(); + }); + + expect(typeCalls()).toHaveLength(1); + expect(typeCalls()[0][1].sessionId).toBe('old-type-session'); + }); + + it('never lets an old utterance commit corrupt a restarted live-type prefix', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + let resolveOldType; + mocks.holder.type = vi + .fn() + .mockImplementationOnce( + () => + new Promise((resolve) => { + resolveOldType = resolve; + }), + ) + .mockResolvedValue('inserted'); + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + const { container } = render(withI18n()); + const oldWs = await startNativeSession('old-commit-session'); + + act(() => oldWs.msg({ type: 'partial', text: 'old' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + act(() => oldWs.msg({ type: 'final', final_kind: 'utterance', text: 'old words' })); + fireEvent.keyDown(window, { key: 'Escape' }); + await waitFor(() => expect(container.querySelector('.capture-pill')).toBeNull()); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ payload: { sessionId: 'new-commit-session' } }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + const newWs = FakeWebSocket.instances[1]; + await act(async () => { + resolveOldType('inserted'); + await Promise.resolve(); + await Promise.resolve(); + }); + act(() => newWs.msg({ type: 'partial', text: 'fresh' })); + + await waitFor(() => expect(typeCalls()).toHaveLength(2)); + expect(typeCalls()[1][1]).toEqual({ + text: 'fresh', + backspaces: 0, + sessionId: 'new-commit-session', + }); + }); + + it('copies the summary without pasting after a later live-type delta fails', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + let typeAttempt = 0; + mocks.holder.type = async () => { + typeAttempt += 1; + if (typeAttempt === 2) throw 'paste: input synthesis failed'; + return 'inserted'; + }; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'partial', text: 'hello' })); + act(() => ws.msg({ type: 'partial', text: 'hello world' })); + await waitFor(() => expect(typeCalls()).toHaveLength(2)); + await act(async () => Promise.resolve()); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); + + await screen.findByText(/Copied/); + expect(pasteCalls()).toEqual([]); + expect(copyCalls()).toEqual([ + ['copy_dictation_output_session', { text: 'hello world', sessionId: 'capture-session-1' }], + ]); + }); + + it('copies the summary when the first live-type call may have partially emitted', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + mocks.holder.type = async () => { + throw 'paste: input synthesis failed'; + }; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'partial', text: 'hello world' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + await act(async () => Promise.resolve()); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); + + await screen.findByText(/Copied/); + expect(pasteCalls()).toEqual([]); + expect(copyCalls()).toEqual([ + ['copy_dictation_output_session', { text: 'hello world', sessionId: 'capture-session-1' }], + ]); + }); + + it('downgrades a zero-emission live-type preflight failure to committed copy', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + mocks.holder.type = async () => { + throw 'preflight: native session is clipboard-only'; + }; + mocks.holder.paste = async () => 'copied'; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'partial', text: 'hello world' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + await act(async () => Promise.resolve()); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); + + await screen.findByText(/Copied/); + expect(pasteCalls().map(([, args]) => args.text)).toEqual(['hello world', 'hello world']); + }); + + it('commits an utterance when its pending first live-type call fails preflight', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + let rejectType; + mocks.holder.type = () => + new Promise((_resolve, reject) => { + rejectType = reject; + }); + mocks.holder.paste = async () => 'copied'; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'partial', text: 'hello world' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); + expect(pasteCalls()).toEqual([]); + + await act(async () => rejectType('preflight: native session is clipboard-only')); + + await screen.findByText(/Copied/); + expect(pasteCalls().map(([, args]) => args.text)).toEqual(['hello world', 'hello world']); + }); + + it('copies the full summary without reinserting when preflight fails after a live prefix', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + let typeAttempt = 0; + mocks.holder.type = async () => { + typeAttempt += 1; + if (typeAttempt === 2) throw 'preflight: native input became unavailable'; + return 'inserted'; + }; + mocks.holder.paste = async () => 'copied'; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'partial', text: 'hello' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + act(() => ws.msg({ type: 'partial', text: 'hello world' })); + await waitFor(() => expect(typeCalls()).toHaveLength(2)); + await act(async () => Promise.resolve()); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); + + await screen.findByText(/Copied/); + expect(pasteCalls()).toEqual([]); + expect(copyCalls()).toEqual([ + ['copy_dictation_output_session', { text: 'hello world', sessionId: 'capture-session-1' }], + ]); + }); + + it('waits for an in-flight native type result before delivering a summary', async () => { + localStorage.setItem('omni_capture_live_typing', '1'); + let rejectType; + mocks.holder.type = () => + new Promise((_resolve, reject) => { + rejectType = reject; + }); + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'partial', text: 'hello world' })); + await waitFor(() => expect(typeCalls()).toHaveLength(1)); + await act(async () => { + ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' }); + await Promise.resolve(); + }); + + expect(pasteCalls()).toEqual([]); + await act(async () => rejectType('paste: input synthesis failed')); + await screen.findByText(/Copied/); + expect(pasteCalls()).toEqual([]); + expect(copyCalls()).toEqual([ + ['copy_dictation_output_session', { text: 'hello world', sessionId: 'capture-session-1' }], + ]); }); it('a refined EOF summary is not re-pasted as a new utterance', async () => { @@ -298,21 +1110,138 @@ describe('CaptureWidget', () => { const ws = await startSession(); // Two per-utterance commits paste live… - act(() => ws.msg({ type: 'final', text: 'Hello world.' })); - act(() => ws.msg({ type: 'final', text: 'Second bit.' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'Hello world.' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'Second bit.' })); // …then the EOF summary arrives with an LLM-refined variant. Its raw // `text` equals the committed join, so it must finalise — never paste // the whole (refined) transcript a third time. act(() => ws.msg({ type: 'final', + final_kind: 'summary', text: 'Hello world. Second bit.', refined_text: 'Hello world, second bit.', }), ); - await screen.findByText(/Pasted/); - expect(pasteCalls().map(([, a]) => a.text)).toEqual(['Hello world.', 'Second bit.']); + await screen.findByText(/Inserted/); + expect(pasteCalls().map(([, a]) => a.text)).toEqual(['Hello world.', ' Second bit.']); + }); + + it('commits repeated identical utterances instead of mistaking the second for EOF', async () => { + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'Yes.' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'Yes.' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'Yes. Yes.' })); + + await screen.findByText(/Inserted/); + expect(pasteCalls().map(([, args]) => args.text)).toEqual(['Yes.', ' Yes.']); + }); + + it('delivers only an uncommitted EOF-summary tail', async () => { + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'First.' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'First. Tail.' })); + + await screen.findByText(/Inserted/); + expect(pasteCalls().map(([, args]) => args.text)).toEqual(['First.', ' Tail.']); + }); + + it('keeps copied as the session outcome and refreshes the full summary once', async () => { + let copied = false; + mocks.holder.paste = async (_cmd, { text }) => { + if (text === ' Second.') copied = true; + return copied ? 'copied' : 'inserted'; + }; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'First.' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'Second.' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'First. Second.' })); + + await screen.findByText(/Copied/); + expect(pasteCalls().map(([, args]) => args.text)).toEqual([ + 'First.', + ' Second.', + 'First. Second.', + ]); + expect(mocks.copyText).not.toHaveBeenCalled(); + }); + + it('refreshes a WebView clipboard fallback without retrying native insertion', async () => { + let attempt = 0; + mocks.holder.paste = async () => { + attempt += 1; + if (attempt === 1) return 'inserted'; + if (attempt === 2) throw 'clipboard: native clipboard unavailable'; + return 'inserted'; + }; + render(withI18n()); + const ws = await startSession(); + + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'First.' })); + await waitFor(() => expect(pasteCalls()).toHaveLength(1)); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'Second.' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'First. Second.' })); + + await screen.findByText(/Copied/); + expect(pasteCalls().map(([, args]) => args.text)).toEqual(['First.', ' Second.']); + expect(mocks.copyText).toHaveBeenNthCalledWith(1, ' Second.'); + expect(mocks.copyText).toHaveBeenNthCalledWith(2, 'First. Second.'); + }); + + it('never lets an old deferred delivery finish or mutate a restarted session', async () => { + let resolveOldPaste; + let pasteAttempt = 0; + mocks.holder.paste = () => { + pasteAttempt += 1; + if (pasteAttempt === 1) { + return new Promise((resolve) => { + resolveOldPaste = resolve; + }); + } + return Promise.resolve('inserted'); + }; + const now = vi.spyOn(Date, 'now').mockReturnValue(1000); + const { container } = render(withI18n()); + const oldWs = await startNativeSession('old-delivery-session'); + + act(() => oldWs.msg({ type: 'final', final_kind: 'utterance', text: 'old words' })); + await waitFor(() => expect(pasteCalls()).toHaveLength(1)); + act(() => oldWs.msg({ type: 'final', final_kind: 'summary', text: 'old words' })); + await act(async () => Promise.resolve()); + fireEvent.keyDown(window, { key: 'Escape' }); + await waitFor(() => expect(container.querySelector('.capture-pill')).toBeNull()); + + now.mockReturnValue(1200); + await act(async () => { + await mocks.holder.handlers['tray-dictate']({ + payload: { sessionId: 'new-delivery-session' }, + }); + }); + await waitFor(() => expect(FakeWebSocket.instances).toHaveLength(2)); + await screen.findByText(/Listening/); + + await act(async () => { + resolveOldPaste('copied'); + await Promise.resolve(); + await Promise.resolve(); + await Promise.resolve(); + }); + + expect(screen.getByText(/Listening/)).toBeInTheDocument(); + expect(pasteCalls()).toEqual([ + ['simulate_paste', { text: 'old words', sessionId: 'old-delivery-session' }], + ]); + expect(mocks.holder.calls).not.toContainEqual([ + 'finish_dictation_output_session', + { sessionId: 'new-delivery-session' }, + ]); }); it('renders the one-time Accessibility setup state when the mount probe fails', async () => { @@ -350,6 +1279,24 @@ describe('CaptureWidget', () => { } }); + it('records from the Accessibility setup state and reports native clipboard fallback', async () => { + mocks.holder.a11y = false; + mocks.holder.paste = async () => 'copied'; + render(withI18n()); + await screen.findByText(/Allow Accessibility/); + expect(screen.getByText('Open Settings')).toBeInTheDocument(); + + const ws = await startNativeSession('a11y-copy-session'); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'clipboard words' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'clipboard words' })); + + await screen.findByText(/Copied/); + expect(pasteCalls().map(([, args]) => args)).toEqual([ + { text: 'clipboard words', sessionId: 'a11y-copy-session' }, + { text: 'clipboard words', sessionId: 'a11y-copy-session' }, + ]); + }); + it('waveform bars move from the worklet mic frames', async () => { const { container } = render(withI18n()); await startSession(); @@ -404,8 +1351,8 @@ describe('CaptureWidget', () => { const ws = await startSession(); // Drive to the error state on real timers so the paste rejection settles. - act(() => ws.msg({ type: 'final', text: 'hello world' })); - act(() => ws.msg({ type: 'final', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'utterance', text: 'hello world' })); + act(() => ws.msg({ type: 'final', final_kind: 'summary', text: 'hello world' })); await screen.findByText(/Accessibility access needed/); vi.useFakeTimers(); diff --git a/frontend/src/i18n/locales/ar.json b/frontend/src/i18n/locales/ar.json index a4990dd1b..89235b04d 100644 --- a/frontend/src/i18n/locales/ar.json +++ b/frontend/src/i18n/locales/ar.json @@ -785,6 +785,8 @@ "transcribing_label": "جارٍ النسخ…", "a11y_setup": "اسمح بتسهيلات الاستخدام حتى يتمكّن الإملاء من الكتابة نيابةً عنك", "pasted": "تم لصقه", + "inserted": "تم الإدخال", + "copied": "تم النسخ إلى الحافظة", "no_speech": "لم يتم اكتشاف أي كلام", "mic_denied": "تم رفض الوصول إلى الميكروفون", "mic_denied_toast": "تم رفض الوصول إلى الميكروفون. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "حذف نموذج الكلام", "engine_unavailable": "محرك الإملاء المباشر غير متوفر في هذا التثبيت. يعود الإملاء إلى مسار النسخ القياسي.", "model_desc": { - "sherpa-parakeet-tdt-v3": "موصى به. إملاء سريع ودقيق عبر 25 لغة أوروبية.", + "sherpa-parakeet-tdt-v3": "إملاء سريع ودقيق عبر 25 لغة أوروبية.", "sherpa-parakeet-tdt-v2": "إملاء سريع ودقيق باللغة الإنجليزية فقط.", "sherpa-zipformer-bilingual-zh-en": "بث الصينية + الإنجليزية مع أجزاء حية.", "sherpa-paraformer-bilingual-zh-en": "تدفق الصينية + الإنجليزية، نموذج مدمج.", "sherpa-zipformer-en-20m": "نموذج صغير للبث باللغة الإنجليزية – أقل زمن وصول.", "sherpa-zipformer-zh-14m": "النموذج الصيني المتدفق الصغير – أقل زمن وصول.", - "sherpa-whisper-tiny": "متعدد اللغات (+90 لغة) مع الكشف التلقائي عن اللغة." + "sherpa-whisper-tiny": "موصى به. إملاء متعدد اللغات لأكثر من 90 لغة مع اكتشاف اللغة تلقائيًا." } }, "profiles": { diff --git a/frontend/src/i18n/locales/de.json b/frontend/src/i18n/locales/de.json index fd738f35b..3743a9202 100644 --- a/frontend/src/i18n/locales/de.json +++ b/frontend/src/i18n/locales/de.json @@ -785,6 +785,8 @@ "transcribing_label": "Transkribieren…", "a11y_setup": "Bedienungshilfen erlauben, damit das Diktat für Sie tippen kann", "pasted": "Eingefügt", + "inserted": "Eingegeben", + "copied": "In die Zwischenablage kopiert", "no_speech": "Keine Sprache erkannt", "mic_denied": "Mikrofonzugriff verweigert", "mic_denied_toast": "Mikrofonzugriff verweigert. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Sprachmodell löschen", "engine_unavailable": "Die Live-Diktier-Engine ist bei dieser Installation nicht verfügbar. Beim Diktat wird auf den Standard-Transkriptionspfad zurückgegriffen.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Empfohlen. Schnelles und genaues Diktieren in 25 europäischen Sprachen.", + "sherpa-parakeet-tdt-v3": "Schnelles und genaues Diktieren in 25 europäischen Sprachen.", "sherpa-parakeet-tdt-v2": "Schnelles, genaues Diktat nur auf Englisch.", "sherpa-zipformer-bilingual-zh-en": "Streaming Chinesisch + Englisch mit Live-Teilabschnitten.", "sherpa-paraformer-bilingual-zh-en": "Streaming Chinesisch + Englisch, kompaktes Modell.", "sherpa-zipformer-en-20m": "Winziges englisches Streaming-Modell – niedrigste Latenz.", "sherpa-zipformer-zh-14m": "Winziges chinesisches Streaming-Modell – niedrigste Latenz.", - "sherpa-whisper-tiny": "Mehrsprachig (über 90 Sprachen) mit automatischer Spracherkennung." + "sherpa-whisper-tiny": "Empfohlen. Mehrsprachiges Diktieren in über 90 Sprachen mit automatischer Spracherkennung." } }, "profiles": { diff --git a/frontend/src/i18n/locales/en.json b/frontend/src/i18n/locales/en.json index 000f61836..3a92b784e 100644 --- a/frontend/src/i18n/locales/en.json +++ b/frontend/src/i18n/locales/en.json @@ -1038,6 +1038,7 @@ "listening_label": "Listening…", "transcribing_label": "Transcribing…", "pasted": "Pasted", + "inserted": "Inserted", "copied": "Copied to clipboard", "no_speech": "No speech detected", "model_downloading": "Downloading voice model…", @@ -1080,13 +1081,13 @@ "delete_confirm_title": "Delete speech model", "engine_unavailable": "The live-dictation engine isn't available on this install. Dictation falls back to the standard transcription path.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Recommended. Fast, accurate dictation across 25 European languages.", + "sherpa-parakeet-tdt-v3": "Fast, accurate dictation across 25 European languages.", "sherpa-parakeet-tdt-v2": "Fast, accurate English-only dictation.", "sherpa-zipformer-bilingual-zh-en": "Streaming Chinese + English with live partials.", "sherpa-paraformer-bilingual-zh-en": "Streaming Chinese + English, compact model.", "sherpa-zipformer-en-20m": "Tiny streaming English model — lowest latency.", "sherpa-zipformer-zh-14m": "Tiny streaming Chinese model — lowest latency.", - "sherpa-whisper-tiny": "Multilingual (90+ languages) with auto language detection." + "sherpa-whisper-tiny": "Recommended. Multilingual dictation across 90+ languages with automatic language detection." } }, "profiles": { diff --git a/frontend/src/i18n/locales/es.json b/frontend/src/i18n/locales/es.json index 9814600a7..e8f9daa62 100644 --- a/frontend/src/i18n/locales/es.json +++ b/frontend/src/i18n/locales/es.json @@ -785,6 +785,8 @@ "transcribing_label": "Transcribiendo…", "a11y_setup": "Permite Accesibilidad para que el dictado pueda escribir por ti", "pasted": "Pegado", + "inserted": "Insertado", + "copied": "Copiado al portapapeles", "no_speech": "No se detectó voz", "mic_denied": "Acceso al micrófono denegado", "mic_denied_toast": "Acceso al micrófono denegado. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Eliminar modelo de voz", "engine_unavailable": "El motor de dictado en vivo no está disponible en esta instalación. El dictado vuelve a la ruta de transcripción estándar.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Recomendado. Dictado rápido y preciso en 25 idiomas europeos.", + "sherpa-parakeet-tdt-v3": "Dictado rápido y preciso en 25 idiomas europeos.", "sherpa-parakeet-tdt-v2": "Dictado rápido y preciso solo en inglés.", "sherpa-zipformer-bilingual-zh-en": "Streaming chino + inglés con parciales en vivo.", "sherpa-paraformer-bilingual-zh-en": "Streaming chino + inglés, modelo compacto.", "sherpa-zipformer-en-20m": "Pequeño modelo de transmisión en inglés: latencia más baja.", "sherpa-zipformer-zh-14m": "Pequeño modelo chino de transmisión: latencia más baja.", - "sherpa-whisper-tiny": "Multilingüe (más de 90 idiomas) con detección automática de idioma." + "sherpa-whisper-tiny": "Recomendado. Dictado multilingüe en más de 90 idiomas con detección automática del idioma." } }, "profiles": { diff --git a/frontend/src/i18n/locales/fr.json b/frontend/src/i18n/locales/fr.json index 8224d812c..796614230 100644 --- a/frontend/src/i18n/locales/fr.json +++ b/frontend/src/i18n/locales/fr.json @@ -785,6 +785,8 @@ "transcribing_label": "Transcription…", "a11y_setup": "Autorisez l'accessibilité pour que la dictée puisse écrire à votre place", "pasted": "Collé", + "inserted": "Inséré", + "copied": "Copié dans le presse-papiers", "no_speech": "Aucune parole détectée", "mic_denied": "Accès au micro refusé", "mic_denied_toast": "Accès au microphone refusé. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Supprimer le modèle vocal", "engine_unavailable": "Le moteur de dictée en direct n'est pas disponible sur cette installation. La dictée revient au chemin de transcription standard.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Recommandé. Dictée rapide et précise dans 25 langues européennes.", + "sherpa-parakeet-tdt-v3": "Dictée rapide et précise dans 25 langues européennes.", "sherpa-parakeet-tdt-v2": "Dictée rapide et précise en anglais uniquement.", "sherpa-zipformer-bilingual-zh-en": "Streaming chinois + anglais avec partiels en direct.", "sherpa-paraformer-bilingual-zh-en": "Streaming chinois + anglais, modèle compact.", "sherpa-zipformer-en-20m": "Petit modèle anglais de streaming – latence la plus faible.", "sherpa-zipformer-zh-14m": "Petit modèle chinois de streaming – latence la plus faible.", - "sherpa-whisper-tiny": "Multilingue (plus de 90 langues) avec détection automatique de la langue." + "sherpa-whisper-tiny": "Recommandé. Dictée multilingue dans plus de 90 langues avec détection automatique de la langue." } }, "profiles": { diff --git a/frontend/src/i18n/locales/hi.json b/frontend/src/i18n/locales/hi.json index fb5cd30cf..74764023a 100644 --- a/frontend/src/i18n/locales/hi.json +++ b/frontend/src/i18n/locales/hi.json @@ -785,6 +785,8 @@ "transcribing_label": "प्रतिलेखन...", "a11y_setup": "एक्सेसिबिलिटी की अनुमति दें ताकि श्रुतलेख आपके लिए टाइप कर सके", "pasted": "चिपकाया गया", + "inserted": "दर्ज किया गया", + "copied": "क्लिपबोर्ड पर कॉपी किया गया", "no_speech": "कोई भाषण नहीं मिला", "mic_denied": "माइक का उपयोग अस्वीकृत", "mic_denied_toast": "माइक्रोफ़ोन पहुंच अस्वीकृत. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "वाक् मॉडल हटाएँ", "engine_unavailable": "इस इंस्टाल पर लाइव-डिक्टेशन इंजन उपलब्ध नहीं है। श्रुतलेखन मानक प्रतिलेखन पथ पर वापस आ जाता है।", "model_desc": { - "sherpa-parakeet-tdt-v3": "अनुशंसित. 25 यूरोपीय भाषाओं में तेज़, सटीक श्रुतलेख।", + "sherpa-parakeet-tdt-v3": "25 यूरोपीय भाषाओं में तेज़ और सटीक श्रुतलेख।", "sherpa-parakeet-tdt-v2": "तेज़, सटीक केवल अंग्रेज़ी श्रुतलेख।", "sherpa-zipformer-bilingual-zh-en": "लाइव आंशिक भाग के साथ चीनी + अंग्रेजी स्ट्रीमिंग।", "sherpa-paraformer-bilingual-zh-en": "स्ट्रीमिंग चीनी + अंग्रेजी, कॉम्पैक्ट मॉडल।", "sherpa-zipformer-en-20m": "छोटा स्ट्रीमिंग अंग्रेजी मॉडल - सबसे कम विलंबता।", "sherpa-zipformer-zh-14m": "छोटा स्ट्रीमिंग चीनी मॉडल - सबसे कम विलंबता।", - "sherpa-whisper-tiny": "ऑटो भाषा पहचान के साथ बहुभाषी (90+ भाषाएँ)।" + "sherpa-whisper-tiny": "अनुशंसित। स्वचालित भाषा पहचान के साथ 90 से अधिक भाषाओं में बहुभाषी श्रुतलेख।" } }, "profiles": { diff --git a/frontend/src/i18n/locales/id.json b/frontend/src/i18n/locales/id.json index 5643a928b..e399b28f9 100644 --- a/frontend/src/i18n/locales/id.json +++ b/frontend/src/i18n/locales/id.json @@ -785,6 +785,8 @@ "transcribing_label": "Mentranskripsikan…", "a11y_setup": "Izinkan Aksesibilitas agar dikte dapat mengetik untuk Anda", "pasted": "Ditempel", + "inserted": "Dimasukkan", + "copied": "Disalin ke papan klip", "no_speech": "Tidak ada ucapan yang terdeteksi", "mic_denied": "Akses mikrofon ditolak", "mic_denied_toast": "Akses mikrofon ditolak. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Hapus model ucapan", "engine_unavailable": "Mesin pendiktean langsung tidak tersedia pada instalasi ini. Dikte kembali ke jalur transkripsi standar.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Direkomendasikan. Dikte yang cepat dan akurat dalam 25 bahasa Eropa.", + "sherpa-parakeet-tdt-v3": "Dikte cepat dan akurat dalam 25 bahasa Eropa.", "sherpa-parakeet-tdt-v2": "Dikte khusus bahasa Inggris yang cepat dan akurat.", "sherpa-zipformer-bilingual-zh-en": "Streaming bahasa Mandarin + Inggris dengan siaran langsung sebagian.", "sherpa-paraformer-bilingual-zh-en": "Streaming bahasa Mandarin + Inggris, model ringkas.", "sherpa-zipformer-en-20m": "Model streaming bahasa Inggris yang kecil — latensi terendah.", "sherpa-zipformer-zh-14m": "Model streaming kecil Tiongkok — latensi terendah.", - "sherpa-whisper-tiny": "Multibahasa (90+ bahasa) dengan deteksi bahasa otomatis." + "sherpa-whisper-tiny": "Direkomendasikan. Dikte multibahasa dalam lebih dari 90 bahasa dengan deteksi bahasa otomatis." } }, "profiles": { diff --git a/frontend/src/i18n/locales/it.json b/frontend/src/i18n/locales/it.json index a21fcd67f..954005dbd 100644 --- a/frontend/src/i18n/locales/it.json +++ b/frontend/src/i18n/locales/it.json @@ -785,6 +785,8 @@ "transcribing_label": "Trascrizione…", "a11y_setup": "Consenti Accessibilità così la dettatura può digitare per te", "pasted": "Incollato", + "inserted": "Inserito", + "copied": "Copiato negli appunti", "no_speech": "Nessun parlato rilevato", "mic_denied": "Accesso al microfono negato", "mic_denied_toast": "Accesso al microfono negato. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Elimina modello vocale", "engine_unavailable": "Il motore di dettatura dal vivo non è disponibile su questa installazione. La dettatura ritorna al percorso di trascrizione standard.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Consigliato. Dettatura rapida e accurata in 25 lingue europee.", + "sherpa-parakeet-tdt-v3": "Dettatura rapida e accurata in 25 lingue europee.", "sherpa-parakeet-tdt-v2": "Dettatura veloce e accurata solo in inglese.", "sherpa-zipformer-bilingual-zh-en": "Streaming cinese + inglese con parziali dal vivo.", "sherpa-paraformer-bilingual-zh-en": "Streaming cinese + inglese, modello compatto.", "sherpa-zipformer-en-20m": "Piccolo modello inglese di streaming: latenza più bassa.", "sherpa-zipformer-zh-14m": "Piccolo modello cinese in streaming: latenza più bassa.", - "sherpa-whisper-tiny": "Multilingue (oltre 90 lingue) con rilevamento automatico della lingua." + "sherpa-whisper-tiny": "Consigliato. Dettatura multilingue in oltre 90 lingue con rilevamento automatico della lingua." } }, "profiles": { diff --git a/frontend/src/i18n/locales/ja.json b/frontend/src/i18n/locales/ja.json index c02aed7fb..cfef231c2 100644 --- a/frontend/src/i18n/locales/ja.json +++ b/frontend/src/i18n/locales/ja.json @@ -785,6 +785,8 @@ "transcribing_label": "文字起こし中…", "a11y_setup": "音声入力が代わりに入力できるよう、アクセシビリティを許可してください", "pasted": "貼り付けた", + "inserted": "入力しました", + "copied": "クリップボードにコピーしました", "no_speech": "音声が検出されませんでした", "mic_denied": "マイクアクセスが拒否されました", "mic_denied_toast": "マイクへのアクセスが拒否されました。 {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "音声モデルの削除", "engine_unavailable": "このインストールではライブディクテーション エンジンは利用できません。ディクテーションは標準の文字起こしパスにフォールバックします。", "model_desc": { - "sherpa-parakeet-tdt-v3": "おすすめです。 25 のヨーロッパ言語にわたる高速かつ正確なディクテーション。", + "sherpa-parakeet-tdt-v3": "25のヨーロッパ言語に対応した高速で正確な音声入力。", "sherpa-parakeet-tdt-v2": "高速かつ正確な英語のみのディクテーション。", "sherpa-zipformer-bilingual-zh-en": "中国語 + 英語のライブ部分をストリーミングします。", "sherpa-paraformer-bilingual-zh-en": "中国語+英語ストリーミング、コンパクトモデル。", "sherpa-zipformer-en-20m": "小さなストリーミング英語モデル — レイテンシが最も低い。", "sherpa-zipformer-zh-14m": "小さなストリーミング中国語モデル - 遅延が最も低い。", - "sherpa-whisper-tiny": "自動言語検出機能を備えた多言語 (90 以上の言語)。" + "sherpa-whisper-tiny": "おすすめ。90以上の言語に対応し、言語を自動検出する多言語音声入力。" } }, "profiles": { diff --git a/frontend/src/i18n/locales/ko.json b/frontend/src/i18n/locales/ko.json index 37ad3259d..4c8648f72 100644 --- a/frontend/src/i18n/locales/ko.json +++ b/frontend/src/i18n/locales/ko.json @@ -785,6 +785,8 @@ "transcribing_label": "스크립트 작성 중…", "a11y_setup": "받아쓰기가 대신 입력할 수 있도록 손쉬운 사용을 허용하세요", "pasted": "붙여넣음", + "inserted": "입력됨", + "copied": "클립보드에 복사됨", "no_speech": "음성이 감지되지 않았습니다.", "mic_denied": "마이크 액세스가 거부되었습니다.", "mic_denied_toast": "마이크 액세스가 거부되었습니다. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "음성 모델 삭제", "engine_unavailable": "The live-dictation engine isn't available on this install. Dictation falls back to the standard transcription path.", "model_desc": { - "sherpa-parakeet-tdt-v3": "추천합니다. Fast, accurate dictation across 25 European languages.", + "sherpa-parakeet-tdt-v3": "25개 유럽 언어를 지원하는 빠르고 정확한 받아쓰기입니다.", "sherpa-parakeet-tdt-v2": "빠르고 정확한 영어 전용 받아쓰기.", "sherpa-zipformer-bilingual-zh-en": "라이브 부분으로 중국어 + 영어 스트리밍.", "sherpa-paraformer-bilingual-zh-en": "중국어 + 영어 스트리밍, 컴팩트 모델.", "sherpa-zipformer-en-20m": "작은 스트리밍 영어 모델 - 지연 시간이 가장 낮습니다.", "sherpa-zipformer-zh-14m": "작은 스트리밍 중국 모델 - 지연 시간이 가장 낮습니다.", - "sherpa-whisper-tiny": "Multilingual (90+ languages) with auto language detection." + "sherpa-whisper-tiny": "추천. 90개 이상의 언어를 지원하며 언어를 자동으로 감지하는 다국어 받아쓰기입니다." } }, "profiles": { diff --git a/frontend/src/i18n/locales/nl.json b/frontend/src/i18n/locales/nl.json index 1774df40a..7efc5c2d1 100644 --- a/frontend/src/i18n/locales/nl.json +++ b/frontend/src/i18n/locales/nl.json @@ -350,7 +350,7 @@ "lines_one": "{{count}} regel", "lines_other": "{{count}} regels", "copy": "Kopiëren", - "copied": "Gekonieerd!", + "copied": "Gekopieerd!", "waiting_output": "Wachten op uitvoer…", "auto_detect": "Automatisch detecteren", "suggest_lang": "Overschakelen naar het Nederlands?", @@ -785,6 +785,8 @@ "transcribing_label": "Transcriberen…", "a11y_setup": "Sta Toegankelijkheid toe zodat dicteren voor je kan typen", "pasted": "Geplakt", + "inserted": "Ingevoegd", + "copied": "Gekopieerd naar klembord", "no_speech": "Geen spraak gedetecteerd", "mic_denied": "Microfoontoegang geweigerd", "mic_denied_toast": "Microfoontoegang geweigerd. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Spraakmodel verwijderen", "engine_unavailable": "De live-dicteerengine is niet beschikbaar bij deze installatie. Het dicteren valt terug op het standaard transcriptiepad.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Aanbevolen. Snel en nauwkeurig dicteren in 25 Europese talen.", + "sherpa-parakeet-tdt-v3": "Snel en nauwkeurig dicteren in 25 Europese talen.", "sherpa-parakeet-tdt-v2": "Snel, nauwkeurig dicteren in het Engels.", - "sherpa-zipformer-bilingual-zh-en": "Streaming Chinees + Engels met live gedeeltelijke beelden.", + "sherpa-zipformer-bilingual-zh-en": "Streaming Chinees + Engels met live tussentijdse resultaten.", "sherpa-paraformer-bilingual-zh-en": "Streaming Chinees + Engels, compact model.", "sherpa-zipformer-en-20m": "Klein streaming Engels model - laagste latentie.", "sherpa-zipformer-zh-14m": "Klein Chinees streamingmodel – laagste latentie.", - "sherpa-whisper-tiny": "Meertalig (90+ talen) met automatische taaldetectie." + "sherpa-whisper-tiny": "Aanbevolen. Meertalig dicteren in meer dan 90 talen met automatische taaldetectie." } }, "profiles": { diff --git a/frontend/src/i18n/locales/pl.json b/frontend/src/i18n/locales/pl.json index d0300a423..9d54902d0 100644 --- a/frontend/src/i18n/locales/pl.json +++ b/frontend/src/i18n/locales/pl.json @@ -785,6 +785,8 @@ "transcribing_label": "Transkrypcja…", "a11y_setup": "Zezwól na Dostępność, aby dyktowanie mogło pisać za Ciebie", "pasted": "Wklejony", + "inserted": "Wstawiono", + "copied": "Skopiowano do schowka", "no_speech": "Nie wykryto mowy", "mic_denied": "Odmowa dostępu do mikrofonu", "mic_denied_toast": "Odmowa dostępu do mikrofonu. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Usuń model mowy", "engine_unavailable": "Mechanizm dyktowania na żywo nie jest dostępny w tej instalacji. Dyktowanie wraca do standardowej ścieżki transkrypcji.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Zalecane. Szybkie i dokładne dyktowanie w 25 językach europejskich.", + "sherpa-parakeet-tdt-v3": "Szybkie i dokładne dyktowanie w 25 językach europejskich.", "sherpa-parakeet-tdt-v2": "Szybkie i dokładne dyktowanie wyłącznie w języku angielskim.", "sherpa-zipformer-bilingual-zh-en": "Transmisja strumieniowa języka chińskiego i angielskiego z fragmentami na żywo.", "sherpa-paraformer-bilingual-zh-en": "Przesyłanie strumieniowe w języku chińskim i angielskim, model kompaktowy.", "sherpa-zipformer-en-20m": "Mały, angielski model przesyłania strumieniowego — najniższe opóźnienie.", "sherpa-zipformer-zh-14m": "Mały chiński model do przesyłania strumieniowego — najniższe opóźnienie.", - "sherpa-whisper-tiny": "Wielojęzyczny (ponad 90 języków) z automatycznym wykrywaniem języka." + "sherpa-whisper-tiny": "Zalecane. Wielojęzyczne dyktowanie w ponad 90 językach z automatycznym wykrywaniem języka." } }, "profiles": { diff --git a/frontend/src/i18n/locales/pt.json b/frontend/src/i18n/locales/pt.json index ca15fed17..e5229e113 100644 --- a/frontend/src/i18n/locales/pt.json +++ b/frontend/src/i18n/locales/pt.json @@ -785,6 +785,8 @@ "transcribing_label": "Transcrevendo…", "a11y_setup": "Permita Acessibilidade para que o ditado possa digitar por você", "pasted": "Colado", + "inserted": "Inserido", + "copied": "Copiado para a área de transferência", "no_speech": "Nenhuma fala detectada", "mic_denied": "Acesso ao microfone negado", "mic_denied_toast": "Acesso ao microfone negado. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Excluir modelo de fala", "engine_unavailable": "O mecanismo de ditado ao vivo não está disponível nesta instalação. O ditado volta ao caminho de transcrição padrão.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Recomendado. Ditado rápido e preciso em 25 idiomas europeus.", + "sherpa-parakeet-tdt-v3": "Ditado rápido e preciso em 25 idiomas europeus.", "sherpa-parakeet-tdt-v2": "Ditado rápido e preciso somente em inglês.", "sherpa-zipformer-bilingual-zh-en": "Streaming de chinês + inglês com parciais ao vivo.", "sherpa-paraformer-bilingual-zh-en": "Streaming Chinês + Inglês, modelo compacto.", "sherpa-zipformer-en-20m": "Modelo inglês de streaming minúsculo – latência mais baixa.", "sherpa-zipformer-zh-14m": "Modelo chinês de streaming minúsculo – latência mais baixa.", - "sherpa-whisper-tiny": "Multilíngue (mais de 90 idiomas) com detecção automática de idioma." + "sherpa-whisper-tiny": "Recomendado. Ditado multilíngue em mais de 90 idiomas com detecção automática de idioma." } }, "profiles": { diff --git a/frontend/src/i18n/locales/ru.json b/frontend/src/i18n/locales/ru.json index 96d89a128..2c4f4a2a4 100644 --- a/frontend/src/i18n/locales/ru.json +++ b/frontend/src/i18n/locales/ru.json @@ -785,6 +785,8 @@ "transcribing_label": "Расшифровка…", "a11y_setup": "Разрешите Универсальный доступ, чтобы диктовка могла печатать за вас", "pasted": "Вставлено", + "inserted": "Введено", + "copied": "Скопировано в буфер обмена", "no_speech": "Речь не обнаружена", "mic_denied": "Доступ к микрофону запрещен", "mic_denied_toast": "Доступ к микрофону запрещен. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Удалить речевую модель", "engine_unavailable": "Механизм живой диктовки недоступен в этой установке. Диктовка возвращается к стандартному пути транскрипции.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Рекомендуется. Быстрая и точная диктовка на 25 европейских языках.", + "sherpa-parakeet-tdt-v3": "Быстрая и точная диктовка на 25 европейских языках.", "sherpa-parakeet-tdt-v2": "Быстрый и точный диктант только на английском языке.", "sherpa-zipformer-bilingual-zh-en": "Потоковое вещание на китайском и английском языках с живыми фрагментами.", "sherpa-paraformer-bilingual-zh-en": "Потоковое вещание на китайском и английском языках, компактная модель.", "sherpa-zipformer-en-20m": "Миниатюрная потоковая английская модель — минимальная задержка.", "sherpa-zipformer-zh-14m": "Миниатюрная потоковая китайская модель — самая низкая задержка.", - "sherpa-whisper-tiny": "Многоязычный (более 90 языков) с автоматическим определением языка." + "sherpa-whisper-tiny": "Рекомендуется. Многоязычная диктовка на более чем 90 языках с автоматическим определением языка." } }, "profiles": { diff --git a/frontend/src/i18n/locales/sv.json b/frontend/src/i18n/locales/sv.json index 045aa91f3..87ba725d6 100644 --- a/frontend/src/i18n/locales/sv.json +++ b/frontend/src/i18n/locales/sv.json @@ -785,6 +785,8 @@ "transcribing_label": "Transkriberar...", "a11y_setup": "Tillåt Hjälpmedel så att diktering kan skriva åt dig", "pasted": "Klistras in", + "inserted": "Infogat", + "copied": "Kopierat till urklipp", "no_speech": "Inget tal upptäckt", "mic_denied": "Mikrofonåtkomst nekad", "mic_denied_toast": "Mikrofonåtkomst nekad. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Ta bort talmodell", "engine_unavailable": "Live-dikteringsmotorn är inte tillgänglig på den här installationen. Diktering faller tillbaka till standardtranskriptionsvägen.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Rekommenderas. Snabb, exakt diktering på 25 europeiska språk.", + "sherpa-parakeet-tdt-v3": "Snabb och exakt diktering på 25 europeiska språk.", "sherpa-parakeet-tdt-v2": "Snabb, exakt diktering endast på engelska.", "sherpa-zipformer-bilingual-zh-en": "Strömmande kinesiska + engelska med livepartialer.", "sherpa-paraformer-bilingual-zh-en": "Streaming kinesiska + engelska, kompakt modell.", "sherpa-zipformer-en-20m": "Liten strömmande engelsk modell — lägsta latens.", "sherpa-zipformer-zh-14m": "Liten strömmande kinesisk modell — lägsta latens.", - "sherpa-whisper-tiny": "Flerspråkig (90+ språk) med automatisk språkdetektering." + "sherpa-whisper-tiny": "Rekommenderas. Flerspråkig diktering på över 90 språk med automatisk språkidentifiering." } }, "profiles": { diff --git a/frontend/src/i18n/locales/th.json b/frontend/src/i18n/locales/th.json index 4938b9554..642d7a4ff 100644 --- a/frontend/src/i18n/locales/th.json +++ b/frontend/src/i18n/locales/th.json @@ -785,6 +785,8 @@ "transcribing_label": "กำลังถอดเสียง...", "a11y_setup": "อนุญาตการช่วยการเข้าถึงเพื่อให้การป้อนตามคำบอกพิมพ์แทนคุณได้", "pasted": "วางแล้ว", + "inserted": "แทรกแล้ว", + "copied": "คัดลอกไปยังคลิปบอร์ดแล้ว", "no_speech": "ไม่พบคำพูด", "mic_denied": "การเข้าถึงไมค์ถูกปฏิเสธ", "mic_denied_toast": "การเข้าถึงไมโครโฟนถูกปฏิเสธ {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "ลบโมเดลคำพูด", "engine_unavailable": "กลไกการเขียนตามคำบอกสดไม่พร้อมใช้งานในการติดตั้งนี้ การเขียนตามคำบอกจะกลับไปใช้เส้นทางการถอดเสียงมาตรฐาน", "model_desc": { - "sherpa-parakeet-tdt-v3": "แนะนำ. การเขียนตามคำบอกที่รวดเร็วและแม่นยำใน 25 ภาษายุโรป", + "sherpa-parakeet-tdt-v3": "การเขียนตามคำบอกที่รวดเร็วและแม่นยำใน 25 ภาษายุโรป", "sherpa-parakeet-tdt-v2": "การเขียนตามคำบอกภาษาอังกฤษเท่านั้นที่รวดเร็วและแม่นยำ", "sherpa-zipformer-bilingual-zh-en": "สตรีมมิ่งภาษาจีน + อังกฤษพร้อมถ่ายทอดสดบางส่วน", "sherpa-paraformer-bilingual-zh-en": "สตรีมมิ่งภาษาจีน+อังกฤษ รุ่นกะทัดรัด", "sherpa-zipformer-en-20m": "โมเดลสตรีมมิ่งภาษาอังกฤษขนาดเล็ก — เวลาแฝงต่ำที่สุด", "sherpa-zipformer-zh-14m": "โมเดลสตรีมมิ่งจีนขนาดเล็ก — เวลาแฝงต่ำที่สุด", - "sherpa-whisper-tiny": "หลายภาษา (90+ ภาษา) พร้อมการตรวจจับภาษาอัตโนมัติ" + "sherpa-whisper-tiny": "แนะนำ การเขียนตามคำบอกหลายภาษากว่า 90 ภาษา พร้อมการตรวจจับภาษาอัตโนมัติ" } }, "profiles": { diff --git a/frontend/src/i18n/locales/tr.json b/frontend/src/i18n/locales/tr.json index f4a421cb4..feda980bb 100644 --- a/frontend/src/i18n/locales/tr.json +++ b/frontend/src/i18n/locales/tr.json @@ -785,6 +785,8 @@ "transcribing_label": "Metne dönüştürülüyor…", "a11y_setup": "Dikte sizin yerinize yazabilsin diye Erişilebilirlik'e izin verin", "pasted": "Yapıştırıldı", + "inserted": "Eklendi", + "copied": "Panoya kopyalandı", "no_speech": "Konuşma algılanmadı", "mic_denied": "Mikrofon erişimi reddedildi", "mic_denied_toast": "Mikrofon erişimi reddedildi. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Konuşma modelini sil", "engine_unavailable": "Canlı dikte motoru bu kurulumda mevcut değil. Dikte, standart transkripsiyon yoluna geri döner.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Tavsiye edilir. 25 Avrupa dilinde hızlı, doğru dikte.", + "sherpa-parakeet-tdt-v3": "25 Avrupa dilinde hızlı ve doğru dikte.", "sherpa-parakeet-tdt-v2": "Hızlı, doğru yalnızca İngilizce dikte.", "sherpa-zipformer-bilingual-zh-en": "Canlı bölümlerle Çince + İngilizce akışı.", "sherpa-paraformer-bilingual-zh-en": "Çince + İngilizce akışı, kompakt model.", "sherpa-zipformer-en-20m": "Küçük akışlı İngilizce modeli — en düşük gecikme.", "sherpa-zipformer-zh-14m": "Küçük akışlı Çin modeli — en düşük gecikme.", - "sherpa-whisper-tiny": "Otomatik dil algılamalı çok dilli (90'dan fazla dil)." + "sherpa-whisper-tiny": "Tavsiye edilir. Otomatik dil algılama ile 90'dan fazla dilde çok dilli dikte." } }, "profiles": { diff --git a/frontend/src/i18n/locales/uk.json b/frontend/src/i18n/locales/uk.json index 31c6d0136..1781521e9 100644 --- a/frontend/src/i18n/locales/uk.json +++ b/frontend/src/i18n/locales/uk.json @@ -785,6 +785,8 @@ "transcribing_label": "Транскрибування…", "a11y_setup": "Дозвольте Доступність, щоб диктування могло друкувати за вас", "pasted": "Вставив", + "inserted": "Введено", + "copied": "Скопійовано в буфер обміну", "no_speech": "Мовлення не виявлено", "mic_denied": "Доступ до мікрофона заборонено", "mic_denied_toast": "Доступ до мікрофона заборонено. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Видалити модель мовлення", "engine_unavailable": "Система живого диктування недоступна під час цієї інсталяції. Диктування повертається до стандартного шляху транскрипції.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Рекомендовано. Швидкий і точний диктант 25 європейськими мовами.", + "sherpa-parakeet-tdt-v3": "Швидке й точне диктування 25 європейськими мовами.", "sherpa-parakeet-tdt-v2": "Швидкий і точний диктант лише англійською мовою.", "sherpa-zipformer-bilingual-zh-en": "Потокова трансляція китайською та англійською мовами з живими частками.", "sherpa-paraformer-bilingual-zh-en": "Потоковий китайський + англійський, компактна модель.", "sherpa-zipformer-en-20m": "Маленька потокова англійська модель — найменша затримка.", "sherpa-zipformer-zh-14m": "Маленька потокова китайська модель — найменша затримка.", - "sherpa-whisper-tiny": "Багатомовний (90+ мов) з автоматичним визначенням мови." + "sherpa-whisper-tiny": "Рекомендовано. Багатомовне диктування понад 90 мовами з автоматичним визначенням мови." } }, "profiles": { diff --git a/frontend/src/i18n/locales/vi.json b/frontend/src/i18n/locales/vi.json index 5fde3a330..94a0baf22 100644 --- a/frontend/src/i18n/locales/vi.json +++ b/frontend/src/i18n/locales/vi.json @@ -785,6 +785,8 @@ "transcribing_label": "Phiên âm…", "a11y_setup": "Cho phép Trợ năng để đọc chính tả có thể gõ thay bạn", "pasted": "Đã dán", + "inserted": "Đã nhập", + "copied": "Đã sao chép vào bảng nhớ tạm", "no_speech": "Không phát hiện thấy giọng nói nào", "mic_denied": "Quyền truy cập micrô bị từ chối", "mic_denied_toast": "Quyền truy cập micrô bị từ chối. {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "Xóa mẫu giọng nói", "engine_unavailable": "Công cụ đọc chính tả trực tiếp không khả dụng trên bản cài đặt này. Đọc chính tả quay trở lại đường dẫn phiên âm tiêu chuẩn.", "model_desc": { - "sherpa-parakeet-tdt-v3": "Khuyến nghị. Đọc chính tả nhanh, chính xác trên 25 ngôn ngữ Châu Âu.", + "sherpa-parakeet-tdt-v3": "Đọc chính tả nhanh, chính xác bằng 25 ngôn ngữ châu Âu.", "sherpa-parakeet-tdt-v2": "Đọc chính tả chỉ bằng tiếng Anh nhanh chóng, chính xác.", "sherpa-zipformer-bilingual-zh-en": "Truyền phát tiếng Trung + tiếng Anh với các phần trực tiếp.", "sherpa-paraformer-bilingual-zh-en": "Truyền phát tiếng Trung + tiếng Anh, model nhỏ gọn.", "sherpa-zipformer-en-20m": "Mô hình phát trực tuyến nhỏ bằng tiếng Anh — độ trễ thấp nhất.", "sherpa-zipformer-zh-14m": "Mô hình phát trực tuyến nhỏ của Trung Quốc — độ trễ thấp nhất.", - "sherpa-whisper-tiny": "Đa ngôn ngữ (hơn 90 ngôn ngữ) với tính năng tự động phát hiện ngôn ngữ." + "sherpa-whisper-tiny": "Khuyến nghị. Đọc chính tả đa ngôn ngữ với hơn 90 ngôn ngữ và tự động phát hiện ngôn ngữ." } }, "profiles": { diff --git a/frontend/src/i18n/locales/zh-CN.json b/frontend/src/i18n/locales/zh-CN.json index e5845b07c..f2b3b857e 100644 --- a/frontend/src/i18n/locales/zh-CN.json +++ b/frontend/src/i18n/locales/zh-CN.json @@ -744,6 +744,8 @@ "transcribing_label": "正在抄写…", "a11y_setup": "允许辅助功能,以便听写为您输入文字", "pasted": "粘贴的", + "inserted": "已输入", + "copied": "已复制到剪贴板", "no_speech": "未检测到语音", "mic_denied": "麦克风访问被拒绝", "mic_denied_toast": "麦克风访问被拒绝。 {{hint}}", @@ -2477,13 +2479,13 @@ "delete_confirm_title": "删除语音模型", "engine_unavailable": "实时听写引擎在此安装中不可用。听写将回退到标准转录流程。", "model_desc": { - "sherpa-parakeet-tdt-v3": "推荐。快速、准确地听写 25 种欧洲语言。", + "sherpa-parakeet-tdt-v3": "支持 25 种欧洲语言的快速、准确听写。", "sherpa-parakeet-tdt-v2": "快速、准确的纯英语听写。", "sherpa-zipformer-bilingual-zh-en": "流式中英,带实时部分结果。", "sherpa-paraformer-bilingual-zh-en": "流式中英,紧凑型模型。", "sherpa-zipformer-en-20m": "小型流式英文模型 — 最低延迟。", "sherpa-zipformer-zh-14m": "小型流式中文模型——最低延迟。", - "sherpa-whisper-tiny": "具有自动语言检测功能的多语言(90 多种语言)。" + "sherpa-whisper-tiny": "推荐。支持 90 多种语言并可自动检测语言的多语言听写。" } }, "profiles": { diff --git a/frontend/src/i18n/locales/zh-TW.json b/frontend/src/i18n/locales/zh-TW.json index d380c6cf6..ba1141453 100644 --- a/frontend/src/i18n/locales/zh-TW.json +++ b/frontend/src/i18n/locales/zh-TW.json @@ -785,6 +785,8 @@ "transcribing_label": "正在抄寫…", "a11y_setup": "允許輔助使用,讓聽寫功能為您輸入文字", "pasted": "貼上的", + "inserted": "已輸入", + "copied": "已複製到剪貼簿", "no_speech": "未偵測到語音", "mic_denied": "麥克風存取被拒絕", "mic_denied_toast": "麥克風存取被拒絕。 {{hint}}", @@ -2470,13 +2472,13 @@ "delete_confirm_title": "刪除語音模型", "engine_unavailable": "即時聽寫引擎在此安裝中不可用。聽寫回到標準轉錄路徑。", "model_desc": { - "sherpa-parakeet-tdt-v3": "推薦。快速、準確地聽寫 25 種歐洲語言。", + "sherpa-parakeet-tdt-v3": "支援 25 種歐洲語言的快速、準確聽寫。", "sherpa-parakeet-tdt-v2": "快速、準確的純英語聽寫。", "sherpa-zipformer-bilingual-zh-en": "串流中文 + 英文,並附有現場部分內容。", "sherpa-paraformer-bilingual-zh-en": "串流中文+英文,緊湊型。", "sherpa-zipformer-en-20m": "小型串流英文模型 — 最低延遲。", "sherpa-zipformer-zh-14m": "小型串流媒體中國模型—最低延遲。", - "sherpa-whisper-tiny": "具有自動語言偵測功能的多語言(90 多種語言)。" + "sherpa-whisper-tiny": "推薦。支援 90 多種語言並可自動偵測語言的多語言聽寫。" } }, "profiles": { diff --git a/frontend/src/store/prefsSlice.ts b/frontend/src/store/prefsSlice.ts index 966e87d05..091715556 100644 --- a/frontend/src/store/prefsSlice.ts +++ b/frontend/src/store/prefsSlice.ts @@ -18,7 +18,7 @@ type DictationMode = 'toggle' | 'hold'; /** Default sherpa dictation model id — matches the backend * `sherpa_dictation.DEFAULT_MODEL_ID`. Used only as the pre-hydration seed; * the authoritative value comes from `GET /dictation/prefs`. */ -const DEFAULT_DICTATION_MODEL_ID = 'sherpa-parakeet-tdt-v3'; +const DEFAULT_DICTATION_MODEL_ID = 'sherpa-whisper-tiny'; /** * Global UI font. Applied app-wide by overriding the `--font-sans` CSS custom @@ -202,7 +202,7 @@ export interface PrefsSlice { * • dictationMode — 'toggle' (press to start, press to stop) | 'hold' * (record while the key is held). * • dictationModelId — the selected sherpa-onnx model id (e.g. - * 'sherpa-parakeet-tdt-v3'); drives `?model=` on the + * 'sherpa-whisper-tiny'); drives `?model=` on the * live `/ws/transcribe` socket. */ dictationEnabled: boolean; @@ -289,7 +289,7 @@ export const createPrefsSlice: StateCreator = (s autoPlayPreview: true, // Seeds only — overwritten by loadDictationPrefs() on init. The backend - // default is enabled:true / mode:'toggle' / model:Parakeet TDT v3. + // default is enabled:true / mode:'toggle' / model:Whisper Tiny. dictationEnabled: true, dictationMode: 'toggle', dictationModelId: DEFAULT_DICTATION_MODEL_ID, diff --git a/frontend/src/test/CaptureWidgetMicPreflight.test.jsx b/frontend/src/test/CaptureWidgetMicPreflight.test.jsx index 325462890..dd1ddf831 100644 --- a/frontend/src/test/CaptureWidgetMicPreflight.test.jsx +++ b/frontend/src/test/CaptureWidgetMicPreflight.test.jsx @@ -72,14 +72,16 @@ function stubInvoke({ mic = 'granted' } = {}) { if (cmd === 'check_accessibility') return true; if (cmd === 'request_dictation_capture') { const event = payload?.action === 'stop' ? 'tray-dictate-stop' : 'tray-dictate'; - if (eventHandlers[event]) return eventHandlers[event](); - captureState.pending = event; + const eventPayload = + event === 'tray-dictate' ? { payload: { sessionId: 'mic-preflight-session' } } : undefined; + if (eventHandlers[event]) return eventHandlers[event](eventPayload); + captureState.pending = { event, eventPayload }; return undefined; } if (cmd === 'mark_dictation_capture_ready' && captureState.pending) { - const pending = captureState.pending; + const { event, eventPayload } = captureState.pending; captureState.pending = null; - return eventHandlers[pending]?.(); + return eventHandlers[event]?.(eventPayload); } return undefined; }); @@ -152,7 +154,7 @@ describe('CaptureWidget — mic permission pre-flight (Tauri)', () => { }); render(); await waitFor(() => expect(eventHandlers['tray-dictate']).toBeTypeOf('function')); - eventHandlers['tray-dictate'](); + eventHandlers['tray-dictate']({ payload: { sessionId: 'mic-preflight-session' } }); expect(await screen.findByText(/Mic access denied/)).toBeInTheDocument(); expect(gum).not.toHaveBeenCalled(); diff --git a/frontend/src/test/CaptureWidgetSetupRace.test.jsx b/frontend/src/test/CaptureWidgetSetupRace.test.jsx index 5dfb9ad08..c1cfc132a 100644 --- a/frontend/src/test/CaptureWidgetSetupRace.test.jsx +++ b/frontend/src/test/CaptureWidgetSetupRace.test.jsx @@ -8,15 +8,17 @@ */ import React from 'react'; import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; -import { render, screen, fireEvent, waitFor } from '@testing-library/react'; +import { render, screen, waitFor } from '@testing-library/react'; -const { toastMock } = vi.hoisted(() => ({ +const { toastMock, eventHandlers, eventState } = vi.hoisted(() => ({ toastMock: Object.assign(vi.fn(), { error: vi.fn(), success: vi.fn(), dismiss: vi.fn(), loading: vi.fn(), }), + eventHandlers: {}, + eventState: { pendingStart: false }, })); vi.mock('react-hot-toast', () => ({ default: toastMock, toast: toastMock })); @@ -25,7 +27,10 @@ vi.mock('@tauri-apps/api/core', () => ({ invoke: (...args) => invokeMock(...args), })); vi.mock('@tauri-apps/api/event', () => ({ - listen: vi.fn(async () => () => {}), + listen: vi.fn(async (name, handler) => { + eventHandlers[name] = handler; + return () => delete eventHandlers[name]; + }), })); vi.mock('@tauri-apps/api/window', () => ({ getCurrentWindow: () => ({ hide: async () => {} }), @@ -103,7 +108,9 @@ class FakeWS { } function pressShortcut() { - fireEvent.keyDown(window, { code: 'Space', ctrlKey: true, shiftKey: true }); + const handler = eventHandlers['tray-dictate']; + if (handler) handler({ payload: { sessionId: 'setup-race-session' } }); + else eventState.pendingStart = true; } let realWebSocket; @@ -114,8 +121,15 @@ beforeEach(() => { invokeMock.mockImplementation(async (cmd) => { if (cmd === 'check_microphone') return 'granted'; if (cmd === 'check_accessibility') return true; + if (cmd === 'mark_dictation_capture_ready' && eventState.pendingStart) { + eventState.pendingStart = false; + return eventHandlers['tray-dictate']?.({ + payload: { sessionId: 'setup-race-session' }, + }); + } return undefined; }); + eventState.pendingStart = false; FakeWS.instances = []; storeState.dictationModelId = 'sherpa-parakeet-v3'; realWebSocket = globalThis.WebSocket; diff --git a/frontend/src/test/VoicePanel.test.jsx b/frontend/src/test/VoicePanel.test.jsx index f9fab9ff7..b84927487 100644 --- a/frontend/src/test/VoicePanel.test.jsx +++ b/frontend/src/test/VoicePanel.test.jsx @@ -38,25 +38,25 @@ const MODELS = { repo_id: 'org/parakeet-v3', label: 'Parakeet TDT v3', tag: 'offline', - recommended: true, - size_gb: 0.18, + recommended: false, + size_gb: 0.67, languages: '25 European languages', - installed: true, + installed: false, }, { id: 'sherpa-whisper-tiny', repo_id: 'org/whisper-tiny', label: 'Whisper Tiny', tag: 'offline', - recommended: false, - size_gb: 0.116, + recommended: true, + size_gb: 0.104, languages: '90+ languages', - installed: false, + installed: true, }, ], engine_available: true, engine_reason: null, - default_model_id: 'sherpa-parakeet-tdt-v3', + default_model_id: 'sherpa-whisper-tiny', }; function withI18n(node) { @@ -83,7 +83,7 @@ describe('VoicePanel', () => { return Promise.resolve({ enabled: true, mode: 'toggle', - model_id: 'sherpa-parakeet-tdt-v3', + model_id: 'sherpa-whisper-tiny', }); return Promise.resolve({}); }); @@ -91,7 +91,7 @@ describe('VoicePanel', () => { useAppStore.setState({ dictationEnabled: true, dictationMode: 'toggle', - dictationModelId: 'sherpa-parakeet-tdt-v3', + dictationModelId: 'sherpa-whisper-tiny', dictationLoaded: true, }); }); @@ -107,39 +107,41 @@ describe('VoicePanel', () => { expect(screen.getByRole('switch', { name: 'Enable Voice Dictation' })).toBeChecked(); // The dropdown trigger shows the selected model once models load. await waitFor(() => - expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Parakeet TDT v3'), + expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Whisper Tiny'), ); }); it('lists models with badges, size and install/delete affordances when expanded', async () => { render(withI18n()); await waitFor(() => - expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Parakeet TDT v3'), + expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Whisper Tiny'), ); fireEvent.click(screen.getByTestId('dictation-model-trigger')); - const v3 = screen.getByTestId('dictation-model-sherpa-parakeet-tdt-v3').closest('li'); - expect(within(v3).getByText('recommended')).toBeInTheDocument(); - expect(within(v3).getByText('offline')).toBeInTheDocument(); - expect(within(v3).getByText('180 MB')).toBeInTheDocument(); + const whisper = screen.getByTestId('dictation-model-sherpa-whisper-tiny').closest('li'); + expect(within(whisper).getByText('recommended')).toBeInTheDocument(); + expect(within(whisper).getByText('offline')).toBeInTheDocument(); + expect(within(whisper).getByText('104 MB')).toBeInTheDocument(); // Installed → delete affordance. - expect(screen.getByTestId('dictation-delete-sherpa-parakeet-tdt-v3')).toBeInTheDocument(); + expect(screen.getByTestId('dictation-delete-sherpa-whisper-tiny')).toBeInTheDocument(); // Not installed → download affordance. - expect(screen.getByTestId('dictation-install-sherpa-whisper-tiny')).toBeInTheDocument(); + expect(screen.getByTestId('dictation-install-sherpa-parakeet-tdt-v3')).toBeInTheDocument(); }); it('writes the model pref and kicks off install when picking an uninstalled model', async () => { render(withI18n()); await waitFor(() => expect(screen.getByTestId('dictation-model-trigger')).toBeInTheDocument()); fireEvent.click(screen.getByTestId('dictation-model-trigger')); - fireEvent.click(screen.getByTestId('dictation-model-sherpa-whisper-tiny')); + fireEvent.click(screen.getByTestId('dictation-model-sherpa-parakeet-tdt-v3')); // Pref write-through (POST /dictation/prefs with the new model id). await waitFor(() => - expect(apiPost).toHaveBeenCalledWith('/dictation/prefs', { model_id: 'sherpa-whisper-tiny' }), + expect(apiPost).toHaveBeenCalledWith('/dictation/prefs', { + model_id: 'sherpa-parakeet-tdt-v3', + }), ); // Uninstalled → download started via the model-store install mutation. - expect(installMutate).toHaveBeenCalledWith('org/whisper-tiny'); + expect(installMutate).toHaveBeenCalledWith('org/parakeet-v3'); }); it('toggles the enable switch through the store write-through', async () => { diff --git a/frontend/src/test/captureSherpaFinal.test.js b/frontend/src/test/captureSherpaFinal.test.js index 76af653d3..b96bfca4c 100644 --- a/frontend/src/test/captureSherpaFinal.test.js +++ b/frontend/src/test/captureSherpaFinal.test.js @@ -10,6 +10,8 @@ import { describe, it, expect } from 'vitest'; import { isSherpaModel, classifySherpaFinal, + sherpaSummaryTail, + aggregateDeliveryKind, computeTypeDelta, parsePasteError, } from '../components/CaptureWidget'; @@ -26,30 +28,54 @@ describe('isSherpaModel', () => { }); describe('classifySherpaFinal', () => { - it('treats the first non-empty offline final as a new utterance (then close finalises)', () => { - // Offline model (Parakeet v3 default): one final, nothing committed yet. - expect(classifySherpaFinal('hello world', [])).toBe('utterance'); + it('uses the explicit utterance kind even when the text repeats', () => { + expect( + classifySherpaFinal({ type: 'final', final_kind: 'utterance', text: 'yes' }, ['yes']), + ).toBe('utterance'); }); - it('treats a streaming per-utterance final as an utterance', () => { - expect(classifySherpaFinal('second sentence', ['first sentence'])).toBe('utterance'); + it('uses the explicit summary kind when EOF includes an uncommitted tail', () => { + expect( + classifySherpaFinal( + { type: 'final', final_kind: 'summary', text: 'first sentence tail words' }, + ['first sentence'], + ), + ).toBe('summary'); }); - it('detects the EOF summary (text === the committed join)', () => { - const committed = ['first sentence', 'second sentence']; - expect(classifySherpaFinal('first sentence second sentence', committed)).toBe('summary'); + it('finalises on an explicit empty summary', () => { + expect(classifySherpaFinal({ type: 'final', final_kind: 'summary', text: '' }, [])).toBe( + 'terminator', + ); }); - it('detects a single-utterance summary (summary equals the one commit)', () => { - expect(classifySherpaFinal('hello world', ['hello world'])).toBe('summary'); + it('ignores empty utterance and unknown final kinds', () => { + expect(classifySherpaFinal({ final_kind: 'utterance', text: '' }, [])).toBe('ignore'); + expect(classifySherpaFinal({ text: 'legacy guess' }, [])).toBe('ignore'); + }); +}); + +describe('sherpaSummaryTail', () => { + it('returns only text not already committed', () => { + expect(sherpaSummaryTail('First sentence. Tail words.', ['First sentence.'])).toBe( + 'Tail words.', + ); }); - it('finalises on an empty no-speech terminator', () => { - expect(classifySherpaFinal('', [])).toBe('terminator'); + it('returns the whole summary when no utterance was committed', () => { + expect(sherpaSummaryTail('Offline final.', [])).toBe('Offline final.'); }); - it('ignores an empty final once utterances were committed (the summary covers it)', () => { - expect(classifySherpaFinal('', ['something'])).toBe('ignore'); + it('does not duplicate a summary already delivered in full', () => { + expect(sherpaSummaryTail('One. Two.', ['One.', 'Two.'])).toBe(''); + }); +}); + +describe('aggregateDeliveryKind', () => { + it('keeps copied as the truthful session outcome', () => { + expect(aggregateDeliveryKind('pasted', 'inserted')).toBe('inserted'); + expect(aggregateDeliveryKind('inserted', 'copied')).toBe('copied'); + expect(aggregateDeliveryKind('copied', 'inserted')).toBe('copied'); }); }); @@ -155,6 +181,10 @@ describe('parsePasteError', () => { kind: 'paste', message: 'key event failed', }); + expect(parsePasteError('preflight: clipboard-only session')).toEqual({ + kind: 'preflight', + message: 'clipboard-only session', + }); }); it('accepts Error objects (Tauri invoke may reject with either shape)', () => { diff --git a/frontend/src/test/dictationPrefs.test.ts b/frontend/src/test/dictationPrefs.test.ts index a82ed5cad..70f2d6456 100644 --- a/frontend/src/test/dictationPrefs.test.ts +++ b/frontend/src/test/dictationPrefs.test.ts @@ -20,6 +20,7 @@ vi.mock('../api/client', () => ({ })); import { useAppStore } from '../store'; +import { createPrefsSlice } from '../store/prefsSlice'; function flush() { // Let the write-through promise (.then) settle. @@ -27,6 +28,11 @@ function flush() { } describe('dictation prefs store wiring', () => { + it('seeds the cross-platform default before backend hydration', () => { + const slice = createPrefsSlice(vi.fn() as any, vi.fn() as any, {} as any); + expect(slice.dictationModelId).toBe('sherpa-whisper-tiny'); + }); + beforeEach(() => { apiJson.mockReset(); apiPost.mockReset(); @@ -34,7 +40,7 @@ describe('dictation prefs store wiring', () => { useAppStore.setState({ dictationEnabled: true, dictationMode: 'toggle', - dictationModelId: 'sherpa-parakeet-tdt-v3', + dictationModelId: 'sherpa-whisper-tiny', dictationLoaded: false, }); }); diff --git a/frontend/src/utils/aec/micCapture.js b/frontend/src/utils/aec/micCapture.js index f79f40695..174fa329f 100644 --- a/frontend/src/utils/aec/micCapture.js +++ b/frontend/src/utils/aec/micCapture.js @@ -5,6 +5,58 @@ const WORKLET_URL = '/aec-worklet.js'; +// Anti-alias filtering for the decimation below. resampleInterleavedFrame +// picks samples by linear interpolation, which is not a low-pass: taking a +// 48 kHz stream to 16 kHz that way folds everything above 8 kHz back down +// into the speech band as tones that were never spoken, and the ASR is fed +// the result. The browser only hands us 48 kHz when it refuses the requested +// 16 kHz AudioContext — WKWebView does — so this is the normal path there, +// not an edge case. +// +// Three cascaded Butterworth-Q biquads (~36 dB/octave) run in the audio +// graph rather than per frame, so the filter keeps its state across frame +// boundaries instead of restarting 50 times a second. The cutoff sits below +// Nyquist to leave room for the rolloff; speech has little energy up there. +const ANTIALIAS_STAGES = 3; +const ANTIALIAS_CUTOFF_RATIO = 0.4; + +export function buildAntiAliasChain(ctx, targetRate) { + if (typeof ctx.createBiquadFilter !== 'function') return []; + const stages = []; + for (let stage = 0; stage < ANTIALIAS_STAGES; stage += 1) { + const filter = ctx.createBiquadFilter(); + filter.type = 'lowpass'; + filter.frequency.value = ANTIALIAS_CUTOFF_RATIO * targetRate; + filter.Q.value = Math.SQRT1_2; // Butterworth — flat passband, no resonant peak + stages.push(filter); + } + return stages; +} + +export function resampleInterleavedFrame(frame, inputRate, outputRate, channels) { + if (inputRate === outputRate || frame.length === 0) return frame; + + const inputFrames = Math.floor(frame.length / channels); + const outputFrames = Math.max(1, Math.round((inputFrames * outputRate) / inputRate)); + const output = new Float32Array(outputFrames * channels); + const sourceStep = inputRate / outputRate; + + for (let outputIndex = 0; outputIndex < outputFrames; outputIndex += 1) { + const sourcePosition = outputIndex * sourceStep; + const lowerIndex = Math.min(Math.floor(sourcePosition), inputFrames - 1); + const upperIndex = Math.min(lowerIndex + 1, inputFrames - 1); + const mix = sourcePosition - lowerIndex; + + for (let channel = 0; channel < channels; channel += 1) { + const lower = frame[lowerIndex * channels + channel]; + const upper = frame[upperIndex * channels + channel]; + output[outputIndex * channels + channel] = lower + (upper - lower) * mix; + } + } + + return output; +} + /** * Start capturing ``stream`` as Float32 mono frames at ``sampleRate``. * @@ -20,15 +72,37 @@ export async function startMicCapture( ) { const Ctx = window.AudioContext || window.webkitAudioContext; const ctx = new Ctx({ sampleRate }); + if (ctx.state === 'suspended') { + try { + await ctx.resume(); + } catch { + /* reported below — a context that never runs emits no frames at all */ + } + } + if (ctx.state === 'suspended') { + // Failing loudly beats the alternative: a suspended context runs no + // worklet, so the pill sits on "Listening" forever while not one frame + // is captured. The caller can surface this; silence cannot be surfaced. + try { + await ctx.close(); + } catch { + /* ignore */ + } + throw new Error('mic-suspended: the audio context could not be resumed'); + } await ctx.audioWorklet.addModule(WORKLET_URL); const src = ctx.createMediaStreamSource(stream); + const sourceFrameSize = Math.max(1, Math.round((frameSize * ctx.sampleRate) / sampleRate)); const node = new AudioWorkletNode(ctx, 'aec-frame-emitter', { - processorOptions: { frameSize, channels }, + processorOptions: { frameSize: sourceFrameSize, channels }, }); - node.port.onmessage = (e) => onFrame(e.data); - // Mic → worklet only. Deliberately NOT connected to destination: we tap the - // mic, we don't want to play it back through the speakers. - src.connect(node); + node.port.onmessage = (e) => + onFrame(resampleInterleavedFrame(e.data, ctx.sampleRate, sampleRate, channels)); + // Mic → [anti-alias] → worklet. Only when the browser refused the requested + // rate; when it honors it there is no decimation and nothing to filter. + const antiAlias = ctx.sampleRate > sampleRate ? buildAntiAliasChain(ctx, sampleRate) : []; + const chain = [src, ...antiAlias, node]; + for (let i = 0; i < chain.length - 1; i += 1) chain[i].connect(chain[i + 1]); const stop = async function stop() { try { @@ -46,6 +120,13 @@ export async function startMicCapture( } catch { /* ignore */ } + for (const filter of antiAlias) { + try { + filter.disconnect(); + } catch { + /* ignore */ + } + } try { await ctx.close(); } catch { @@ -53,8 +134,8 @@ export async function startMicCapture( } }; // Existing callers use this value as a function. The property lets generic - // PCM/WAV recording encode the frames at the AudioContext's actual rate. - stop.sampleRate = ctx.sampleRate; + // PCM/WAV recording encode the delivered frames at their actual rate. + stop.sampleRate = sampleRate; stop.channels = channels; return stop; } diff --git a/frontend/src/utils/aec/micCapture.test.js b/frontend/src/utils/aec/micCapture.test.js new file mode 100644 index 000000000..69d927c41 --- /dev/null +++ b/frontend/src/utils/aec/micCapture.test.js @@ -0,0 +1,169 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { startMicCapture } from './micCapture'; + +let context; +let workletNode; +let contextSampleRate; + +class FakeAudioContext { + constructor() { + context = this; + this.sampleRate = contextSampleRate; + this.state = 'suspended'; + this.audioWorklet = { addModule: vi.fn().mockResolvedValue(undefined) }; + this.resume = vi.fn(async () => { + this.state = 'running'; + }); + this.close = vi.fn().mockResolvedValue(undefined); + this.source = { + connect: vi.fn(), + disconnect: vi.fn(), + }; + this.filters = []; + } + + createMediaStreamSource() { + return this.source; + } + + createBiquadFilter() { + const filter = { + type: '', + frequency: { value: 0 }, + Q: { value: 0 }, + connect: vi.fn(), + disconnect: vi.fn(), + }; + this.filters.push(filter); + return filter; + } +} + +class FakeAudioWorkletNode { + constructor(_context, _name, options) { + workletNode = this; + this.options = options; + this.port = { onmessage: null }; + this.disconnect = vi.fn(); + } +} + +describe('startMicCapture', () => { + beforeEach(() => { + contextSampleRate = 48000; + vi.stubGlobal('AudioContext', FakeAudioContext); + vi.stubGlobal('AudioWorkletNode', FakeAudioWorkletNode); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + context = undefined; + workletNode = undefined; + contextSampleRate = undefined; + }); + + it('resumes a suspended AudioContext before capturing microphone frames', async () => { + const stop = await startMicCapture({}, vi.fn()); + + expect(context.resume).toHaveBeenCalledOnce(); + expect(context.state).toBe('running'); + + await stop(); + }); + + it('delivers fixed-size 16 kHz frames when the AudioContext runs at 48 kHz', async () => { + const frames = []; + const stop = await startMicCapture({}, (frame) => frames.push(frame), { + sampleRate: 16000, + frameSize: 4, + }); + + expect(workletNode.options.processorOptions).toEqual({ frameSize: 12, channels: 1 }); + workletNode.port.onmessage({ + data: new Float32Array([-1, -0.75, -0.5, -0.25, 0, 0.25, 0.5, 0.75, 1, 0.75, 0.5, 0.25]), + }); + + expect(frames).toEqual([new Float32Array([-1, -0.25, 0.5, 0.75])]); + expect(stop.sampleRate).toBe(16000); + + await stop(); + }); + + it('preserves worklet frames when the AudioContext honors the requested rate', async () => { + contextSampleRate = 16000; + const onFrame = vi.fn(); + const stop = await startMicCapture({}, onFrame, { sampleRate: 16000, frameSize: 4 }); + const frame = new Float32Array([-1, -0.25, 0.5, 0.75]); + + workletNode.port.onmessage({ data: frame }); + + expect(onFrame).toHaveBeenCalledOnce(); + expect(onFrame).toHaveBeenCalledWith(frame); + + await stop(); + }); + + it('low-passes before decimating when the browser refuses the requested rate', async () => { + // WKWebView hands back 48 kHz whatever we ask for; interpolating straight + // down to 16 kHz would fold everything above 8 kHz into the speech band. + const stop = await startMicCapture({}, vi.fn(), { sampleRate: 16000, frameSize: 4 }); + + expect(context.filters).toHaveLength(3); + for (const filter of context.filters) { + expect(filter.type).toBe('lowpass'); + expect(filter.frequency.value).toBeLessThan(16000 / 2); // below the fold frequency + expect(filter.Q.value).toBeCloseTo(Math.SQRT1_2, 5); // Butterworth, no resonant peak + } + + // Mic → filters → worklet, in order. + expect(context.source.connect).toHaveBeenCalledWith(context.filters[0]); + expect(context.filters[0].connect).toHaveBeenCalledWith(context.filters[1]); + expect(context.filters[1].connect).toHaveBeenCalledWith(context.filters[2]); + expect(context.filters[2].connect).toHaveBeenCalledWith(workletNode); + + await stop(); + for (const filter of context.filters) expect(filter.disconnect).toHaveBeenCalled(); + }); + + it('skips the filter chain when the AudioContext honors the requested rate', async () => { + contextSampleRate = 16000; + const stop = await startMicCapture({}, vi.fn(), { sampleRate: 16000, frameSize: 4 }); + + // No decimation happens, so there is nothing to anti-alias. + expect(context.filters).toHaveLength(0); + expect(context.source.connect).toHaveBeenCalledWith(workletNode); + + await stop(); + }); + + it('fails loudly when the AudioContext cannot be resumed', async () => { + // A suspended context runs no worklet: every frame is silently lost and + // the pill sits on "Listening" forever. Better to surface it. + const failing = class extends FakeAudioContext { + constructor(...args) { + super(...args); + this.resume = vi.fn(async () => { + throw new Error('user gesture required'); + }); + } + }; + vi.stubGlobal('AudioContext', failing); + + await expect(startMicCapture({}, vi.fn())).rejects.toThrow(/mic-suspended/); + expect(context.close).toHaveBeenCalled(); + }); + + it('fails loudly when resume resolves but the context stays suspended', async () => { + const stuck = class extends FakeAudioContext { + constructor(...args) { + super(...args); + this.resume = vi.fn(async () => {}); // resolves, state never changes + } + }; + vi.stubGlobal('AudioContext', stuck); + + await expect(startMicCapture({}, vi.fn())).rejects.toThrow(/mic-suspended/); + expect(context.close).toHaveBeenCalled(); + }); +}); diff --git a/frontend/src/utils/aec/playbackTap.js b/frontend/src/utils/aec/playbackTap.js index 7d83a7ea2..7160cd337 100644 --- a/frontend/src/utils/aec/playbackTap.js +++ b/frontend/src/utils/aec/playbackTap.js @@ -12,6 +12,7 @@ // the player never silences playback. import { publishFarEnd } from './farEndBus'; +import { buildAntiAliasChain, resampleInterleavedFrame } from './micCapture'; const WORKLET_URL = '/aec-worklet.js'; @@ -44,11 +45,21 @@ export async function attachPlaybackTap(mediaEl, { sampleRate = 16000, frameSize /* gesture may be required; harmless */ } } + const channels = 1; + const sourceFrameSize = Math.max(1, Math.round((frameSize * ctx.sampleRate) / sampleRate)); const node = new AudioWorkletNode(ctx, 'aec-frame-emitter', { - processorOptions: { frameSize }, + processorOptions: { frameSize: sourceFrameSize, channels }, }); - node.port.onmessage = (e) => publishFarEnd(e.data); - src.connect(node); + node.port.onmessage = (e) => + publishFarEnd(resampleInterleavedFrame(e.data, ctx.sampleRate, sampleRate, channels)); + // The far-end reference decimates exactly like the mic path, so it needs the + // same anti-alias low-pass before resampling — an aliased reference makes + // the AEC subtract tones the speaker never played. Filters sit only on the + // tap branch: the src → destination edge stays untouched, so what the user + // hears is unchanged. + const antiAlias = ctx.sampleRate > sampleRate ? buildAntiAliasChain(ctx, sampleRate) : []; + const chain = [src, ...antiAlias, node]; + for (let i = 0; i < chain.length - 1; i += 1) chain[i].connect(chain[i + 1]); return async function detach() { try { @@ -61,6 +72,20 @@ export async function attachPlaybackTap(mediaEl, { sampleRate = 16000, frameSize } catch { /* ignore */ } - // Intentionally leave ctx + src→destination intact (see header note). + for (const filter of antiAlias) { + try { + filter.disconnect(); + } catch { + /* ignore */ + } + } + // Detach the tap edge too: the ctx and src are memoised per element, so a + // filter left hanging off src would accumulate one dead chain per AEC + // toggle. The audible src→destination edge stays (see header note). + try { + src.disconnect(antiAlias[0] ?? node); + } catch { + /* ignore */ + } }; } diff --git a/frontend/src/utils/aec/playbackTap.test.js b/frontend/src/utils/aec/playbackTap.test.js new file mode 100644 index 000000000..5f9e8879d --- /dev/null +++ b/frontend/src/utils/aec/playbackTap.test.js @@ -0,0 +1,116 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +vi.mock('./farEndBus', () => ({ publishFarEnd: vi.fn() })); + +import { publishFarEnd } from './farEndBus'; +import { attachPlaybackTap } from './playbackTap'; + +let context; +let workletNode; +let contextSampleRate; + +class FakeAudioContext { + constructor() { + context = this; + this.sampleRate = contextSampleRate; + this.state = 'running'; + this.destination = {}; + this.audioWorklet = { addModule: vi.fn().mockResolvedValue(undefined) }; + this.source = { connect: vi.fn(), disconnect: vi.fn() }; + this.filters = []; + } + + createMediaElementSource() { + return this.source; + } + + createBiquadFilter() { + const filter = { + type: '', + frequency: { value: 0 }, + Q: { value: 0 }, + connect: vi.fn(), + disconnect: vi.fn(), + }; + this.filters.push(filter); + return filter; + } +} + +class FakeAudioWorkletNode { + constructor(_context, _name, options) { + workletNode = this; + this.options = options; + this.port = { onmessage: null }; + this.disconnect = vi.fn(); + } +} + +describe('attachPlaybackTap', () => { + beforeEach(() => { + contextSampleRate = 48000; + vi.stubGlobal('AudioContext', FakeAudioContext); + vi.stubGlobal('AudioWorkletNode', FakeAudioWorkletNode); + publishFarEnd.mockClear(); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + context = undefined; + workletNode = undefined; + contextSampleRate = undefined; + }); + + it('delivers fixed-size 16 kHz reference frames from a 48 kHz context', async () => { + const detach = await attachPlaybackTap({}, { sampleRate: 16000, frameSize: 4 }); + + expect(workletNode.options.processorOptions).toEqual({ frameSize: 12, channels: 1 }); + workletNode.port.onmessage({ + data: new Float32Array([-1, -0.75, -0.5, -0.25, 0, 0.25, 0.5, 0.75, 1, 0.75, 0.5, 0.25]), + }); + + expect(publishFarEnd).toHaveBeenCalledWith(new Float32Array([-1, -0.25, 0.5, 0.75])); + + await detach(); + expect(workletNode.disconnect).toHaveBeenCalledOnce(); + expect(context.source.connect).toHaveBeenCalledWith(context.destination); + }); + + it('low-passes the far-end reference before decimating (48 kHz context)', async () => { + // The AEC reference decimates exactly like the mic path; without the + // filter, content above 8 kHz folds into the reference and the canceller + // subtracts tones the speaker never played. + const detach = await attachPlaybackTap({}, { sampleRate: 16000, frameSize: 4 }); + + expect(context.filters).toHaveLength(3); + for (const filter of context.filters) { + expect(filter.type).toBe('lowpass'); + expect(filter.frequency.value).toBeLessThan(16000 / 2); + } + // Tap branch only: element → filters → worklet, while the audible + // element → destination edge stays direct. + expect(context.source.connect).toHaveBeenCalledWith(context.destination); + expect(context.source.connect).toHaveBeenCalledWith(context.filters[0]); + expect(context.filters[0].connect).toHaveBeenCalledWith(context.filters[1]); + expect(context.filters[1].connect).toHaveBeenCalledWith(context.filters[2]); + expect(context.filters[2].connect).toHaveBeenCalledWith(workletNode); + + await detach(); + for (const filter of context.filters) expect(filter.disconnect).toHaveBeenCalled(); + // Only the tap edge is removed; the audible src→destination edge stays. + expect(context.source.disconnect).toHaveBeenCalledWith(context.filters[0]); + expect(context.source.disconnect).not.toHaveBeenCalledWith(context.destination); + }); + + it('skips the filter chain when the context honors the requested rate', async () => { + contextSampleRate = 16000; + const detach = await attachPlaybackTap({}, { sampleRate: 16000, frameSize: 4 }); + + expect(context.filters).toHaveLength(0); + expect(context.source.connect).toHaveBeenCalledWith(workletNode); + + await detach(); + expect(context.source.disconnect).toHaveBeenCalledWith(workletNode); + expect(context.source.disconnect).not.toHaveBeenCalledWith(context.destination); + }); +}); diff --git a/tests/test_asr_model_missing.py b/tests/test_asr_model_missing.py index 47b448d79..57d928393 100644 --- a/tests/test_asr_model_missing.py +++ b/tests/test_asr_model_missing.py @@ -19,6 +19,19 @@ from fastapi.testclient import TestClient +@pytest.fixture(autouse=True) +def _clear_installed_repo_memo(): + """_repo_installed memoizes positives module-globally and never + invalidates, and test_installed_positive_is_memoized writes the very repo + the missing-model tests probe — so run order decided what these tests saw + (CodeRabbit on #1610). Deterministic now: empty before, empty after.""" + from services import asr_backend + + asr_backend._INSTALLED_REPO_MEMO.clear() + yield + asr_backend._INSTALLED_REPO_MEMO.clear() + + @pytest.fixture(scope="module") def client(): from main import app @@ -94,8 +107,8 @@ def test_dictation_sherpa_selected_but_not_installed(self): assert payload is not None assert payload["error"] == "asr_model_missing" rec = payload["recommended"] - # The curated sherpa dictation entry, with the dictation_id the client - # needs to also set dictation.model_id so the retry picks it up. + # The explicitly selected sherpa entry, with the dictation_id the + # client needs to set so the retry picks it up. assert rec["repo_id"] == "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" assert rec["dictation_id"] == "sherpa-parakeet-tdt-v3" @@ -109,6 +122,55 @@ def test_dictation_sherpa_installed_is_fine(self): patch.object(sd, "is_installed", return_value=True): assert asr_backend.asr_model_missing_error(purpose="dictation") is None + def test_demoted_sherpa_preflights_capture_fallback(self): + """The next session follows demotion even when `?model=` persists.""" + from api.routers.setup import models as setup_models + from services import asr_backend + from services import sherpa_dictation as sd + + fallback_repo = "Systran/faster-whisper-large-v3" + with patch.object(asr_backend.SherpaDictationBackend, "is_available", + return_value=(True, "ready")), \ + patch.object(sd, "is_demoted", return_value=True), \ + patch.object(sd, "is_installed", return_value=True), \ + patch.object(asr_backend, "_capture_whisper_repo", + return_value=fallback_repo), \ + patch.object(setup_models, "is_cached", return_value=False), \ + patch.object(setup_models, "cache_is_complete", return_value=True): + payload = asr_backend.asr_model_missing_error( + purpose="dictation", + sherpa_model_id="sherpa-parakeet-tdt-v3", + ) + + assert payload is not None + assert payload["missing_repo_id"] == fallback_repo + assert payload["recommended"]["repo_id"] == "csukuangfj/sherpa-onnx-whisper-tiny" + + def test_demoted_default_never_recommends_itself(self): + """Installing the CTA recommendation must escape, not clear, a demotion loop.""" + from api.routers.setup import models as setup_models + from services import asr_backend + from services import sherpa_dictation as sd + + fallback_repo = "Systran/faster-whisper-large-v3" + with patch.object(asr_backend.SherpaDictationBackend, "is_available", + return_value=(True, "ready")), \ + patch.object(sd, "is_demoted", + side_effect=lambda mid: mid == "sherpa-whisper-tiny"), \ + patch.object(asr_backend, "_capture_whisper_repo", + return_value=fallback_repo), \ + patch.object(setup_models, "is_cached", return_value=False), \ + patch.object(setup_models, "cache_is_complete", return_value=True): + payload = asr_backend.asr_model_missing_error( + purpose="dictation", + sherpa_model_id="sherpa-whisper-tiny", + ) + + assert payload is not None + assert payload["missing_repo_id"] == fallback_repo + assert payload["recommended"]["repo_id"] == fallback_repo + assert "dictation_id" not in payload["recommended"] + def test_never_raises(self): from services import asr_backend with patch.object(asr_backend, "active_backend_id", @@ -125,6 +187,24 @@ def test_custom_pin_outside_catalog_fails_open(self): patch.dict(os.environ, {"ASR_MODEL_FASTER": "someorg/custom-whisper"}): assert asr_backend.asr_model_missing_error() is None + def test_silent_recovery_requires_even_a_custom_fallback_to_be_installed(self): + """The recovery path never turns fail-open into an implicit download.""" + from api.routers.setup import models as setup_models + from services import asr_backend + + with patch.object(asr_backend, "_capture_whisper_repo", + return_value="someorg/custom-whisper"), \ + patch.object(setup_models, "is_cached", return_value=False), \ + patch.object(setup_models, "cache_is_complete", return_value=False): + payload = asr_backend.asr_model_missing_error( + purpose="dictation", + skip_sherpa=True, + require_installed=True, + ) + + assert payload is not None + assert payload["missing_repo_id"] == "someorg/custom-whisper" + def test_pytorch_whisper_default_repo_fails_open(self): """openai/whisper-large-v3-turbo (the pytorch-whisper default) is not a catalog entry — the preflight stays out of the way (auto-download, @@ -168,7 +248,7 @@ def test_sherpa_offline_backend_maps_to_configured_model(self): payload = asr_backend.asr_model_missing_error() assert payload is not None assert payload["missing_repo_id"] == ( - "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" + "csukuangfj/sherpa-onnx-whisper-tiny" ) def test_installed_positive_is_memoized(self): diff --git a/tests/test_capture_ws.py b/tests/test_capture_ws.py index 07f4fe324..c6fb9e48f 100644 --- a/tests/test_capture_ws.py +++ b/tests/test_capture_ws.py @@ -12,6 +12,7 @@ """ import os import time +import types import pytest @@ -67,6 +68,96 @@ def _audio_chunk(n_bytes: int = 20_000) -> bytes: return b"\x00" * n_bytes +def test_select_sherpa_spec_ignores_demoted_query_override(monkeypatch): + """A persisted frontend query must not resurrect a silent recognizer.""" + from api.routers import capture_ws as cw + from services import sherpa_dictation as sd + + model_id = "sherpa-parakeet-tdt-v3" + websocket = types.SimpleNamespace(query_params={"model": model_id}) + monkeypatch.setattr(sd, "is_demoted", lambda mid: mid == model_id) + + assert cw._select_sherpa_spec(websocket) is None + + +def test_demoted_sherpa_query_keeps_pcm_transport_for_legacy_fallback( + client, monkeypatch, +): + """Demotion changes the recognizer, not the bytes already sent by the UI.""" + from api.routers import capture_ws as cw + from services import sherpa_dictation as sd + + model_id = "sherpa-parakeet-tdt-v3" + monkeypatch.setattr(sd, "is_demoted", lambda mid: mid == model_id) + sample_rates = [] + + async def fallback(_chunks, *, pcm_sr=None): + sample_rates.append(pcm_sr) + return { + "text": "legacy fallback heard pcm", + "segments": [], + "language": "en", + "engine": "stub", + } + + monkeypatch.setattr(cw, "_transcribe_buffer_full", fallback) + with client.websocket_connect( + f"/ws/transcribe?model={model_id}&sr=16000" + ) as ws: + ws.send_bytes(_audio_chunk()) + ws.send_text("EOF") + for _ in range(10): + if ws.receive_json().get("type") == "final": + break + + assert sample_rates == [16000] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("result", "expected"), + [ + ({"text": "top-level text"}, "top-level text"), + ( + {"segments": [{"text": "segment one"}, {"text": "segment two"}]}, + "segment one segment two", + ), + ( + {"chunks": [{"text": "chunk one"}, {"text": "chunk two"}]}, + "chunk one chunk two", + ), + ], +) +async def test_partial_text_normalizes_every_asr_result_shape( + monkeypatch, tmp_path, result, expected, +): + """Live partials work for backends that expose only segments/chunks. + + WhisperX, Faster Whisper, Moonshine, and OpenAI-compatible ASR do not add a + top-level ``text`` field. The capture seam must consume the shared ASR + result contract instead of silently dropping their partial transcript. + """ + from api.routers import capture_ws as cw + from services import asr_backend + + wav = tmp_path / "partial.wav" + wav.write_bytes(b"placeholder") + + class StubBackend: + def transcribe(self, _path, *, word_timestamps=False): + assert word_timestamps is False + return result + + async def run_inline(_executor, fn, **_kwargs): + return fn() + + monkeypatch.setattr(cw, "_pcm16_to_wav", lambda _pcm, _sr: str(wav)) + monkeypatch.setattr(asr_backend, "get_capture_asr_backend", lambda: StubBackend()) + monkeypatch.setattr(asr_backend, "run_transcribe_guarded", run_inline) + + assert await cw._transcribe_buffer([b"\x00" * 4000], pcm_sr=16000) == expected + + def test_eof_text_frame_triggers_final_without_disconnect(client): """Client sends audio + 'EOF' text frame, expects `final` over open socket.""" with client.websocket_connect("/ws/transcribe") as ws: diff --git a/tests/test_dictation_model_copy.py b/tests/test_dictation_model_copy.py new file mode 100644 index 000000000..334f4e3b6 --- /dev/null +++ b/tests/test_dictation_model_copy.py @@ -0,0 +1,24 @@ +"""Keep localized dictation recommendation copy aligned with model policy.""" + +from __future__ import annotations + +import json +from pathlib import Path + + +_LOCALES = Path(__file__).resolve().parents[1] / "frontend" / "src" / "i18n" / "locales" + + +def test_recommended_badge_copy_belongs_to_whisper_tiny_in_every_locale(): + natural_prefixes = { + "ja": "おすすめ", + "uk": "рекомендовано", + "vi": "khuyến nghị", + } + for path in sorted(_LOCALES.glob("*.json")): + panel = json.loads(path.read_text(encoding="utf-8"))["voicePanel"] + badge = panel["badge_recommended"].casefold() + recommendation = natural_prefixes.get(path.stem, badge) + descriptions = panel["model_desc"] + assert recommendation not in descriptions["sherpa-parakeet-tdt-v3"].casefold(), path.name + assert recommendation in descriptions["sherpa-whisper-tiny"].casefold(), path.name diff --git a/tests/test_dictation_recovery_tail_bound.py b/tests/test_dictation_recovery_tail_bound.py new file mode 100644 index 000000000..33455836c --- /dev/null +++ b/tests/test_dictation_recovery_tail_bound.py @@ -0,0 +1,147 @@ +"""Silent-model recovery audio must be bounded (#1610 review). + +Both dictation WebSocket paths retained every PCM byte of a session so the +silent-model fallback could re-transcribe it. Nothing capped that: an open mic +at 16 kHz mono int16 added ~115 MB per hour, held for the life of the session +and only ever read when the fallback actually fired. Streaming and offline +both did it. + +The tail is what matters — recovery re-transcribes what the user just said — +while the silent-model gate measures how much audio the session carried, so +the true total is tracked separately and stays truthful after trimming. +""" +from __future__ import annotations + +import importlib + +import pytest + + +@pytest.fixture +def ws(): + return importlib.import_module("api.routers.capture_ws") + + +SR = 16000 +BYTES_PER_S = SR * 2 + + +def test_a_long_session_stops_growing(ws): + tail = ws.RecoveryTail(SR, seconds=2.0) + for _ in range(600): # 60 s of 100 ms frames + tail.extend(b"\x01\x02" * (SR // 10)) + assert len(tail.tail()) == 2 * BYTES_PER_S + + +def test_the_true_total_survives_trimming(ws): + """is_model_silent gates on how much audio the session carried; capping + the buffer must not make a long session look too short to be recoverable.""" + tail = ws.RecoveryTail(SR, seconds=1.0) + for _ in range(30): + tail.extend(b"\x00\x01" * SR) # 1 s each + assert tail.total_bytes == 30 * BYTES_PER_S + assert len(tail.tail()) == BYTES_PER_S + assert ws.is_model_silent("", True, tail.total_bytes) is True + + +def test_the_retained_audio_is_the_most_recent(ws): + """Head-trimming, not head-keeping — the useful speech is the latest.""" + tail = ws.RecoveryTail(SR, seconds=1.0) + tail.extend(b"\xaa\xaa" * SR) # older + tail.extend(b"\xbb\xbb" * SR) # newer + assert tail.tail() == b"\xbb\xbb" * SR + + +def test_a_short_session_is_kept_whole(ws): + tail = ws.RecoveryTail(SR, seconds=120.0) + tail.extend(b"\x01\x02" * SR) + assert tail.tail() == b"\x01\x02" * SR + assert tail.total_bytes == BYTES_PER_S + + +@pytest.mark.parametrize( + ("value", "expected"), + [ + (None, 120.0), ("bad", 120.0), ("nan", 120.0), ("inf", 120.0), + ("-1", 120.0), ("0", 120.0), ("60", 60.0), ("999999", 300.0), + ], +) +def test_recovery_tail_environment_override_is_finite_and_bounded(ws, value, expected): + assert ws._bounded_recovery_tail_seconds(value) == expected + + +@pytest.mark.parametrize("sample_rate,seconds", [(0, 120.0), (16000, 0.0), (-1, -1.0)]) +def test_a_nonsense_bound_still_yields_a_usable_buffer(ws, sample_rate, seconds): + """A bad sr query param or env override must not produce a zero-length + buffer that silently disables recovery.""" + tail = ws.RecoveryTail(sample_rate, seconds=seconds) + tail.extend(b"\x01\x02" * 100) + assert len(tail.tail()) >= 2 + + +@pytest.mark.parametrize("sr", ["1000000000", "4000", "0", "-16000", "junk", ""]) +def test_an_absurd_client_sample_rate_is_not_believed(ws, sr): + """`?sr=` sizes RecoveryTail's byte ceiling (rate × RECOVERY_TAIL_SECONDS), + so an unclamped client value re-opens the unbounded-memory path (#1610 + review). Out-of-range and garbage rates fall back to 16 kHz.""" + assert ws._bounded_sample_rate({"sr": sr}) == 16000 + + +def test_supported_client_sample_rates_pass_through(ws): + for sr in (8000, 16000, 44100, 48000, 96000): + assert ws._bounded_sample_rate({"sr": str(sr)}) == sr + + +def test_the_sherpa_paths_use_the_bounded_rate(ws): + """Structural: both sherpa handlers get their rate from _sherpa_session, + which must parse via the clamped helper — a raw int() of the query param + is exactly the bug.""" + import inspect + + src = inspect.getsource(ws._sherpa_session) + assert "_bounded_sample_rate(" in src + assert 'int(websocket.query_params.get("sr"' not in src + + +def test_both_socket_paths_use_the_bounded_buffer(ws): + """Structural: the streaming path and the offline path both had the leak, + so a fix applied to only one of them is not a fix.""" + import inspect + + src = inspect.getsource(ws) + assert src.count("RecoveryTail(pcm_sr)") == 2 + assert "session_pcm = bytearray()" not in src + + +def test_trimming_never_splits_a_sample(ws): + """int16 mono PCM: transport frames can carry odd byte counts (a sample + split across two WebSocket messages), but the *stream* stays aligned — a + sample starts at every even global offset. Trimming must remove an even + number of bytes so the retained tail still starts on a sample boundary. + + The failure needs a stream that ends mid-sample (the session closed on a + torn frame): with an odd total, an odd-trimming buffer ends with an odd + cumulative removal, the tail starts mid-sample, and every decoded value + is byte-shifted garbage. (An even total self-rebalances across trims, + which is why the obvious version of this test cannot fail.) + """ + import struct + + n = SR * 2 # 2 s of samples; sample k holds the value k + stream = b"".join(struct.pack("= 2, f"silence gate never committed mid-session: {msgs}" assert finals[0]["text"] == "Utterance one." + assert all(m["final_kind"] == "utterance" for m in finals[:-1]) + assert finals[-1]["final_kind"] == "summary" # EOF final = committed pieces + the drained live tail (utterance 2). assert finals[-1]["text"] == "Utterance one. Utterance one." # O(n²) fix: every decode was bounded by ONE utterance window — never a