feat: LiteRT engine consolidation + on-device STT (flutter_gemma_speech, moonshine) - #389
Merged
Conversation
…FFI + engine web JS
Expose LiteRtBindings via a public native-only litertlm/lib/litert_bindings.dart (avoids an implementation_imports lint on embeddings' native-only file), plus trivial prefer_initializing_formals / unnecessary_library_name fixes so flutter_gemma_embeddings analyzes clean again.
…tch + active-model persistence
…ate worker + tokenizer); moonshine profile
…r-path leak, NaN decode diagnostic, hiddenBufPtr leak, unlock status checks
… moonshine pipeline
…core to 1.4.0 (+CHANGELOG); speech requires flutter_gemma ^1.4.0
…repo.ModelType.stt (was silently bucketed into embedding in getInstalledModels/isAnyModelInstalled)
…dow (unconditional in-progress completer guard, mirrors desktop)
…r awaits in stt_screen recording paths
… the consolidated single-hook engine + wired moonshine STT
…port-leak guard, drop dead crypto dep, fix path-import shadow, RuntimeConfig.tokenizerPath doc, stt_screen reentrancy guards
…twork install + bundled clip via public API)
…LAPPDATA × Task.split) On Windows getWriteTargetPath resolves to %LOCALAPPDATA%\flutter_gemma, which is not one of background_downloader's recognized BaseDirectories (path_provider maps applicationSupport -> Roaming, applicationDocuments -> Documents). Task.split therefore falls back to BaseDirectory.root with the drive STRIPPED, and on Windows the root base reconstructs against '' (= $CWD), so a fresh download landed at <cwd>\Users\..\AppData\Local\flutter_gemma\ while getReadTargetPath / validateModelFiles look at the absolute path. install() "succeeded" but isModelInstalled() stayed false, and getActiveModel/getActiveStt threw "model file paths not found". Affects every fresh network download (inference/embedding/STT) since the 0.15.1 LOCALAPPDATA switch (#179); latent because Windows tests use Documents-staged models and no test exercised a real download landing. macOS/Linux are unaffected (applicationSupport is a recognized base). - smart_downloader: resolveDownloadDirectory() hands the DownloadTask the absolute directory for the root-fallback case so the file lands exactly at targetPath; taskId still derives from the split triple, so resume/reclaim identity is unchanged. - network_source_handler: install()/installWithProgress() now fail loudly when no file is present at targetPath after a "successful" download instead of saving a 0-byte "installed" record. - test: download_directory_resolution_test guards the download-landing invariant the path-resolution round-trip test could not. Verified end-to-end: Windows STT green (file at %LOCALAPPDATA%\flutter_gemma, transcript ok), macOS STT green (no regression), core suite 481/481.
…ec/CLAUDE versions New in 1.4: on-device speech-to-text via the new flutter_gemma_speech package (selectable ASR, moonshine today) + the Windows fresh-download fix. - Core README: STT feature bullet, "What's new in 1.4", install snippet + pick-by-need, initialize() param table, and a Speech-to-Text row in the platform feature matrix; modular-packages bullet now lists speech. - CHANGELOG 1.4.0: user-facing STT line + the Windows fresh-download fix. - Website: new /docs/speech guide; sidebar, footer, and features-card entries; installation/packages/models/getting-started/genkit/migration/troubleshooting updated for STT; version refs bumped (flutter_gemma ^1.4.0, litertlm ^1.3.0, embeddings ^1.0.4). - Sync ios/macos flutter_gemma.podspec s.version and CLAUDE.md Current Version to 1.4.0 (+ speech 0.1.0). - Doc audit fixes: CONTRIBUTING SDK floor (Flutter 3.44 / Dart 3.12), native-v0.14.0, FFI-client path; flutter_gemma_embeddings README web-asset path (packages/flutter_gemma_embeddings/web/); flutter_gemma_agent CHANGELOG 8 starter skills; "litertlm 1.2.0+" Windows discrete-GPU regression labels; litertlm README notes it exposes LiteRtBindings for embeddings + speech.
…en download-path tests Follow-up to the Windows download-path fix, from the final PR review: - Extract `assertInstalledFilePresent(sizeBytes, targetPath)` and apply it in NetworkSourceHandler (was inline), AssetSourceHandler, and FileSourceHandler — the same "download/copy succeeded but no usable file landed → silently save a broken install record" defect the network fix closed. BundledSourceHandler is intentionally excluded (a bundled native resource is not a stat-able file, so getFileSize can be 0 legitimately). Message now distinguishes 0-byte/missing. - resolveDownloadDirectory: add an injectable `p.Context` (defaults to the host context) so the Windows landing behaviour is deterministically testable on a POSIX CI host; correct the doc/inline comments — the stored taskId is derived from the split triple regardless, and the reclaim recompute only matters on the Android-only reclaim path (which never takes the root fallback). - Tests: add a host-independent Windows round-trip that reproduces the drive-strip landing bug and FAILS if the fix is reverted; add 0-byte guard tests for the network/asset/file handlers (throws + never persists metadata). Core suite 486/486; analyze clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates the shared LiteRT runtime into
flutter_gemma_litertlm(now the full LiteRT engine) and adds on-device Speech-to-Text via a newflutter_gemma_speechpackage. First STT model: moonshine-tiny (raw PCM, no mel frontend). The model is selectable like inference models (SttModelTypeon the spec) — one generic backend + recognizer driven by a per-model profile.Phase 1 — engine consolidation + core STT contracts
flutter_gemma_litertlmabsorbs the LiteRt interpreter FFI (LiteRtBindings, moved in from embeddings) and the weblitert.js/tensorflow.jsruntime (moved in from core) → it is now the sole owner oflibLiteRtLm; the stale-owner coordination marker is retired.flutter_gemma_embeddingsbecomes thin logic depending onlitertlm(drops its own hook + FFI + engine web JS). Public API (LiteRtEmbeddingBackend) unchanged.flutter_gemma(core) loses the engine web JS → truly engine-free.ModelManagementType.stt,SttModelSpec+SttModelType,SttBackendProvider/SttRegistry,SpeechRecognizer,FlutterGemma.initialize(sttBackends:),installStt()/getActiveStt()+createSttModeldispatch (mobile/web/desktop) + active-model persistence.Phase 2 —
flutter_gemma_speech(STT, moonshine)litertlm(sharedlibLiteRtLm, no own hook).LiteRtSttBackend(sole.tfliteSTT backend;canHandle => true; returns the coreSpeechRecognizerinterface). Concrete pipeline (LiteRtSpeechRecognizer,SttModelProfile,SttWorker,SttCore,HfTokenizer) is internal tolib/src/, not exported.SttModelProfile.forType(sttModelType). moonshine profile filled; whisper/parakeet (log-mel) arms throwUnimplementedError(follow-on).Validation (on-device, macOS)
desktop_embedding_testgreen — embeddings still produce a 768-dim vector through the moved FFI onlibLiteRtLm0.14.0.installStt→getActiveStt→transcribe) transcribes a known clip →"Ever tried ever failed, no matter."Follow-ups (not in this PR)
libLiteRtLmafter a version bump).