Goal: Stream tokens as they are generated, and let the user stop generation
mid-flight, keeping the partial reply.
Why: nativeGenerateStreamInit currently pre-generates all tokens, then
replays them — so time-to-first-token equals full generation time and a Stop
button cannot interrupt. This is purely cosmetic streaming and blocks any
multi-step agent loop (which issues several model calls per turn).
Scope:
Acceptance criteria:
- On a 3B model, first token renders well before generation completes.
- Tapping Stop halts within ~one token and keeps the text generated so far.
flutter build apk succeeds; Dart suite + analyze green.
Goal: Stream tokens as they are generated, and let the user stop generation
mid-flight, keeping the partial reply.
Why:
nativeGenerateStreamInitcurrently pre-generates all tokens, thenreplays them — so time-to-first-token equals full generation time and a Stop
button cannot interrupt. This is purely cosmetic streaming and blocks any
multi-step agent loop (which issues several model calls per turn).
Scope:
nativeGenerateStreamInit(prompt decode +sampler + persistent state only) into per-token
nativeGenerateStreamNext,carrying the hold-back buffer across calls.
g_should_stopbecomesstd::atomic<bool>, set lock-free bynativeStopGeneration, checked eachNext.Future<void> stop()toInferenceBackend;LlamaCppBackendforwards to
stopGeneration(),MediaPipeBackendto_chat.stopGeneration().ChatNotifier.stop(); the partial reply is saved via the existingonDonepath (trace tagged as stopped).Acceptance criteria:
flutter build apksucceeds; Dart suite + analyze green.