Skip to content

True incremental streaming + Stop button #2

Description

@ASahu16

Goal: Stream tokens as they are generated, and let the user stop generation
mid-flight, keeping the partial reply.

Why: nativeGenerateStreamInit currently pre-generates all tokens, then
replays them — so time-to-first-token equals full generation time and a Stop
button cannot interrupt. This is purely cosmetic streaming and blocks any
multi-step agent loop (which issues several model calls per turn).

Scope:

  • Native: move generation out of nativeGenerateStreamInit (prompt decode +
    sampler + persistent state only) into per-token nativeGenerateStreamNext,
    carrying the hold-back buffer across calls.
  • Native: g_should_stop becomes std::atomic<bool>, set lock-free by
    nativeStopGeneration, checked each Next.
  • Dart: add Future<void> stop() to InferenceBackend; LlamaCppBackend
    forwards to stopGeneration(), MediaPipeBackend to _chat.stopGeneration().
  • ChatNotifier.stop(); the partial reply is saved via the existing
    onDone path (trace tagged as stopped).
  • UI: Send button becomes a Stop (■) button while streaming.
  • iOS/macOS native mirror ⏸️ (deferred, tracked separately).

Acceptance criteria:

  • On a 3B model, first token renders well before generation completes.
  • Tapping Stop halts within ~one token and keeps the text generated so far.
  • flutter build apk succeeds; Dart suite + analyze green.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions