Proposal: evolve Windows Voice Assistant from one-shot wake to Talk Mode

[calebeden]

Summary

After #1097 is fixed and #1071 is merged, evolve the Windows Voice Assistant from its initial one-shot wake flow into two explicit experiences:

1. Voice Wake: wait for a configurable phrase, handle one request, speak one response, then return to wake listening.
2. Talk Mode: run an ongoing listen, answer, speak loop until the user stops it.

#1071 should remain the foundation. It already establishes the important Windows-specific building blocks: local Whisper transcription, energy VAD, transcript-based wake matching, exclusive microphone ownership, canonical native Chat routing, exact gateway-run response correlation, configured TTS playback, settings readiness, and lifecycle cleanup.

Its transcript gate should be treated as a prototype and fallback, not the final wake-detection architecture. Generic speech-to-text is not optimized to produce a stable spelling for short names or invented words. A phrase such as "Hey Claw" may be transcribed as "Hey Klaw," "Hey Klaue," or another phonetically similar form, causing an exact text gate to miss a valid activation.

Reliable always-on Voice Wake will probably require a dedicated acoustic wake-word model that detects the target sound pattern before command transcription.

This proposal separates future modes instead of incrementally adding every behavior to the one-shot coordinator.

Entry conditions

• Bug: Voice recording freezes app (UI hang + 100% CPU) when WASAPI capture device delivers no data #1097 is fixed with deterministic no-audio coverage and current-head Windows proof that capture failure leaves the UI responsive.
• feat: add wake phrase voice assistant preview #1071 is merged with its one-shot behavior and run-ownership invariants intact.

Longer-lived capture or continuous conversation should not ship before both conditions are met.

Background

#1071 intentionally implements one narrow flow:

Wake phrase and request in one silence-bounded utterance -> selected native Chat session -> one correlated response -> configured TTS -> resume wake listening.

It does not add a wake-word model, continuous conversation, barge-in, or a new gateway or node protocol.

Research against the macOS implementation in openclaw/openclaw@030dbe9 shows that OpenClaw treats Voice Wake and Talk Mode as separate experiences:

• Voice Wake uses ordinary on-device speech recognition plus deterministic transcript matching, not a dedicated acoustic wake-word model.
• Voice Wake normally handles one command and returns to passive listening.
• Talk Mode runs a continuing listening -> thinking -> speaking -> listening loop.
• Talk Mode can be started manually or entered after a wake phrase.
• Barge-in and speech interruption are Talk Mode concerns, not prerequisites for basic wake activation.

Windows should follow that product separation while retaining its native speech stack. Windows uses local Whisper and energy VAD where macOS uses Apple Speech.

Why transcript matching is not sufficient

The macOS implementation uses deterministic transcript matching, but that does not establish that transcript matching will be reliable enough on Windows. The #1071 prototype exposes a predictable failure mode:

1. Ambient speech is converted into text by a general-purpose Whisper model.
2. VoiceWakeGate requires the configured phrase to match the opening transcript tokens.
3. A phonetically correct but differently spelled transcript does not activate the Assistant.

Adding aliases, edit distance, or phonetic matching could improve recall in the short term, but it also expands the false-activation surface and becomes difficult to reason about across accents and languages. These techniques are useful fallback experiments, not a substitute for acoustic keyword spotting.

A dedicated wake-word detector evaluates audio against a phrase-specific model and emits a confidence score without depending on the transcription spelling. This should improve recognition of names and invented words such as "Claw," but it is not automatic or universal. The model still needs representative positive and negative audio, noise and accent coverage, threshold tuning, packaging, and sustained CPU measurements.

There is also a product tradeoff with #1071's arbitrary one-to-three-word wake phrase setting. Most dedicated detectors support fixed model-backed phrases, generated phrase-specific models, or user enrollment. The first reliable release should therefore do one of the following:

• ship one or more supported model-backed phrases such as "OpenClaw" or "Hey Claw";
• generate or download a model when a custom phrase is configured;
• add local user enrollment for a custom phrase.

Free-form custom phrases that continue using transcript matching should be labeled experimental and should not carry the same reliability claim as model-backed phrases.

Proposed user experience

Experience	Activation	Behavior
Off	Disable Voice Wake and stop Talk Mode	No Assistant-owned microphone capture
Voice Wake	Enable Voice Wake and say the configured phrase	Capture one request, speak one matching response, then return to wake listening
Wake into Talk Mode	Enable Trigger Talk Mode, then say the configured phrase	Enter an ongoing conversation without requiring the phrase for each turn
Manual Talk Mode	Start Talk Mode from native Chat or an always-reachable tray control	Enter the same ongoing conversation without a wake phrase
Push-to-talk	Use the existing push-to-talk interaction	Remain an independent one-request input path

Voice Wake should support both:

• The existing phrase-plus-request form: "OpenClaw, summarize my unread messages."
• A trigger-only form: say "OpenClaw," pause, then speak the request within a bounded command window.

The Windows one-shot behavior should continue speaking its response. This preserves #1071 behavior even though the macOS one-shot forwarder does not itself own speech output.

Recommended product decisions

1. Preserve wake-one-shot compatibility. Existing feat: add wake phrase voice assistant preview #1071 settings should continue to enable Voice Wake without migration or surprise behavior changes.
2. Keep Talk Mode transient. Starting Talk Mode should be a runtime action, not a persisted always-on startup mode.
3. Pin a Talk Mode session. Resolve the visible selected Chat session when Talk Mode starts and keep that session for the conversation. Voice Wake can continue resolving the selected session for each independent turn.
4. Ship sequential Talk Mode first. The first Talk Mode milestone should stop listening while thinking and speaking. Barge-in should be a later, separately proven capability.
5. Treat transcript matching as provisional. Preserve it as a compatibility fallback and test seam, but require model-backed acoustic detection before calling always-on Voice Wake production-ready.
6. Avoid protocol expansion. The initial work should continue through the canonical Chat and TTS paths. It should not add voice.*, MCP, or Windows node commands.

Architecture direction

Do not grow App.xaml.cs into the runtime owner. App should remain the composition root and forward startup, settings, connection, notification, and shutdown events to focused owners.

The future design should preserve these boundaries:

• VoiceService: microphone capture, energy VAD, local Whisper lifecycle, silence-bounded utterances, and capture health.
• Wake detector: continuous local acoustic keyword spotting, confidence policy, supported phrase identity, and activation events. A transcript implementation can remain available as an explicit fallback.
• Wake runtime: wake activation handling, trigger-only command window, and selection of one-shot versus Talk Mode activation.
• Talk runtime: the active conversation loop and explicit stop behavior.
• Shared turn path: canonical Chat send, accepted gateway-run binding, response claiming, exact cancellation, timeout handling, and late-response suppression.
• Speaker: configured Windows, Piper, or ElevenLabs playback behind the existing abstraction.
• Top-level controller: mutual exclusion and transitions between Off, Voice Wake, and Talk Mode. It should not duplicate capture, Chat, or TTS implementation.

While the transcript fallback remains supported, the current VoiceWakeGate boolean extraction result will likely need a richer result that distinguishes:

• no wake match;
• wake phrase only;
• wake phrase with a trailing request.

The exact-run protections from #1071 are invariants, not implementation details to relax. Every Talk Mode turn must own one accepted gateway run, speak at most one matching final response, and cancel only that run.

Proposed delivery phases

Phase 1: prepare mode seams

• Characterize the merged one-shot behavior before refactoring.
• Separate wake activation, turn execution, and post-turn transition policy without changing the user-visible flow.
• Keep App limited to composition and lifecycle forwarding.
• Preserve settings compatibility, localization, accessibility, and current one-shot proof.

Phase 2: add model-backed wake detection

• Define a wake-detector abstraction that receives local microphone audio and reports phrase identity plus confidence.
• Select an engine that supports the required Windows architectures, offline inference, acceptable licensing, and bounded CPU and memory use.
• Ship at least one supported model-backed default phrase.
• Decide whether custom phrases use generated models, local enrollment, or an experimental transcript fallback.
• Keep wake audio local and do not log raw audio or confidence traces that could expose user content.
• Tune activation thresholds against positive speech, conversational negatives, media playback, room noise, accents, and varying microphone quality.
• Make the detection threshold and debounce policy centralized and testable.
• Demonstrate sustained listening without UI degradation, thermal instability, or excessive battery use.

This is the production-readiness boundary for always-on Voice Wake. Transcript-only matching may remain available before this phase, but it should continue to be described as preview behavior with known false-negative and false-positive limitations.

Phase 3: complete Voice Wake

• Use the acoustic activation event to open a bounded command-capture window.
• Continue supporting phrase and request in one utterance when the detector and buffered audio allow the command tail to be recovered safely.
• Add clear states for waiting for wake, waiting for command, dispatching, waiting for reply, and speaking.
• Return to passive wake listening after success, timeout, cancellation, or a recoverable failure.
• Keep microphone ownership and capture-health diagnostics visible to native Chat.

Initial timing should follow the researched behavior unless Windows proof indicates otherwise:

• approximately 2 seconds of silence to finish command speech;
• approximately 5 seconds to begin a command after a trigger-only wake;
• a bounded hard limit for command capture;
• a short debounce before passive listening resumes.

These should be centralized runtime policy values rather than scattered UI constants.

Phase 4: manual Talk Mode

• Add explicit Start Talk Mode and Stop Talk Mode actions.
• Pin the selected native Chat session when Talk Mode starts.
• Implement the sequential loop: listen, send, wait for the owned response, speak, resume listening.
• Allow only one owned turn at a time.
• Stop and release capture promptly from every state.
• Recover to listening after nonfatal transcription, gateway, timeout, or TTS errors when safe.
• Show the current Talk Mode state and pinned conversation.

This phase should not include barge-in.

Phase 5: wake into Talk Mode

• Add a Trigger Talk Mode setting under Voice Wake.
• Route a wake activation into the same Talk runtime used by manual activation.
• If the wake utterance contains trailing request text, allow it to become the first Talk Mode turn.
• Return to Voice Wake, rather than Off, when a wake-started Talk session ends.
• Keep manual and wake-started Talk sessions behaviorally identical after activation.

Phase 6: interruption and polish

• Add optional barge-in with explicit echo and false-trigger handling.
• Allow a documented key or UI action to stop speech without ending Talk Mode.
• Add audio feedback and accessible visual state for transitions.
• Evaluate a Windows system-TTS fallback policy separately from configured-provider errors.
• Measure and tune latency, ambient false activation, sustained CPU use, and model memory on supported Windows architectures.

Safety and privacy invariants

• Voice features remain off by default.
• Persistent microphone ownership is always visible and can always be stopped.
• A production-ready Voice Wake path uses a local model-backed detector rather than depending on exact transcription spelling.
• A no-data capture device fails within a bounded time, releases resources, and leaves the UI responsive.
• Raw audio and ambient transcripts are not logged.
• Voice Wake sends only command speech captured after activation or safely recovered from the wake utterance.
• Talk Mode sends only finalized user turns while the mode is visibly active.
• Disabling Voice Wake or stopping Talk Mode disposes subscriptions, cancels only owned work, and releases microphone capture.
• An unrelated response in the same Chat session is never claimed, spoken, or cancelled.
• TTS failure cannot strand the runtime in Speaking or prevent microphone release.
• Session switching cannot silently move an active Talk Mode conversation to another session.

Validation expectations

Each phase should include:

• focused policy and state-machine tests;
• deterministic fake-audio tests for silence, trigger-only timeout, cancellation, and capture failure;
• a versioned wake-word corpus with positive, near-match, conversational negative, media playback, accent, distance, and noise cases;
• explicit false-reject and false-activation targets for each supported model-backed phrase;
• gateway event-ordering and exact-run correlation regressions;
• settings persistence, localization, accessibility, and lifecycle coverage;
• current-head Windows proof of the user-visible flow;
• CPU and responsiveness proof for sustained wake listening;
• microphone release proof after Off, Stop, failure, disconnect, and app shutdown.

Non-goals

• Cloud speech-to-text for passive wake listening.
• A background Windows service that listens while the Tray app is not running.
• New gateway, MCP, or Windows node voice commands.
• New TTS providers or credential storage changes.
• Barge-in in the first Talk Mode release.
• A reliability guarantee for arbitrary free-form phrases without a generated model or user enrollment.
• Exact implementation parity with Apple Speech or macOS-only MLX TTS.

Completion criteria

This proposal is complete when Windows supports:

• one-shot Voice Wake with both same-utterance and trigger-then-command activation;
• local model-backed wake detection with measured false-reject and false-activation behavior;
• manual continuous Talk Mode;
• optional wake into Talk Mode;
• explicit stop and reliable recovery from every runtime state;
• exact per-turn gateway-run ownership across both modes;
• current-head Windows evidence that sustained listening is responsive, bounded, and releases the microphone correctly.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 12, 2026, 3:18 PM ET / 19:18 UTC.

Summary
Keep open for a maintainer product-direction decision. Current main provides manual speech input and TTS, while the one-shot Voice Wake foundation remains an open PR; no shipped Talk Mode or wake runtime resolves this proposal.

Reproducibility: not applicable. this is a proposed product capability, not a report of broken existing behavior.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.
• Include redacted logs or terminal output.

Maintainer decision needed

Question	Recommendation
Should Windows adopt a staged Voice Wake and transient Talk Mode product after the one-shot preview lands, and what reliability and privacy boundary should the first supported wake experience have?	Sponsor a staged, narrow program: After the one-shot preview is accepted, approve a Phase 1 mode contract and privacy/resource requirements before implementation begins.

Why: Current main has only manual speech input, while the proposal introduces persistent microphone behavior, new settings, model and licensing choices, and a new conversation mode that cannot be selected mechanically.

Next step
The proposal depends on an open foundation PR and requires product decisions about persistent capture, wake reliability, supported phrases, and Talk Mode semantics.

Review details

Best possible solution:

Review the one-shot foundation separately, then sponsor a narrow Phase 1 that defines mode boundaries, privacy and resource policy, and dedicated owners outside App before adding wake detection or Talk Mode.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a proposed product capability, not a report of broken existing behavior.

Is this the best way to solve the issue?

Unclear: separating wake and conversation modes is plausible, but the first supported wake approach and privacy/resource contract require maintainer selection after the open foundation PR is resolved.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The open one-shot foundation PR is not a stable contract yet, so implementing dependent modes now risks duplicating or conflicting with its runtime and response-correlation ownership.
• Always-on microphone capture and a model-backed detector need explicit privacy, supported-phrase, licensing, CPU/battery, accuracy, and failure-recovery policy before a broad implementation begins.

Codex review notes: model internal, reasoning high; reviewed against 4206611f2b2e.

Label changes

Label justifications:

• P3: This is a speculative product program with no confirmed current regression and requires explicit direction before implementation.

Evidence reviewed

What I checked:

• Current voice modes: VoiceService defines only Inactive, PushToTalk, and VoiceChat. It has no Voice Wake or Talk Mode runtime. (src/OpenClaw.Tray.WinUI/Services/VoiceService.cs:13, 4206611f2b2e)
• Current chat path is one-shot: Native Chat calls ListenOnceAsync with a 10-second timeout, confirming the shipped UI path is bounded manual transcription rather than an ongoing voice assistant. (src/OpenClaw.Tray.WinUI/Windows/ChatWindow.xaml.cs:489, 4206611f2b2e)
• No proposed settings surface on main: Current persisted voice settings cover STT, silence timeout, and TTS, not wake phrases, wake activation, or Talk Mode policy. (src/OpenClaw.Shared/SettingsData.cs:77, 4206611f2b2e)
• Legacy continuous overlay is not active: App explicitly disables the voice overlay and routes Voice actions to Voice settings, so the existing VoiceChat helper is not a shipped Talk Mode surface. (src/OpenClaw.Tray.WinUI/App.xaml.cs:1134, 4206611f2b2e)
• Foundation remains open: Provided GitHub context identifies feat: add wake phrase voice assistant preview #1071 as the still-open, dirty one-shot preview foundation; it does not implement this issue’s Talk Mode or model-backed wake scope. (31c906917f00)
• Release and history boundary: The inspected default branch is the v2026.7.1 release commit. Its local history is a single grafted snapshot, so it proves current behavior and current-snapshot authorship but cannot establish deeper feature ownership. (src/OpenClaw.Tray.WinUI/Services/VoiceService.cs:13, 4206611f2b2e)

Likely related people:

• Scott Hanselman: The available current-main snapshot attributes the VoiceService and App paths to this release commit; deeper history is unavailable in the grafted checkout. (role: current source snapshot author; confidence: low; commits: 4206611f2b2e; files: src/OpenClaw.Tray.WinUI/Services/VoiceService.cs, src/OpenClaw.Tray.WinUI/App.xaml.cs)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.