Spacing, capitalization, and punctuation are context-blind — no awareness of text surrounding the cursor

[pianoandglass]

Problem

FreeFlow currently has no awareness of the text surrounding the cursor when pasting dictated text. The spacing logic in writeTranscriptToPasteboard blindly appends a space after sentence-ending punctuation (.!?), and the LLM is not given any surrounding text context. This causes several issues:

1. Trailing whitespace artifact

Every dictation that ends with punctuation appends a trailing space, even when the cursor is at the end of a text field (no text follows). The user sees a visible trailing space after every sentence they dictate.

Current behavior: "Hello world." (with trailing space)
Expected behavior: "Hello world." (no trailing space; consecutive dictations should add leading space instead)

2. Incorrect capitalization when inserting mid-sentence

The LLM always capitalizes the first word of the transcript because it has no knowledge of what comes before the cursor. When inserting text mid-sentence, this produces incorrect output.

Example: Cursor is after "I think " → dictates "we should go" → gets "We should go" (uppercase W)
Expected: "we should go" (lowercase, since we're mid-sentence)

3. Capitalization fails for list items

After a Markdown/plaintext list marker like "- ", "* ", or "1. ", the first letter should be capitalized, but the system doesn't detect list boundaries.

4. Duplicate/conflicting punctuation when inserting mid-text

The LLM adds terminal punctuation assuming the transcript ends the sentence. When the text following the cursor already has punctuation, this creates duplicates like ".." or conflicts like ".?".

5. No surrounding context sent to LLM

The post-processing LLM receives only CONTEXT (app activity summary) and RAW_TRANSCRIPTION, but never the text before and after the cursor. This means the LLM cannot make informed grammatical or spelling decisions based on the actual insertion point.

Proposed Solution

1. Read surrounding text via Accessibility API — Use kAXValueAttribute + kAXSelectedTextRangeAttribute to read precedingText and followingText from the focused text field.
2. Apply deterministic formatting rules — Capitalize at sentence boundaries, newlines, and list markers; lowercase mid-sentence; remove conflicting punctuation; clean all Unicode whitespace variants.
3. Smart spacing — Leading space only when precedingText ends with a word character; trailing space only when followingText starts with a word character; no trailing space at end of field.
4. Feed context to LLM — Include PRECEDING_TEXT and FOLLOWING_TEXT in the user prompt so the model can make better cleanup decisions.
5. Electron/Web hack — For Chromium-based apps where the AX tree doesn't auto-sync, perform a zero-net-movement keypress (→←) to force refresh.
6. Skip screenshot when text context is available — Saves ~200ms per dictation when the Accessibility API can read the text field directly.

[marcbodea]

Very excited for a clean implementation of this, would increase the quality substantially. Let's see if it's possible without doing too much voodoo magic.

Skip screenshot when text context is available — Saves ~200ms per dictation when the Accessibility API can read the text field directly.

Not sure on this, I think it still helps to have additional context

[pianoandglass]

I'm still working on implementing this robustly. It works locally most of the time, but there are edge cases where I need to make sure everything runs correctly, and I'm still figuring out how to add new formatting logic to the local ContextualFormattingService.swift without breaking the existing ones. Right now, I'm focusing on temporary files so I can make debugging more efficient and keep the rules deterministic.

The insertion of new transcriptions really needs to be fluid. If it works well, I think it’s a game-changing feature and would even be worth implementing in FreeFlow 2.0. No voodoo magic required — just making FreeFlow actually flow. 😄

I want to ensure everything works smoothly by using deterministic regex rules, instead of sending the text back to an LLM just to fix capitalization and spacing. If those rules aren’t sufficient for more advanced contextual formatting, then an extra LLM call might be justified, but I want to exhaust all other options first.

Not sure on this, I think it still helps to have additional context

In which situations do you think screenshots provide meaningful context? In my testing, they don't significantly improve transcription post-processing quality, but they do introduce latency on slower connections and consume vision-model quota.

In my tests, what has been most useful so far is lightweight app context (e.g. "The user is dictating in WhatsApp" or "The user is dictating in Antigravity IDE"). That way, the system prompt can br used to adjust punctuation and tone according to current app.

Anyway, I agree that sending screenshots to the LLM for [context] should be optional, and it should either be disabled by default or controlled with a toggle.

[marcbodea]

One of the main benefits I've found of sending screenshots for context is in email dictation, where the context model extracts names with correct spelling of the recipients

[pianoandglass]

Thanks for the input! This one is almost ready to go up as a PR — I'm just finishing the final refinements, and I'm factoring your point about screenshot context into them.