Skip to content

Stream widget chat responses (token streaming) #170

Description

@rahulsainlll

The per-product widget chat (apps/dashboard/app/api/widget/[publicKey]/chat/route.ts) waits for the full LLM reply before returning, so the user stares at nothing for ~2s. This is the biggest perceived-latency problem.

Goal

Stream tokens so text appears live (first token < ~1s), like ChatGPT/Fin.

Scope

  • Change the chat route to return a streamed response (ReadableStream / SSE) instead of buffering the full reply.
  • Update the Studio preview (features/products/test-preview-panel.tsx) and the embed widget (public/embed.js) to consume the stream and render incrementally.
  • Plain-text replies stream; the confirm-action ({reply, action}) case can stay non-streamed for v1.

Acceptance

  • First token visible in well under 1s on a warm route.
  • Streaming works in both the Studio preview and the embedded widget.
  • No regression to the confirm-action flow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions