Skip to content

feat: Voice Workspace canvas enrichment (Mermaid, images, panel history) + orb polish #979

Description

@vybe

Summary

Enrich the Voice Workspace canvas (the agent-controlled panel beside the voice orb) and polish the orb animation. Today the canvas supports only markdown and html panel types and overwrites itself on every update; the orb pulses directly off a raw amplitude value with no temporal smoothing. This adds Mermaid diagrams, image display, and client-side panel history, smooths and enlarges the orb, and adds a graceful transition when the canvas updates.

These are borrowed from a local prototype of the same voice-workspace concept that has a richer panel renderer and a smoother orb.

Context

Trinity already ships voice workspace mode (#699/#860): src/backend/services/gemini_voice.py declares four panel tools (show_markdown, update_panel, append_to_panel, clear_panel) and src/frontend/src/views/AgentWorkspace.vue renders the panel (markdown via DOMPurify, HTML via a sandboxed iframe with Chart.js bundled) on a 300 ms poll of GET /api/agents/{name}/voice/{session_id}/panel.

Gaps vs. the prototype:

  • No diagrams — mermaid is not a dependency and there is no show_diagram tool. For a "thinking whiteboard," flowcharts / sequence / mindmap / timeline diagrams are arguably higher-value than the Chart.js charts already supported.
  • No images — the agent cannot show a generated chart, screenshot, or diagram asset.
  • No history — the panel is overwritten on every update; once the agent redraws, the previous "whiteboard" is gone.
  • Jerky orb — renderFrame reads voice.amplitude raw each frame (no smoothing) and the core size 45 + energy*20 is computed inline (also unsmoothed), so the orb jumps; it also goes flat when silent (no idle "breathe").
  • Abrupt canvas swaps — new panel content replaces old with no transition.

Scope

1. Mermaid diagrams (top borrow)

  • Add a show_diagram(diagram, title?) panel tool and a mermaid panel type.
  • Render the diagram inside the sandboxed iframe (bundle mermaid the same way Chart.js is injected), keeping SVG injection out of the parent DOM / DOMPurify path and consistent with the existing security model.
  • Follow-up (optional, deferred): auto-render ```mermaid fenced blocks inside show_markdown — messier because markdown renders in the parent DOM via DOMPurify.

2. show_image(src, title?, caption?)

  • Web URLs render directly; agent-workspace file paths are served through the existing file-download endpoint so the backend streams the bytes (no widening of the iframe sandbox, no arbitrary local-FS read). Validate the path is workspace-confined.

3. Client-side panel history navigation (frontend-only, no backend change)

  • Keep a ring buffer of the last ~40 panel snapshots; add prev/next + a dropdown selector in the panel header. "Live" follows the latest; navigating back pins a snapshot until the user returns to live (or a brand-new update arrives).

4. Orb polish — less jerky, slightly larger, larger soft-glow amplitude

  • Smooth energy with a persistent value lerped toward the target using asymmetric attack/release (attack ~0.18, release ~0.10) instead of using the raw amplitude each frame.
  • Smooth the core size with a persistent coreSize lerped toward targetCoreSize rather than computing 45 + energy*20 inline.
  • Add a low-amplitude idle "breathe" floor so the orb gently moves at rest.
  • Enlarge: bump the core base (~45 → ~58) and/or the global spread/size (1.1 → 1.25); increase the energy→core-size coupling (*20 → ~*32) for a larger soft-glow swing.

5. Graceful canvas-update transition

  • Cross-fade / subtle slide-in of new panel content, a brief "updated" flash on the header timestamp, and (optionally) a thin top loader bar. Respect prefers-reduced-motion.

Acceptance Criteria

  • show_diagram(diagram, title?) is declared as a panel tool, added to _PANEL_TOOL_NAMES, handled in _execute_panel_tool (sets panel_state.type = "mermaid"), and documented in WORKSPACE_PANEL_INSTRUCTIONS.
  • mermaid is a frontend dependency and a mermaid-type panel renders a valid diagram inside the sandboxed iframe; invalid syntax shows a contained error, not a broken panel.
  • show_image(src, title?, caption?) displays a web-URL image and an agent-workspace file-path image (served via the existing download endpoint); paths outside the workspace are rejected.
  • Panel header shows prev/next + selector when ≥2 snapshots exist; navigating back shows the historical snapshot and "live" returns to the latest. No backend changes for history.
  • Orb no longer visibly steps/jerks with amplitude changes; energy and core size are temporally smoothed; the orb gently breathes when idle; it is visibly larger with a larger soft-glow swing.
  • Canvas updates transition smoothly (cross-fade or slide-in) and honor prefers-reduced-motion.
  • All three new/changed surfaces stay in sync per Architectural Invariant feat: SMARTS trading pipeline with Telegram notifications and Miro visualization #13 where applicable (backend tool decl ↔ frontend renderer).

Technical Notes

  • Backend: src/backend/services/gemini_voice.py — _PANEL_TOOLS, _PANEL_TOOL_NAMES, _execute_panel_tool, WORKSPACE_PANEL_INSTRUCTIONS. Panel tools execute in-process (no agent container call), so new types are cheap to add.
  • Frontend: src/frontend/src/views/AgentWorkspace.vue — panel render branch (panelState.type), the iframe srcdoc builder (where Chart.js is already injected — add mermaid the same way), the poll handler (history ring buffer hook), and the orb renderFrame / drawCore loop (smoothing, size, glow).
  • Frontend deps: src/frontend/package.json already has chart.js, marked, dompurify — add mermaid.
  • Endpoint contract for the panel is unchanged (GET /api/agents/{name}/voice/{session_id}/panel in src/backend/routers/voice.py); items 3–5 are frontend-only.
  • Keep the security posture: agent HTML/diagrams render only inside the opaque-origin sandboxed iframe; do not move agent-supplied markup into the parent DOM.

Out of Scope

  • Server-push for panel updates (stays on the existing 300 ms poll).
  • In-markdown ```mermaid auto-rendering (tracked as the optional follow-up above).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions