A voice-driven HeyGen LiveAvatar that listens to you and renders real-time motion-graphic overlays by prompting Claude Code with the Hyperframes skill.
Talk to the avatar. When you ask for something visual, a background Claude Code session writes a composition.html and it overlays onto the avatar's video stream — live, transparent, looping.
- LiveAvatar FULL mode handles speech recognition, LLM reply, TTS, and WebRTC video. The avatar holds normal small-talk.
- Each user transcript is appended to a turn buffer, flushed on
USER_SPEAK_ENDEDor after a 3s debounce. - The flushed turn is sent to a Haiku classifier (
claude -p --model haiku) that returns either{"intent":"chat"}or{"intent":"generate","prompt":"..."}. ASR transcripts are noisy, so the classifier is told to phonetically reconstruct mangled subjects. - A
generatedecision spawns Claude Code (claude -p --output-format stream-json --verbose) inside a per-job workspace. The hyperframes skill writescomposition.html. - A
chokidarwatcher picks up the new file and the backend pushes ahyperframes_readyevent over SSE. The frontend swaps the URL on a<hyperframes-player>overlay that loops on top of the avatar video. - Single-flight job control: a new generation cancels the previous one.
Prereqs:
- Node 20+
- Claude Code installed and authenticated (uses your subscription — no Anthropic API key)
- The Hyperframes skill installed:
npx add-skill heygen-com/hyperframes - A LiveAvatar API key from https://app.liveavatar.com
git clone https://github.com/heygen-com/liveavatar-hyperframes-demo
cd liveavatar-hyperframes-demo
npm install
cp .env.example .env
# fill in LIVEAVATAR_API_KEY (+ optional avatar/voice IDs)
npm run devOpen http://localhost:3000, click Start session, allow mic access, then say something like "animate three orange cats" or "show me a glowing sunset". The status pill turns purple while Claude Code works; the overlay fades in when the composition is ready and loops until the next one.
| Var | Notes |
|---|---|
LIVEAVATAR_API_KEY |
required — from https://app.liveavatar.com |
LIVEAVATAR_AVATAR_ID |
required if SANDBOX=false; pick a VIDEO avatar from /v1/avatars |
LIVEAVATAR_VOICE_ID |
optional |
LIVEAVATAR_SANDBOX |
true = free 1-min sessions on the fixed sandbox avatar |
BACKEND_PORT |
default 4000 |
voice SSE events
user ──────────────────► LiveAvatar ──────────────► frontend
(FULL) ▲ │
│ │ <hyperframes-player>
│ ▼
backend ◄──── overlay on avatar video
│
turn buffer ────┤
▼
Haiku classifier (intent: chat | generate)
│
▼ (generate)
Claude Code (-p, stream-json)
│
▼
workspace/<jobId>/composition.html
│
chokidar watcher
│
▼
hyperframes_ready
backend/server.js— Express on:4000. Session tokens, classifier, generator spawn, SSE broadcast, file watcher.frontend/src/main.js— LiveAvatar SDK wiring, status pill, log panel,<hyperframes-player>overlay.workspace/<jobId>/composition.html— generated artifacts. Served at/workspace/....frontend/test-player.html— standalone player harness for debugging compositions outside the avatar.
- Generation isn't instant. Even with
--model haiku, a hyperframes job takes ~30s–2min because Claude Code reads the skill's reference files before writing. The avatar acknowledges briefly so you're not staring at silence. - LiveAvatar context "Hyperframes Studio v2" instructs the avatar to react to ideas but never claim it's generating — only the real pipeline calls
session.repeat("On it…")when a generate decision actually fires. - 5-min session timeout. Frontend keep-alive pings every 2 min.
session.repeat()interrupts the avatar's current speech — useful for "On it" / "Here it is!" cues.- Workspace dirs are gitignored. Generated compositions don't ship with the repo.
MIT