feat: Kaori 3D companion — interactive stage, live voice, trick demos - #4
Merged
Merged
Conversation
Adds the app's first 3D capability: an interactive VRM companion stage where Kaori renders, talks, emotes, and physically demonstrates tricks — the Phase-2 payoff of the 3D Companions PRD. Rendering (src/components/companion/): - KaoriStage: react-three-fiber (native) + expo-gl + @pixiv/three-vrm. three pinned to 0.170.0; navigator.userAgent shim for GLTFLoader on Hermes; MToon downgraded to unlit textured MeshBasicMaterial and textures re-baked through expo-gl's working localUri path (upstream fiber texture upload is a no-op on expo-gl); frustumCulled off; orbit/pinch camera; device-only (simulator GL can't compile the shaders — honest fallback card). - riderFundamentals: reusable rider motion vocabulary (stance, anatomical crouch + hip-drop, wind-up coil, arms, jump arc) shared across tricks. - trickAnimations: TRICKS timeline registry (frontside 360) + speech-synced demo state machine (on-board session, auto-loop, phase cues). Voice (src/hooks, src/lib/companion/): - Kith WebSocket client + gapless streamed-TTS player (react-native-audio-api AudioBufferQueueSourceNode) + expo-speech-recognition mic input. Same bot-chat brain; replies stream back as ElevenLabs audio with mouth-sync, gesture and emotion animation. Her spoken sentences cue the trick choreography. Entry points: bot-chat header + home CompanionWidget (both Kaori-gated). Also folds in this session's pre-existing WIP and fixes the repo's pre-existing biome + TypeScript errors so CI (biome + tsc + check-prod-ready) is green. New deps: three 0.170.0, @react-three/fiber 9.6.1, @pixiv/three-vrm 3.5.2, expo-gl, expo-asset, react-native-audio-api 0.13.1, expo-speech-recognition 3.1.3, @types/three; react-native-worklets pinned to 0.5.1 (SDK 54). Adds mic + speech-recognition permissions and a patch-package patch for audio-api's iOS 26 AVAudioSession constant. Requires a new dev-client build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds the app's first 3D capability: an interactive VRM companion stage where Kaori renders, talks, emotes, and physically demonstrates tricks. This is the Phase‑2 payoff of the 3D Companions PRD — the companion stops being a chat bubble and becomes a coach with a body. Reachable from the bot‑chat header and the home Companion widget (both Kaori‑gated).
Environment Variables
No environment variable changes in the app. (
KITH_WS_URLis a derived constant insrc/constants/api.ts; the Kith sidecar + ElevenLabs key live server‑side.)Change Summary
KaoriStage— react‑three‑fiber (native) + expo‑gl +@pixiv/three-vrmVRM renderer; orbit/pinch camera; procedural idle; device‑only with a fallback card on simulators.riderFundamentals— reusable rider motion vocabulary (stance, anatomical crouch + hip‑drop, wind‑up coil, arms, jump arc) shared across tricks.trickAnimations—TRICKStimeline registry (frontside 360) + speech‑synced demo state machine (on‑board session, auto‑loop, phase cues).react-native-audio-api), andexpo-speech-recognitionmic input. Same bot‑chat brain; replies stream back as ElevenLabs audio with mouth‑sync + gesture + emotion animation, and her sentences cue the trick choreography.User._id/statstyping, a crashing couch‑API import, an expo‑video listener comparing an object to a string, null‑vs‑undefined,OsPermissionexport,Buttonstyle callback).riderFundamentals(shared primitives) +trickAnimations(per‑trick timelines).three@0.170.0,@react-three/fiber@9.6.1,@pixiv/three-vrm@3.5.2,expo-gl,expo-asset,react-native-audio-api@0.13.1,expo-speech-recognition@3.1.3,@types/three;react-native-workletspinned to0.5.1(Expo SDK 54).package-lock.jsonupdated. Newpatches/react-native-audio-api+0.13.1.patch(iOS‑26AVAudioSessionconstant).Technical Impact
src/components/companion/*(new),src/hooks/*(new),src/lib/companion/*(new),app/(tabs)/homies/companion-stage/*(new), plus wiring in the home screen, homies layout, bot‑chat header,metro.config.js(vrm assetExt + single‑three resolver),app.config.js(speech‑recognition plugin + mic/speech permissions)./bot-chatendpoints).x-kith-sessionheader (backend side handled separately).Testing Evidence
npx biome check .(0 errors),npx tsc --noEmit(0 errors),bash scripts/check-prod-ready.sh(pass) — the exact gates CI runs.assembleRelease) compiled clean earlier this session with the full dependency set + patch.Migration / Deployment Notes
wss://api.thetrickbook.com/kith/wsin prod) — tracked separately in the Backend repo.assets/models/kaori.vrmbinary. The repo isn't on git‑LFS; consider moving large model assets to LFS if more companions are added.Screenshots / Examples
Verified live on device (screenshots shared during development): full‑color Kaori on the stage, and mid‑trick during a frontside‑360 demo.