feat: Enable Native Picture-in-Picture (PiP) for Lip-Sync Video Preview - #1173
Conversation
|
@vivek0028 is attempting to deploy a commit to the itzzavdhesh's projects Team on Vercel. A member of the Team first needs to authorize it. |
github-actions
Bot
added
bug
✍️ DCO Sign-off NeededHey @vivek0028! 👋 One or more commits in this PR are missing a Warning
How to fix: For the latest commit: git commit --amend --signoff
git push --force-with-leaseFor multiple commits, replace git rebase --signoff HEAD~N
git push --force-with-leaseThis comment will update automatically after you push. 🤖 VoiceForge Automation · Updates automatically on edits |
🎉 PR Ready for Mentor ReviewHey @vivek0028! 👋 Your PR passed all checks and is now in the GSSoC review queue. Note 🔗 Closing: #1063 · 📐 87 lines across 3 file(s) · 📬 Already requested or no eligible reviewer found @sabeenaviklar @Anushreebasics @itsdakshjain @snehkris @1754riya @Mrigakshi-Rathore @Itzzavdheshh @Nitya-003 @4f4d @lovestaco, this PR is ready for your review — please confirm scope, check behavior and tests, then approve or request changes. Important This is not an approval. Please wait for mentor feedback before expecting a merge. If changes are requested, push them to this same branch and keep the PR focused on the linked issue. 🤖 VoiceForge Automation · Updates automatically on edits |
📝 WalkthroughWalkthroughAdds Clear speech controls, native Picture-in-Picture support for video previews, conditional blur controls, and a Retry Camera action that reruns webcam setup. ChangesMedia controls and recovery
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant VideoPreview
participant PreviewCanvas
participant PipVideo
participant BrowserPiP
User->>VideoPreview: Click PiP Mode
VideoPreview->>PreviewCanvas: Capture canvas stream
VideoPreview->>PipVideo: Assign captured stream
VideoPreview->>BrowserPiP: Request or exit Picture-in-Picture
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
vivek0028
changed the title
github-actions
Bot
added
assignee-match
gssoc26
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@client/src/components/TextToSpeech.jsx`:
- Around line 220-224: Update the submit function and button guard in
TextToSpeech so both paths use the same eligibility rule, including rejecting
submissions when status === "speaking". Ensure handleKeyDown’s direct submit()
call cannot start another request while speech is generating, preferably by
reusing a shared canSubmit predicate.
- Around line 220-223: Update submit() so it invokes onSpeak() only once with
the intended text payload, removing the second call that duplicates
neutral-preset requests. Preserve the subsequent text-clearing behavior after
the single request.
In `@client/src/components/VideoPreview.jsx`:
- Around line 32-50: Update the VideoPreview PiP state around togglePiP to track
whether PiP is active, register enterpictureinpicture and leavepictureinpicture
listeners on the PiP video element, and clean them up appropriately. Bind the
PiP button label and aria-pressed attribute to this state so both programmatic
toggles and native PiP exits stay synchronized.
- Around line 33-45: Update isPiPSupported in VideoPreview to require
document.pictureInPictureEnabled plus captureStream availability on the canvas
referenced by ref and requestPictureInPicture availability on
pipVideoRef.current before rendering or enabling the PiP control. Keep togglePiP
unchanged for supported environments and ensure unsupported browsers do not
expose an actionable control.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 8b6a89f1-d702-42f1-a286-c578305a8a31
📒 Files selected for processing (3)
client/src/components/TextToSpeech.jsxclient/src/components/VideoPreview.jsxclient/src/pages/Call.jsx
| <div className="mt-4 flex gap-3"> | ||
| <button | ||
| type="button" | ||
| onClick={submit} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Prevent duplicate speech requests from one click.
This action reaches submit(), which currently calls onSpeak() once with finalText and then again with trimmedText. For the neutral preset those payloads are identical, so one Speak click generates twice. Remove the second call and clear the text after the single intended request.
Proposed fix
await onSpeak(finalText, voice_settings_override);
- if (!trimmedText || disabled || characterCount > MAX_CHARS) return;
- await onSpeak(trimmedText);
setText("");📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <div className="mt-4 flex gap-3"> | |
| <button | |
| type="button" | |
| onClick={submit} | |
| await onSpeak(finalText, voice_settings_override); | |
| setText(""); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@client/src/components/TextToSpeech.jsx` around lines 220 - 223, Update
submit() so it invokes onSpeak() only once with the intended text payload,
removing the second call that duplicates neutral-preset requests. Preserve the
subsequent text-clearing behavior after the single request.
| <div className="mt-4 flex gap-3"> | ||
| <button | ||
| type="button" | ||
| onClick={submit} | ||
| disabled={disabled || !trimmedText || status === "speaking" || characterCount > MAX_CHARS} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep submit and the button’s busy guard in sync.
handleKeyDown calls submit() directly, but submit() does not check status === "speaking". Pressing Enter while speech is generating therefore bypasses the disabled Speak button and can start another request. Add the status check inside submit() or share one canSubmit predicate between both paths.
Proposed fix
async function submit() {
- if (!trimmedText || disabled) return;
+ if (!trimmedText || disabled || status === "speaking") return;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@client/src/components/TextToSpeech.jsx` around lines 220 - 224, Update the
submit function and button guard in TextToSpeech so both paths use the same
eligibility rule, including rejecting submissions when status === "speaking".
Ensure handleKeyDown’s direct submit() call cannot start another request while
speech is generating, preferably by reusing a shared canSubmit predicate.
| const pipVideoRef = React.useRef(null); | ||
| const isPiPSupported = typeof document !== "undefined" && document.pictureInPictureEnabled; | ||
|
|
||
| const togglePiP = async () => { | ||
| try { | ||
| if (document.pictureInPictureElement) { | ||
| await document.exitPictureInPicture(); | ||
| } else { | ||
| if (!pipVideoRef.current.srcObject) { | ||
| const stream = ref.current.captureStream(30); | ||
| pipVideoRef.current.srcObject = stream; | ||
| await pipVideoRef.current.play(); | ||
| } | ||
| await pipVideoRef.current.requestPictureInPicture(); | ||
| } | ||
| } catch (error) { | ||
| console.error("PiP error:", error); | ||
| } | ||
| }; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate file and relevant symbols"
fd -a 'VideoPreview\.jsx$' . || true
echo
echo "Show relevant lines and outline"
if [ -f client/src/components/VideoPreview.jsx ]; then
wc -l client/src/components/VideoPreview.jsx
ast-grep outline client/src/components/VideoPreview.jsx --view compact || true
echo
sed -n '1,90p' client/src/components/VideoPreview.jsx | cat -n
echo "--- lines 400-440 ---"
sed -n '400,440p' client/src/components/VideoPreview.jsx | cat -n
fi
echo
echo "Search PiP usages/listeners/state"
rg -n "pictureInPicture|requestPictureInPicture|pipe|piped|pip|PiP|enterpictureinpicture|leavepictureinpicture|aria-pressed|captureStream" .Repository: itzzavdhesh/VoiceForge
Length of output: 10367
Track PiP lifecycle state.
The PiP button always shows “PiP Mode” and has no aria-pressed; native leavePictureInPicture events from the user closing PiP won’t update it. Add enterpictureinpicture/leavepictureinpicture listeners and bind the label/aria-pressed to active PiP state.
Also applies to: 422-429
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@client/src/components/VideoPreview.jsx` around lines 32 - 50, Update the
VideoPreview PiP state around togglePiP to track whether PiP is active, register
enterpictureinpicture and leavepictureinpicture listeners on the PiP video
element, and clean them up appropriately. Bind the PiP button label and
aria-pressed attribute to this state so both programmatic toggles and native PiP
exits stay synchronized.
Source: MCP tools
| const isPiPSupported = typeof document !== "undefined" && document.pictureInPictureEnabled; | ||
|
|
||
| const togglePiP = async () => { | ||
| try { | ||
| if (document.pictureInPictureElement) { | ||
| await document.exitPictureInPicture(); | ||
| } else { | ||
| if (!pipVideoRef.current.srcObject) { | ||
| const stream = ref.current.captureStream(30); | ||
| pipVideoRef.current.srcObject = stream; | ||
| await pipVideoRef.current.play(); | ||
| } | ||
| await pipVideoRef.current.requestPictureInPicture(); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate VideoPreview.jsx"
fd -a 'VideoPreview\.jsx$' . || true
file="$(fd 'VideoPreview\.jsx$' . | head -n1 || true)"
if [ -n "${file}" ]; then
echo "File: ${file}"
wc -l "$file"
echo "--- outline ---"
ast-grep outline "$file" --view compact || true
echo "--- relevant sections ---"
sed -n '1,80p' "$file"
echo "--- button/render section ---"
sed -n '390,470p' "$file"
echo "--- capture stream/piP refs searches ---"
rg -n "captureStream|pictureInPicture|requestPictureInPicture|exitPictureInPicture|pipVideoRef|ref current" "$file"
fiRepository: itzzavdhesh/VoiceForge
Length of output: 6961
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
# Behavioral model of the relevant capability gating:
# browser exposes document.pictureInPictureEnabled as true,
# but HTMLCanvasElement.prototype.captureStream or HTMLVideoElement.prototype.requestPictureInPicture is missing.
models = [
("pips true, capture missing", {"pictureInPictureEnabled": True, "exitPitchInPicture": lambda: None, "canvas_captureStream": None, "video_requestPictureInPicture": lambda: None}),
("pips true, request missing", {"pictureInPictureEnabled": True, "exitPictureInPicture": lambda: None, "canvas_captureStream": lambda n: None, "video_requestPictureInPicture": None}),
("pips true, both present", {"pictureInPictureEnabled": True, "exitPictureInPicture": lambda: None, "canvas_captureStream": lambda n: None, "video_requestPictureInPicture": lambda: None}),
]
for name, caps in models:
is_supported = caps.get("pictureInPictureEnabled")
try:
if is_supported:
if caps.get("canvas_captureStream") is None:
raise TypeError("captureStream missing")
if caps.get("video_requestPictureInPicture") is None:
raise TypeError("requestPictureInPicture missing")
result = "ok"
else:
result = "button hidden"
except TypeError as exc:
result = str(exc)
print(f"{name}: isPiPSupported={is_supported}, toggle outcome={result}")
PYRepository: itzzavdhesh/VoiceForge
Length of output: 406
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate other PiP references and React usage"
rg -n "pictureInPicture|captureStream|exitPictureInPicture|requestPictureInPicture|CanvasCaptureMediaStreamTrack|captureStream" . --glob '!node_modules' --glob '!dist' --glob '!build' || true
echo "--- package hints for React/JSX target ---"
for f in package.json client/package.json client/src/components/VideoPreview.jsx; do
[ -f "$f" ] && { echo "### $f"; sed -n '1,80p' "$f"; }
doneRepository: itzzavdhesh/VoiceForge
Length of output: 5301
🌐 Web query:
MDN Picture-in-Picture API requestPictureInPicture HTMLCanvasElement captureStream PictureInPictureEnabled compatibility
💡 Result:
The Picture-in-Picture API is primarily designed for the HTMLVideoElement interface and does not natively support HTMLCanvasElement directly through a requestPictureInPicture call [1][2]. However, you can achieve Picture-in-Picture functionality for an HTMLCanvasElement by leveraging the captureStream method [3]. Workaround: HTMLCanvasElement to Picture-in-Picture To display a canvas in Picture-in-Picture mode, you must capture a MediaStream from the canvas and assign it to an HTMLVideoElement [3]. 1. Capture the stream: Call canvas.captureStream on your HTMLCanvasElement to get a MediaStream [4][3]. 2. Create a video element: Create an off-screen or hidden HTMLVideoElement and set its srcObject to the captured stream [3]. 3. Trigger PiP: Call the requestPictureInPicture method on that video element [3]. Example: const canvas = document.querySelector('canvas'); const video = document.createElement('video'); video.muted = true; video.srcObject = canvas.captureStream(60); // 60 FPS video.play; video.requestPictureInPicture; Compatibility and Alternatives The standard Picture-in-Picture API is strictly limited to HTMLVideoElement for security and usability reasons [2][5]. If your goal is to display arbitrary HTML content (like interactive canvas elements, custom controls, or layouts) in an always-on-top window, you should use the Document Picture-in-Picture API [6][7]. This newer API allows you to open a dedicated, always-on-top window that can contain any HTML content, avoiding the need for canvas-to-video workarounds [6][5]. Document.pictureInPictureEnabled remains the standard way to check if the browser supports the original video-based Picture-in-Picture API [1][8].
Citations:
- 1: https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API
- 2: https://www.w3.org/TR/picture-in-picture/
- 3: https://github.com/w3c/picture-in-picture/blob/main/explainer.md
- 4: https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream
- 5: https://github.com/wicg/document-picture-in-picture
- 6: https://developer.mozilla.org/en-US/docs/Web/API/Document_Picture-in-Picture_API
- 7: https://developer.chrome.com/docs/web-platform/document-picture-in-picture
- 8: https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/requestPictureInPicture
🌐 Web query:
specification Picture-in-Picture API HTMLCanvasElement captureStream requestPictureInPicturePictureInPictureEnabled
💡 Result:
The Picture-in-Picture API is designed primarily for the HTMLVideoElement interface [1][2][3]. It does not directly support the HTMLCanvasElement [2][4]. However, developers can achieve a Picture-in-Picture effect for canvas content by utilizing the HTMLCanvasElement.captureStream method [5][6][2]. The standard workaround involves the following steps: 1. Capture a MediaStream from the canvas using canvas.captureStream [5][7][2]. 2. Assign this stream to the srcObject property of an HTMLVideoElement [8][2][4]. 3. Call the requestPictureInPicture method on that video element [8][2][9]. Key components related to this process include: - HTMLCanvasElement.captureStream: Generates a MediaStream containing a real-time capture of the canvas's contents [5][6]. - requestPictureInPicture: A method on the HTMLVideoElement interface used to initiate the floating window [1][9]. - pictureInPictureEnabled: A property on the Document interface that returns a boolean indicating whether the Picture-in-Picture mode is enabled and allowed in the current context [1][2][9]. For more flexible use cases involving arbitrary HTML content (beyond just video or canvas), the Document Picture-in-Picture API is an alternative that allows opening an always-on-top window populated with any HTMLElement [10][3]. Unlike the standard Picture-in-Picture API, this is a separate, dedicated API [11][3].
Citations:
- 1: https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API
- 2: https://github.com/w3c/picture-in-picture/blob/main/explainer.md
- 3: https://developer.mozilla.org/en-US/docs/Web/API/Document_Picture-in-Picture_API
- 4: https://stackoverflow.com/questions/61301087/is-it-possible-to-use-pip-mode-which-is-not-for-videos
- 5: https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream
- 6: https://developer.chrome.com/blog/capture-stream
- 7: https://developer.mozilla.org/en-US/docs/Web/API/CanvasCaptureMediaStreamTrack
- 8: https://docs.videosdk.live/javascript/guide/video-and-audio-calling-api-sdk/render-media/picture-in-picture
- 9: https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/requestPictureInPicture
- 10: https://developer.chrome.com/docs/web-platform/document-picture-in-picture
- 11: https://github.com/WICG/document-picture-in-picture/blob/main/README.md
🌐 Web query:
MDN HTMLCanvasElement captureStream not exposed Picture-in-Picture available pictureInPictureEnabled
💡 Result:
The HTMLCanvasElement.captureStream method and the Picture-in-Picture (PiP) API are distinct features with different purposes and APIs [1][2]. HTMLCanvasElement.captureStream is a method that allows you to generate a MediaStream from a canvas element [1][3]. It is not a Picture-in-Picture feature and is not governed by the Picture-in-Picture API [1][2]. Therefore, the property document.pictureInPictureEnabled does not relate to whether captureStream can be used on a canvas [2]. If you wish to use Picture-in-Picture with a canvas, you cannot use the traditional HTMLVideoElement.requestPictureInPicture method, as that is specifically for video elements [2]. Instead, you should use the Document Picture-in-Picture API, which allows you to open an always-on-top window and populate it with arbitrary HTML content, including a canvas element [4]. To check for support for this, you should look for the 'documentPictureInPicture' property on the window object: if ('documentPictureInPicture' in window) { // The Document Picture-in-Picture API is supported. } [4]
Citations:
- 1: https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream
- 2: https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API
- 3: https://www.w3.org/TR/2025/WD-mediacapture-fromelement-20250212/
- 4: https://developer.chrome.com/docs/web-platform/document-picture-in-picture
Gate the PiP control on every required API.
isPiPSupported only checks document.pictureInPictureEnabled, but this flow also needs HTMLCanvasElement.captureStream() and HTMLVideoElement.requestPictureInPicture(). If a browser exposes PiP support without either capability, the button still renders and clicks only surface a caught TypeError. Check both methods before rendering/enabling the control.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@client/src/components/VideoPreview.jsx` around lines 33 - 45, Update
isPiPSupported in VideoPreview to require document.pictureInPictureEnabled plus
captureStream availability on the canvas referenced by ref and
requestPictureInPicture availability on pipVideoRef.current before rendering or
enabling the PiP control. Keep togglePiP unchanged for supported environments
and ensure unsupported browsers do not expose an actionable control.
Source: MCP tools
🎊 PR Merged SuccessfullyHey @vivek0028! 👋 Congratulations and thank you for your contribution to VoiceForge! Note 🔗 Linked issue(s): #1063 · ✅ Marked as merged and complete Maintainers may still handle final cleanup, release notes, or follow-up tracking after the merge. 🤖 VoiceForge Automation · Updates automatically on edits |
itzzavdhesh
added
gssoc:approved
itzzavdhesh
added
SSoC26
3 participants
🚀 Program
GSSoC
📝 Description
This PR adds native Picture-in-Picture (PiP) support for the lip-synced avatar preview on the Call page.
VoiceForge is often used alongside video conferencing applications such as Zoom, Microsoft Teams, and Google Meet. Since these applications may occupy most or all of the screen, the VoiceForge preview can become difficult to monitor during a live call.
With this change, users can move the lip-synced avatar preview into a floating, resizable PiP window that stays above other application windows, making it easier to continuously monitor the avatar while multitasking.
Changes Made
Added PiP control logic
togglePiPinVideoPreview.jsx.document.pictureInPictureElementto safely enter or exit PiP mode.Added canvas-to-video stream routing
requestPictureInPicture()operates on<video>elements rather than<canvas>.<video>element usingpipVideoRef.captureStream(30)and routes the resulting 30 FPS stream to the hidden video element before entering PiP.Added browser compatibility handling
document.pictureInPictureEnabled.Cleaned up the VideoPreview UI
🔗 Related Issue
Closes #1063
🔄 Type of Change
🧪 How to Test
http://localhost:5173in a browser that supports native Picture-in-Picture.Tested
✅ Checklist
PR Title:
feat: Enable Native Picture-in-Picture (PiP) for Lip-Sync Video PreviewSummary by cubic
Adds native Picture-in-Picture to the lip-sync video preview so users can pop it out and keep it visible during calls. Also adds a Clear button in TTS and a Retry Camera control, and removes the duplicate Blur toggle.
New Features
captureStream(30)from the canvas; toggles enter/exit and only shows when supported.Bug Fixes
Written for commit cac2d78. Summary will update on new commits.
Summary by CodeRabbit
New Features
UI Improvements