diff --git a/examples/autozoom/README.md b/examples/autozoom/README.md new file mode 100644 index 0000000..35ee340 --- /dev/null +++ b/examples/autozoom/README.md @@ -0,0 +1,31 @@ +# Autozoom + +Demonstrates cinematic autozoom that eases into each user action and releases afterwards. Small form fields and buttons benefit the most — on mobile viewports the effect makes details readable. + +## Features demonstrated + +- `autoZoom: true` — enable with defaults +- Multiple interactions grouped into zoom "sessions" +- Zoom settles before each click, holds through the action, releases at the end + +## Run + +```bash +cd examples/autozoom +webreel record +``` + +## Tuning + +Pass an object instead of `true` to override defaults: + +```json +{ + "autoZoom": { + "enabled": true, + "approachS": 1.2, + "holdAfterS": 0.8, + "paddingRatio": 0.4 + } +} +``` diff --git a/examples/autozoom/videos/autozoom.mp4 b/examples/autozoom/videos/autozoom.mp4 new file mode 100644 index 0000000..4084d40 --- /dev/null +++ b/examples/autozoom/videos/autozoom.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:88558055f4c9520056976875c4b51657cad6a8855d6e8cc7f6b9c36dccf9d0ed +size 4077700 diff --git a/examples/autozoom/videos/autozoom.png b/examples/autozoom/videos/autozoom.png new file mode 100644 index 0000000..d56eca5 --- /dev/null +++ b/examples/autozoom/videos/autozoom.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:300f2856b405e181902caf31050f684e9e0665cfceada1491787f7d630fd434e +size 102878 diff --git a/examples/autozoom/web/index.html b/examples/autozoom/web/index.html new file mode 100644 index 0000000..538b38d --- /dev/null +++ b/examples/autozoom/web/index.html @@ -0,0 +1,268 @@ + + + + + + Autozoom Demo + + + + +
+

Profile settings

+

Small UI elements that benefit from autozoom.

+
+ + +
+
+ + +
+
+ +
+ +
+ + + +
+
+
+
+ + +
+
+
+ + + diff --git a/examples/autozoom/webreel.config.json b/examples/autozoom/webreel.config.json new file mode 100644 index 0000000..0009b2e --- /dev/null +++ b/examples/autozoom/webreel.config.json @@ -0,0 +1,26 @@ +{ + "$schema": "https://webreel.dev/schema/v1.json", + "videos": { + "autozoom": { + "url": "./web/index.html", + "viewport": { "width": 1920, "height": 1080 }, + "zoom": 2, + "waitFor": ".card", + "autoZoom": true, + "defaultDelay": 500, + "steps": [ + { "action": "pause", "ms": 1500 }, + { "action": "type", "text": "Jane Doe", "selector": "#name", "charDelay": 80 }, + { + "action": "type", + "text": "jane@acme.com", + "selector": "#email", + "charDelay": 80 + }, + { "action": "click", "selector": "#role-btn" }, + { "action": "click", "text": "Engineer" }, + { "action": "click", "selector": "#save", "delay": 1200 } + ] + } + } +} diff --git a/packages/@webreel/core/LICENSE b/packages/@webreel/core/LICENSE new file mode 100644 index 0000000..8226d36 --- /dev/null +++ b/packages/@webreel/core/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + +Copyright 2025 Vercel Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/packages/@webreel/core/README.md b/packages/@webreel/core/README.md index d757292..84cd7a0 100644 --- a/packages/@webreel/core/README.md +++ b/packages/@webreel/core/README.md @@ -1,13 +1,15 @@ -# @webreel/core +# @lgariv/webreel-core -Chrome automation, recording, and overlay engine for webreel. +> **Fork notice.** This is a fork of [`@webreel/core`](https://github.com/vercel-labs/webreel) with added autozoom filter generation and a click-revealed UI mutation observer. Original work © Vercel, licensed under Apache 2.0. Modifications in [`lgariv-dn/webreel`](https://github.com/lgariv-dn/webreel). -Launches a headless Chrome instance via the Chrome DevTools Protocol, captures screenshots at ~60fps, and encodes the result to MP4 with ffmpeg. Provides actions for clicking, typing, dragging, and cursor animation, plus on-screen overlays for keystroke labels and a custom cursor. +Chrome automation, recording, and overlay engine for `@lgariv/webreel`. + +Launches a headless Chrome instance via the Chrome DevTools Protocol, captures screenshots at ~60fps, and encodes the result to MP4 with ffmpeg. Provides actions for clicking, typing, dragging, and cursor animation, plus on-screen overlays for keystroke labels, a custom cursor, and optional autozoom filters for cinematic framing. ## Installation ```bash -npm install @webreel/core +npm install @lgariv/webreel-core ``` ## Examples diff --git a/packages/@webreel/core/package.json b/packages/@webreel/core/package.json index 4fca6ad..e3b3179 100644 --- a/packages/@webreel/core/package.json +++ b/packages/@webreel/core/package.json @@ -1,15 +1,15 @@ { - "name": "@webreel/core", - "version": "0.1.4", - "description": "Core recording engine for webreel - headless Chrome capture, cursor animation, and video compositing.", - "homepage": "https://webreel.dev", + "name": "@lgariv/webreel-core", + "version": "0.1.4-beta-20260418T165223Z", + "description": "Core recording engine for @lgariv/webreel (fork of vercel-labs/webreel) — headless Chrome capture, cursor animation, video compositing, autozoom.", + "homepage": "https://github.com/lgariv-dn/webreel", "repository": { "type": "git", - "url": "https://github.com/vercel-labs/webreel.git", + "url": "https://github.com/lgariv-dn/webreel.git", "directory": "packages/@webreel/core" }, "bugs": { - "url": "https://github.com/vercel-labs/webreel/issues" + "url": "https://github.com/lgariv-dn/webreel/issues" }, "keywords": [ "video", @@ -37,7 +37,9 @@ }, "files": [ "dist", - "assets" + "assets", + "LICENSE", + "README.md" ], "scripts": { "clean": "rm -rf dist", diff --git a/packages/@webreel/core/src/__tests__/revealObserver.test.ts b/packages/@webreel/core/src/__tests__/revealObserver.test.ts new file mode 100644 index 0000000..4803b90 --- /dev/null +++ b/packages/@webreel/core/src/__tests__/revealObserver.test.ts @@ -0,0 +1,90 @@ +import { describe, it, expect } from "vitest"; +import { installRevealObserver, collectReveals, __TESTING__ } from "../revealObserver.js"; + +interface Call { + expression: string; +} + +function createMockClient( + response: { value?: unknown } | (() => { value?: unknown }) | Error, +) { + const calls: Call[] = []; + const client = { + Runtime: { + evaluate: async (params: { expression: string }) => { + calls.push({ expression: params.expression }); + if (response instanceof Error) throw response; + const r = typeof response === "function" ? response() : response; + return { result: r }; + }, + }, + } as never; + return { client, calls }; +} + +describe("installRevealObserver", () => { + it("returns a handle when the page returns a number", async () => { + const { client } = createMockClient({ value: 42 }); + const handle = await installRevealObserver(client); + expect(handle).toEqual({ id: 42 }); + }); + + it("returns null when the page returns a non-number", async () => { + const { client } = createMockClient({ value: null }); + const handle = await installRevealObserver(client); + expect(handle).toBeNull(); + }); + + it("returns null when Runtime.evaluate throws", async () => { + const { client } = createMockClient(new Error("cdp failed")); + const handle = await installRevealObserver(client); + expect(handle).toBeNull(); + }); + + it("evaluates the MutationObserver install IIFE", async () => { + const { client, calls } = createMockClient({ value: 1 }); + await installRevealObserver(client); + expect(calls).toHaveLength(1); + expect(calls[0].expression).toContain("MutationObserver"); + expect(calls[0].expression).toContain("__wrReveals"); + expect(calls[0].expression).toContain("observer.observe"); + }); +}); + +describe("collectReveals", () => { + it("returns the array of bounding boxes the page yields", async () => { + const boxes = [{ x: 10, y: 20, width: 100, height: 50 }]; + const { client, calls } = createMockClient({ value: boxes }); + const result = await collectReveals(client, { id: 7 }); + expect(result).toEqual(boxes); + // The collect expression must reference the handle id so the page + // script finds the right observer state. + expect(calls[0].expression).toContain("(7)"); + expect(calls[0].expression).toContain("disconnect"); + expect(calls[0].expression).toContain("getBoundingClientRect"); + }); + + it("returns empty array when the page returns non-array", async () => { + const { client } = createMockClient({ value: "oops" }); + const result = await collectReveals(client, { id: 1 }); + expect(result).toEqual([]); + }); + + it("returns empty array when Runtime.evaluate throws", async () => { + const { client } = createMockClient(new Error("cdp failed")); + const result = await collectReveals(client, { id: 1 }); + expect(result).toEqual([]); + }); +}); + +describe("reveal scripts (sanity check on string contents)", () => { + it("INSTALL_SCRIPT filters overlay elements and tracks visibility", () => { + expect(__TESTING__.INSTALL_SCRIPT).toContain("preVisible"); + expect(__TESTING__.INSTALL_SCRIPT).toContain("attributeFilter"); + }); + + it("COLLECT_SCRIPT filters __demo- and tiny mutations", () => { + expect(__TESTING__.COLLECT_SCRIPT).toContain("__demo-"); + expect(__TESTING__.COLLECT_SCRIPT).toContain("MIN_AREA"); + }); +}); diff --git a/packages/@webreel/core/src/autozoom.test.ts b/packages/@webreel/core/src/autozoom.test.ts new file mode 100644 index 0000000..54d87ad --- /dev/null +++ b/packages/@webreel/core/src/autozoom.test.ts @@ -0,0 +1,284 @@ +import { describe, it, expect } from "vitest"; +import { + buildAutoZoomFilter, + computeCropForEvent, + generateZoomKeyframes, + unionBboxes, + type AutoZoomConfig, + type ZoomEvent, +} from "./autozoom.js"; + +const DEFAULTS = { + approachS: 0.5, + settleBeforeS: 0.15, + holdAfterS: 0.3, + releaseS: 0.5, + paddingRatio: 0.3, + minZoomRatio: 0.6, + skipZoomRatio: 0.75, + sessionGapS: 4.0, + minPanS: 0.8, +} as const; + +const VIEWPORT = { width: 1920, height: 1080 }; + +describe("unionBboxes", () => { + it("unions two disjoint boxes", () => { + const u = unionBboxes([ + { x: 0, y: 0, width: 10, height: 10 }, + { x: 100, y: 100, width: 20, height: 20 }, + ]); + expect(u).toEqual({ x: 0, y: 0, width: 120, height: 120 }); + }); + + it("returns the outer box when one contains another", () => { + const u = unionBboxes([ + { x: 0, y: 0, width: 100, height: 100 }, + { x: 10, y: 10, width: 20, height: 20 }, + ]); + expect(u).toEqual({ x: 0, y: 0, width: 100, height: 100 }); + }); + + it("returns null for empty input", () => { + expect(unionBboxes([])).toBeNull(); + }); +}); + +describe("computeCropForEvent", () => { + it("enforces minZoomRatio for tiny targets", () => { + const crop = computeCropForEvent( + { x: 100, y: 100, width: 10, height: 10 }, + VIEWPORT, + DEFAULTS, + ); + expect(crop).not.toBeNull(); + expect(crop!.w).toBeGreaterThanOrEqual(VIEWPORT.width * DEFAULTS.minZoomRatio); + expect(crop!.h).toBeGreaterThanOrEqual(VIEWPORT.height * DEFAULTS.minZoomRatio); + }); + + it("returns null when target fills most of viewport (skipZoomRatio)", () => { + const crop = computeCropForEvent( + { x: 0, y: 0, width: VIEWPORT.width, height: VIEWPORT.height }, + VIEWPORT, + DEFAULTS, + ); + expect(crop).toBeNull(); + }); + + it("matches viewport aspect ratio", () => { + const crop = computeCropForEvent( + { x: 200, y: 200, width: 300, height: 300 }, + VIEWPORT, + DEFAULTS, + ); + const aspect = VIEWPORT.width / VIEWPORT.height; + expect(crop).not.toBeNull(); + expect(crop!.w / crop!.h).toBeCloseTo(aspect, 3); + }); + + it("clamps crop to viewport edges", () => { + const crop = computeCropForEvent( + { x: 0, y: 0, width: 20, height: 20 }, + VIEWPORT, + DEFAULTS, + ); + expect(crop).not.toBeNull(); + expect(crop!.x).toBe(0); + expect(crop!.y).toBe(0); + expect(crop!.x + crop!.w).toBeLessThanOrEqual(VIEWPORT.width); + expect(crop!.y + crop!.h).toBeLessThanOrEqual(VIEWPORT.height); + }); +}); + +describe("generateZoomKeyframes", () => { + const cfg: AutoZoomConfig = { enabled: true }; + + it("returns empty when disabled", () => { + const kf = generateZoomKeyframes( + [{ timeMs: 1000, box: { x: 100, y: 100, width: 200, height: 200 } }], + VIEWPORT, + 5, + { enabled: false }, + ); + expect(kf).toEqual([]); + }); + + it("returns empty when no events", () => { + expect(generateZoomKeyframes([], VIEWPORT, 5, cfg)).toEqual([]); + }); + + it("generates approach/settle/hold/release keyframes for a single event", () => { + const events: ZoomEvent[] = [ + { timeMs: 3000, box: { x: 500, y: 400, width: 200, height: 200 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 10, cfg); + expect(kf.length).toBeGreaterThanOrEqual(5); + expect(kf[0].timeS).toBe(0); + expect(kf[0].w).toBe(VIEWPORT.width); + expect(kf[kf.length - 1].w).toBe(VIEWPORT.width); + }); + + it("skips a click after a url change (navigation)", () => { + const events: ZoomEvent[] = [ + { + timeMs: 1000, + box: { x: 100, y: 100, width: 200, height: 200 }, + url: "https://a.test/", + }, + { + timeMs: 3000, + box: { x: 500, y: 500, width: 200, height: 200 }, + url: "https://b.test/", + }, + ]; + const kfBoth = generateZoomKeyframes(events, VIEWPORT, 10, cfg); + const kfFirstOnly = generateZoomKeyframes([events[0]], VIEWPORT, 10, cfg); + expect(kfBoth.length).toBe(kfFirstOnly.length); + }); + + it("merges events within sessionGapS into one session (no release between)", () => { + const events: ZoomEvent[] = [ + { timeMs: 2000, box: { x: 100, y: 100, width: 200, height: 200 } }, + { timeMs: 3000, box: { x: 900, y: 500, width: 200, height: 200 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 10, cfg); + const zoomedOuts = kf.filter((k) => k.w === VIEWPORT.width); + // Two events, 1s gap (< sessionGapS=1.2) → single session → 3 full-view + // keyframes (initial, approach start, final release). + expect(zoomedOuts.length).toBe(3); + }); + + it("starts a new session when gap exceeds sessionGapS", () => { + const events: ZoomEvent[] = [ + { timeMs: 1000, box: { x: 100, y: 100, width: 200, height: 200 } }, + { timeMs: 9000, box: { x: 900, y: 500, width: 200, height: 200 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 15, cfg); + const zoomedOuts = kf.filter((k) => k.w === VIEWPORT.width); + // Two separate sessions → 5 full-view keyframes (each session adds + // approach-start + release, plus one shared initial at t=0). + expect(zoomedOuts.length).toBe(5); + }); + + it("emits monotonically-increasing keyframe times", () => { + const events: ZoomEvent[] = [ + { timeMs: 1000, box: { x: 100, y: 100, width: 200, height: 200 } }, + { timeMs: 2000, box: { x: 900, y: 500, width: 200, height: 200 } }, + { timeMs: 8000, box: { x: 400, y: 300, width: 200, height: 200 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 20, cfg); + for (let i = 1; i < kf.length; i++) { + expect(kf[i].timeS).toBeGreaterThanOrEqual(kf[i - 1].timeS); + } + }); + + it("keeps sessions separate across medium gaps (no mergeBuffer)", () => { + // 6s gap: clearly beyond sessionGapS=4.0, so the sessions must NOT merge. + // We want the camera to fully release between unrelated events. + const events: ZoomEvent[] = [ + { timeMs: 1000, box: { x: 100, y: 100, width: 200, height: 200 } }, + { timeMs: 7000, box: { x: 900, y: 500, width: 200, height: 200 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 12, cfg); + const zoomedOuts = kf.filter((k) => k.w === VIEWPORT.width); + expect(zoomedOuts.length).toBe(5); + }); + + it("skips intermediate targets whose pan would be shorter than minPanS", () => { + // Three events: A (t=2), B (t=2.5, 0.5s gap — below minPanS=0.8 after + // hold), C (t=5, separate session). B gets dropped; A and C each form + // their own session with distinct crops. + const events: ZoomEvent[] = [ + { timeMs: 2000, box: { x: 200, y: 100, width: 200, height: 50 } }, + { timeMs: 2500, box: { x: 200, y: 300, width: 200, height: 50 } }, + { timeMs: 5000, box: { x: 1000, y: 600, width: 200, height: 50 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 10, cfg); + const distinctCrops = new Set( + kf.map((k) => `${k.x.toFixed(0)},${k.y.toFixed(0)},${k.w.toFixed(0)}`), + ); + expect(distinctCrops.size).toBe(3); + }); + + it("uses last-kept crop for release when the final event is skipped by minPanS", () => { + // Session with two events close together: B (t=4.5) and C (t=4.9, 0.4s + // gap). C's pan is too short (< minPanS) so it's skipped; release should + // use B's crop, not C's. Event A at t=2 is in its own earlier session. + const events: ZoomEvent[] = [ + { timeMs: 2000, box: { x: 100, y: 100, width: 200, height: 50 } }, + { timeMs: 4500, box: { x: 800, y: 400, width: 200, height: 50 } }, + { timeMs: 4900, box: { x: 1500, y: 900, width: 200, height: 50 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 8, cfg); + const lastCropKf = [...kf].reverse().find((k) => k.w !== VIEWPORT.width)!; + // B is centered around x=800; C around x=1500. The last kept crop should + // be positioned for B, not C — its left edge should be well under 1000. + expect(lastCropKf.x).toBeLessThan(1000); + }); + + it("harmonizes crop size within a session so zoom level stays constant", () => { + // A big target and a small target in the same session → small one should + // inherit the big one's crop size (so the zoom doesn't jump). + const events: ZoomEvent[] = [ + { timeMs: 1000, box: { x: 100, y: 100, width: 800, height: 400 } }, + { timeMs: 2000, box: { x: 1500, y: 800, width: 80, height: 30 } }, + ]; + const kf = generateZoomKeyframes(events, VIEWPORT, 5, cfg); + const sessionCrops = kf.filter((k) => k.w !== VIEWPORT.width); + const widths = new Set(sessionCrops.map((k) => k.w)); + expect(widths.size).toBe(1); + }); +}); + +describe("buildAutoZoomFilter", () => { + it("returns null when disabled", () => { + expect(buildAutoZoomFilter([], VIEWPORT, 1, 5, 60, { enabled: false })).toBeNull(); + }); + + it("returns null when no events produce keyframes", () => { + expect(buildAutoZoomFilter([], VIEWPORT, 1, 5, 60, { enabled: true })).toBeNull(); + }); + + it("emits a zoompan filter string with expected params", () => { + const events: ZoomEvent[] = [ + { timeMs: 2000, box: { x: 200, y: 200, width: 200, height: 200 } }, + ]; + const filter = buildAutoZoomFilter(events, VIEWPORT, 1, 10, 60, { + enabled: true, + }); + expect(filter).not.toBeNull(); + expect(filter!).toMatch(/^zoompan=z='/); + expect(filter!).toContain(`s=${VIEWPORT.width}x${VIEWPORT.height}`); + expect(filter!).toContain("fps=60"); + }); + + it("emits smoothstep easing (register-based) in the filter expression", () => { + const events: ZoomEvent[] = [ + { timeMs: 2000, box: { x: 200, y: 200, width: 200, height: 200 } }, + ]; + const filter = buildAutoZoomFilter(events, VIEWPORT, 1, 10, 60, { + enabled: true, + }); + expect(filter).not.toBeNull(); + // st(0, progress) and st(1, smoothstepped) and ld(1) usage + expect(filter!).toContain("st(0,"); + expect(filter!).toContain("st(1,ld(0)*ld(0)*(3-2*ld(0)))"); + expect(filter!).toContain("*ld(1)"); + }); + + it("scales event boxes by cssZoom before keyframe generation", () => { + // Box big enough that the minZoomRatio floor doesn't clip both outputs to + // the same crop — otherwise cssZoom=1 and cssZoom=2 both bottom out at the + // minimum and look identical. + const events: ZoomEvent[] = [ + { timeMs: 2000, box: { x: 100, y: 100, width: 500, height: 300 } }, + ]; + const filterAt1 = buildAutoZoomFilter(events, VIEWPORT, 1, 10, 60, { + enabled: true, + }); + const filterAt2 = buildAutoZoomFilter(events, VIEWPORT, 2, 10, 60, { + enabled: true, + }); + expect(filterAt1).not.toBe(filterAt2); + }); +}); diff --git a/packages/@webreel/core/src/autozoom.ts b/packages/@webreel/core/src/autozoom.ts new file mode 100644 index 0000000..c9d5ee5 --- /dev/null +++ b/packages/@webreel/core/src/autozoom.ts @@ -0,0 +1,339 @@ +import type { BoundingBox } from "./types.js"; + +export interface AutoZoomConfig { + enabled: boolean; + approachS?: number; + settleBeforeS?: number; + holdAfterS?: number; + releaseS?: number; + paddingRatio?: number; + minZoomRatio?: number; + skipZoomRatio?: number; + sessionGapS?: number; + minPanS?: number; +} + +export interface ZoomEvent { + timeMs: number; + box: BoundingBox; + url?: string; + // Optional: extend the camera hold until at least this time. Used for type + // actions where `timeMs` anchors on the input click (so the camera arrives + // in time) but the hold must cover the typing span, which ends later. + holdUntilMs?: number; +} + +export interface ZoomKeyframe { + timeS: number; + x: number; + y: number; + w: number; + h: number; +} + +type ResolvedConfig = { + approachS: number; + settleBeforeS: number; + holdAfterS: number; + releaseS: number; + paddingRatio: number; + minZoomRatio: number; + skipZoomRatio: number; + sessionGapS: number; + minPanS: number; +}; + +const DEFAULTS: ResolvedConfig = { + approachS: 0.5, + settleBeforeS: 0.15, + holdAfterS: 0.3, + releaseS: 0.5, + paddingRatio: 0.3, + minZoomRatio: 0.6, + skipZoomRatio: 0.75, + sessionGapS: 4.0, + minPanS: 0.8, +}; + +export function unionBboxes(boxes: BoundingBox[]): BoundingBox | null { + if (boxes.length === 0) return null; + let minX = Infinity; + let minY = Infinity; + let maxX = -Infinity; + let maxY = -Infinity; + for (const b of boxes) { + if (b.x < minX) minX = b.x; + if (b.y < minY) minY = b.y; + if (b.x + b.width > maxX) maxX = b.x + b.width; + if (b.y + b.height > maxY) maxY = b.y + b.height; + } + return { x: minX, y: minY, width: maxX - minX, height: maxY - minY }; +} + +function centerCropWithin( + cx: number, + cy: number, + w: number, + h: number, + viewport: { width: number; height: number }, +): { x: number; y: number; w: number; h: number } { + return { + w, + h, + x: Math.max(0, Math.min(viewport.width - w, cx - w / 2)), + y: Math.max(0, Math.min(viewport.height - h, cy - h / 2)), + }; +} + +export function computeCropForEvent( + box: BoundingBox, + viewport: { width: number; height: number }, + cfg: ResolvedConfig, +): { x: number; y: number; w: number; h: number } | null { + let w = box.width * (1 + 2 * cfg.paddingRatio); + let h = box.height * (1 + 2 * cfg.paddingRatio); + + w = Math.max(w, viewport.width * cfg.minZoomRatio); + h = Math.max(h, viewport.height * cfg.minZoomRatio); + + const aspect = viewport.width / viewport.height; + if (w / h > aspect) h = w / aspect; + else w = h * aspect; + + w = Math.min(w, viewport.width); + h = Math.min(h, viewport.height); + + if ( + w >= viewport.width * cfg.skipZoomRatio && + h >= viewport.height * cfg.skipZoomRatio + ) { + return null; + } + + const cx = box.x + box.width / 2; + const cy = box.y + box.height / 2; + return centerCropWithin(cx, cy, w, h, viewport); +} + +export function generateZoomKeyframes( + events: ZoomEvent[], + viewport: { width: number; height: number }, + durationS: number, + userCfg: AutoZoomConfig, +): ZoomKeyframe[] { + if (!userCfg.enabled) return []; + const cfg: ResolvedConfig = { ...DEFAULTS, ...userCfg }; + + interface Target { + timeS: number; + holdUntilS: number; + box: BoundingBox; + crop: { x: number; y: number; w: number; h: number }; + } + + const targets: Target[] = []; + let prevUrl: string | undefined; + for (const e of events) { + if (e.url && prevUrl && e.url !== prevUrl) { + prevUrl = e.url; + continue; + } + const crop = computeCropForEvent(e.box, viewport, cfg); + prevUrl = e.url ?? prevUrl; + if (!crop) continue; + const timeS = e.timeMs / 1000; + const holdUntilS = Math.max(timeS, (e.holdUntilMs ?? e.timeMs) / 1000); + targets.push({ timeS, holdUntilS, box: e.box, crop }); + } + + if (targets.length === 0) return []; + + const full = { x: 0, y: 0, w: viewport.width, h: viewport.height }; + + // Group targets by sessionGapS. Events within a short gap (default 1.2s — + // button-click → modal case) form one session; anything further apart gets + // its own pulse so the camera releases all the way back to wide between. + const sessions: Target[][] = [[targets[0]]]; + for (let i = 1; i < targets.length; i++) { + const prev = targets[i - 1]; + const curr = targets[i]; + if (curr.timeS - prev.timeS <= cfg.sessionGapS) { + sessions[sessions.length - 1].push(curr); + } else { + sessions.push(curr === prev ? [] : [curr]); + } + } + + // Spatial sub-grouping within a session. We greedily extend the current + // sub-group with each next target as long as the union bbox of the + // sub-group still fits into a crop (doesn't trigger skipZoomRatio). If it + // won't fit, we close the current sub-group and start a new one with this + // target. Each sub-group shares ONE union crop, so the camera holds stable + // within a sub-group and pans between sub-groups. This matches Cursor: + // button+menu or trigger+modal stay in one frame; distinct regions of the + // page get distinct zoom positions. + type Group = { targets: Target[]; crop: Target["crop"] }; + for (const session of sessions) { + if (session.length < 2) continue; + const first = session[0]; + const groups: Group[] = [{ targets: [first], crop: first.crop }]; + for (let i = 1; i < session.length; i++) { + const curr = session[i]; + const last = groups[groups.length - 1]; + const candidateBoxes = last.targets.map((t) => t.box).concat([curr.box]); + const candidateUnion = unionBboxes(candidateBoxes); + const candidateCrop = candidateUnion + ? computeCropForEvent(candidateUnion, viewport, cfg) + : null; + if (candidateCrop) { + last.targets.push(curr); + last.crop = candidateCrop; + } else { + groups.push({ targets: [curr], crop: curr.crop }); + } + } + for (const g of groups) for (const t of g.targets) t.crop = g.crop; + + // Size-harmonize across sub-groups so crop zoom stays constant while the + // camera pans between them. + let maxW = 0; + let maxH = 0; + for (const t of session) { + if (t.crop.w > maxW) maxW = t.crop.w; + if (t.crop.h > maxH) maxH = t.crop.h; + } + for (const t of session) { + if (t.crop.w === maxW && t.crop.h === maxH) continue; + const cx = t.crop.x + t.crop.w / 2; + const cy = t.crop.y + t.crop.h / 2; + t.crop = centerCropWithin(cx, cy, maxW, maxH, viewport); + } + } + + const kf: ZoomKeyframe[] = [{ timeS: 0, ...full }]; + for (const session of sessions) { + const first = session[0]; + const actualLast = session[session.length - 1]; + + const settleTime = first.timeS - cfg.settleBeforeS; + const approachStart = Math.max(0, settleTime - cfg.approachS); + kf.push({ timeS: approachStart, ...full }); + kf.push({ timeS: Math.max(0, settleTime), ...first.crop }); + + // Track the last target we actually panned to. Skip intermediate targets + // whose pan duration would be shorter than cfg.minPanS — blink-and-miss + // transitions that add visual noise without being readable. Also skip + // when the next target's crop is identical to the current one (the + // union-crop case): no pan to emit, just keep tracking lastKept so the + // release time extends to cover the session's full span. + let lastKept = first; + for (let i = 1; i < session.length; i++) { + const curr = session[i]; + const sameCrop = + curr.crop.x === lastKept.crop.x && + curr.crop.y === lastKept.crop.y && + curr.crop.w === lastKept.crop.w && + curr.crop.h === lastKept.crop.h; + if (sameCrop) { + lastKept = curr; + continue; + } + const gap = curr.timeS - lastKept.timeS; + const arriveBy = curr.timeS - cfg.settleBeforeS; + const holdEnd = Math.min(lastKept.timeS + gap * 0.4, arriveBy - 0.1); + const panDuration = arriveBy - holdEnd; + if (panDuration < cfg.minPanS) continue; + + if (holdEnd > lastKept.timeS + 0.05) { + kf.push({ timeS: holdEnd, ...lastKept.crop }); + } + kf.push({ timeS: Math.max(holdEnd + 0.05, arriveBy), ...curr.crop }); + lastKept = curr; + } + + // Release uses the session's true last event for TIMING (so hold is + // sized correctly) but the last position we actually panned to for + // the CROP (otherwise release would teleport to a never-visited spot). + // holdUntilS carries a per-event hold extension (e.g. typing spans from + // click to last keystroke — the camera should stay on the field until + // typing actually ends, not just for holdAfterS after the click). + const holdEnd = actualLast.holdUntilS + cfg.holdAfterS; + const releaseEnd = holdEnd + cfg.releaseS; + kf.push({ timeS: holdEnd, ...lastKept.crop }); + kf.push({ timeS: releaseEnd, ...full }); + } + + return kf; +} + +export function buildAutoZoomFilter( + events: ZoomEvent[], + viewport: { width: number; height: number }, + cssZoom: number, + durationS: number, + fps: number, + userCfg: AutoZoomConfig, +): string | null { + if (!userCfg.enabled || events.length === 0) return null; + + const scaled: ZoomEvent[] = events.map((e) => ({ + ...e, + box: { + x: e.box.x * cssZoom, + y: e.box.y * cssZoom, + width: e.box.width * cssZoom, + height: e.box.height * cssZoom, + }, + })); + + const kf = generateZoomKeyframes(scaled, viewport, durationS, userCfg); + if (kf.length < 2) return null; + + if (process.env.WEBREEL_DEBUG_ZOOM) { + for (const k of kf) { + const z = (viewport.width / Math.max(1, k.w)).toFixed(2); + console.error( + `kf t=${k.timeS.toFixed(2)}s z=${z}x crop=${k.x.toFixed(0)},${k.y.toFixed(0)} ${k.w.toFixed(0)}×${k.h.toFixed(0)}`, + ); + } + } + + const zExpr = easedBetweens(kf, (k) => viewport.width / Math.max(1, k.w)); + const xExpr = easedBetweens(kf, (k) => k.x); + const yExpr = easedBetweens(kf, (k) => k.y); + return `zoompan=z='${zExpr}':x='${xExpr}':y='${yExpr}':d=1:s=${viewport.width}x${viewport.height}:fps=${fps}`; +} + +// Smoothstep easing between keyframes: p² × (3 − 2p). Uses FFmpeg expression +// register slots to compute the eased progress once per frame, then lerp at +// the eased position. Each segment is a single smoothstep (velocity zero at +// both ends) — since we no longer insert mid-motion waypoints, every segment +// is a standalone motion from a hold to the next hold, which is the shape +// smoothstep handles best. +function easedBetweens(kf: ZoomKeyframe[], val: (k: ZoomKeyframe) => number): string { + if (kf.length === 1) return val(kf[0]).toFixed(4); + let expr = ""; + let segments = 0; + for (let i = 0; i < kf.length - 1; i++) { + const a = kf[i]; + const b = kf[i + 1]; + if (b.timeS === a.timeS) continue; + const va = val(a); + const vb = val(b); + const dt = b.timeS - a.timeS; + const t0 = a.timeS.toFixed(3); + const t1 = b.timeS.toFixed(3); + const dtS = dt.toFixed(3); + const v0 = va.toFixed(4); + const delta = (vb - va).toFixed(4); + const seg = + `if(between(in_time,${t0},${t1}),` + + `0*st(0,(in_time-${t0})/${dtS})+` + + `0*st(1,ld(0)*ld(0)*(3-2*ld(0)))+` + + `${v0}+(${delta})*ld(1)`; + expr = expr ? `${expr},${seg}` : seg; + segments++; + } + const tail = val(kf[kf.length - 1]).toFixed(4); + return `${expr},${tail}${")".repeat(segments)}`; +} diff --git a/packages/@webreel/core/src/compositor.ts b/packages/@webreel/core/src/compositor.ts index 5645266..4bc6faf 100644 --- a/packages/@webreel/core/src/compositor.ts +++ b/packages/@webreel/core/src/compositor.ts @@ -21,6 +21,7 @@ interface OverlayContext { export interface ComposeOptions { sfx?: SfxConfig; crf?: number; + zoomFilter?: string; } export async function compose( @@ -53,6 +54,7 @@ export async function compose( zoom, tempComposed, crf, + options?.zoomFilter, ); const ext = extname(outputPath).toLowerCase(); @@ -105,15 +107,131 @@ async function compositeFrames( zoom: number, outputPath: string, crf: number, + zoomFilter?: string, ): Promise { const { width, height, fps } = timeline; + // Build the overlay context once — sharp metadata and cursor-scale caches + // are identical across stages, so we share the context. + const cursorMeta = await sharp(cursorPng).metadata(); + if (!cursorMeta.width || !cursorMeta.height) { + throw new Error("Failed to read cursor image dimensions from sharp metadata"); + } + const cursorWidth = cursorMeta.width; + const cursorHeight = cursorMeta.height; + + const scaledCursorCache = new Map(); + scaledCursorCache.set(100, cursorPng); + + const getScaledCursor = async (scale: number): Promise => { + const key = Math.round(scale * 100); + let cached = scaledCursorCache.get(key); + if (cached) return cached; + const sw = Math.max(1, Math.round(cursorWidth * scale)); + const sh = Math.max(1, Math.round(cursorHeight * scale)); + cached = await sharp(cursorPng).resize(sw, sh).png().toBuffer(); + scaledCursorCache.set(key, cached); + return cached; + }; + + const hotspot = timeline.theme.cursorHotspot ?? "top-left"; + const hotspotOffsetX = hotspot === "center" ? Math.round(cursorWidth / 2) : 0; + const hotspotOffsetY = hotspot === "center" ? Math.round(cursorHeight / 2) : 0; + + const ctx: OverlayContext = { + cursorPng, + cursorWidth, + cursorHeight, + hotspotOffsetX, + hotspotOffsetY, + getScaledCursor, + zoom, + hudConfig: timeline.theme.hud, + }; + + // Pipeline layering: + // - No autozoom: single overlay pass draws cursor+HUD on raw → output. + // - Autozoom: three sequential ffmpeg invocations. Stage A overlays + // cursor-only (not HUD) on raw. Stage B applies zoompan + // on the cursor-overlaid intermediate. Stage C overlays + // HUD on the zoomed frame. HUD stays at the final + // viewport coordinates regardless of how the camera + // crops/scales — captions never get cropped by zoom. + // + // Why three stages instead of one? (1) zoompan + image2pipe in the same + // filter_complex deadlocks when the pipe reader can't drain fast enough + // (see detailed note in the original single-pass version of this file). + // (2) HUD on top of the zoomed frame must run after zoompan or it gets + // cropped out of the camera window. + if (!zoomFilter) { + await runOverlayStage( + ffmpegPath, + cleanVideoPath, + timeline, + ctx, + "both", + outputPath, + crf, + fps, + width, + height, + ); + return; + } + + const cursorStagePath = resolve(homedir(), ".webreel", `_cursor_${Date.now()}.mp4`); + const zoomStagePath = resolve(homedir(), ".webreel", `_zoom_${Date.now()}.mp4`); + + try { + await runOverlayStage( + ffmpegPath, + cleanVideoPath, + timeline, + ctx, + "cursor", + cursorStagePath, + crf, + fps, + width, + height, + ); + await applyZoomPass(ffmpegPath, cursorStagePath, zoomFilter, zoomStagePath, crf, fps); + await runOverlayStage( + ffmpegPath, + zoomStagePath, + timeline, + ctx, + "hud", + outputPath, + crf, + fps, + width, + height, + ); + } finally { + rmSync(cursorStagePath, { force: true }); + rmSync(zoomStagePath, { force: true }); + } +} + +async function runOverlayStage( + ffmpegPath: string, + inputPath: string, + timeline: TimelineData, + ctx: OverlayContext, + layer: OverlayLayer, + outputPath: string, + crf: number, + fps: number, + width: number, + height: number, +): Promise { const ffmpeg = spawn( ffmpegPath, [ "-y", "-i", - cleanVideoPath, + inputPath, "-f", "image2pipe", "-framerate", @@ -147,51 +265,46 @@ async function compositeFrames( { stdio: ["pipe", "pipe", "pipe"] }, ); - const cursorMeta = await sharp(cursorPng).metadata(); - if (!cursorMeta.width || !cursorMeta.height) { - throw new Error("Failed to read cursor image dimensions from sharp metadata"); - } - const cursorWidth = cursorMeta.width; - const cursorHeight = cursorMeta.height; - - const scaledCursorCache = new Map(); - scaledCursorCache.set(100, cursorPng); - - const getScaledCursor = async (scale: number): Promise => { - const key = Math.round(scale * 100); - let cached = scaledCursorCache.get(key); - if (cached) return cached; - const sw = Math.max(1, Math.round(cursorWidth * scale)); - const sh = Math.max(1, Math.round(cursorHeight * scale)); - cached = await sharp(cursorPng).resize(sw, sh).png().toBuffer(); - scaledCursorCache.set(key, cached); - return cached; - }; - - const hotspot = timeline.theme.cursorHotspot ?? "top-left"; - const hotspotOffsetX = hotspot === "center" ? Math.round(cursorWidth / 2) : 0; - const hotspotOffsetY = hotspot === "center" ? Math.round(cursorHeight / 2) : 0; - - const ctx: OverlayContext = { - cursorPng, - cursorWidth, - cursorHeight, - hotspotOffsetX, - hotspotOffsetY, - getScaledCursor, - zoom, - hudConfig: timeline.theme.hud, - }; - const overlayCache = new Map(); const hudCache = new Map(); const stdin = ffmpeg.stdin; if (!stdin) throw new Error("ffmpeg process has no stdin pipe"); - const drain = (): Promise => new Promise((res) => stdin.once("drain", res)); + // ffmpeg may close its read side of the pipe early when `overlay=shortest=1` + // truncates to the shorter input (e.g., raw video has fewer frames than the + // timeline). Node surfaces that as an EPIPE 'error' event on stdin and an + // uncaught error will crash the process. Swallow it and stop writing — the + // frames already fed are enough for ffmpeg to produce output up to the + // truncation point. + let stdinClosed = false; + stdin.on("error", (err) => { + const nodeErr = err as NodeJS.ErrnoException; + if (nodeErr && nodeErr.code === "EPIPE") { + stdinClosed = true; + } else { + throw err; + } + }); + + // Wait for the stream's buffer to drain. Races against the error event so + // we don't hang forever if ffmpeg closed the read side before drain fires. + const drain = (): Promise => + new Promise((res) => { + const onDrain = () => { + stdin.off("error", onError); + res(); + }; + const onError = () => { + stdin.off("drain", onDrain); + res(); + }; + stdin.once("drain", onDrain); + stdin.once("error", onError); + }); for (let i = 0; i < timeline.frames.length; i++) { + if (stdinClosed) break; const frame = timeline.frames[i]; const overlayPng = await renderOverlayFrame( frame, @@ -200,13 +313,80 @@ async function compositeFrames( ctx, overlayCache, hudCache, + layer, ); + if (stdinClosed) break; const ok = stdin.write(overlayPng); if (!ok) await drain(); } - stdin.end(); + if (!stdin.writableEnded) { + try { + stdin.end(); + } catch { + // stream may already be errored — safe to ignore + } + } + + const stderrChunks: Buffer[] = []; + ffmpeg.stderr?.on("data", (chunk: Buffer) => stderrChunks.push(chunk)); + + await new Promise((resolveAll, rejectAll) => { + ffmpeg.on("close", (code) => { + if (code === 0) { + resolveAll(); + } else { + const stderr = Buffer.concat(stderrChunks).toString().slice(-2000); + rejectAll( + new Error( + `Overlay-stage ffmpeg (layer=${layer}) exited with code ${code}${stderr ? `:\n${stderr}` : ""}`, + ), + ); + } + }); + ffmpeg.on("error", rejectAll); + }); +} + +async function applyZoomPass( + ffmpegPath: string, + inputPath: string, + zoomFilter: string, + outputPath: string, + crf: number, + fps: number, +): Promise { + const ffmpeg = spawn( + ffmpegPath, + [ + "-y", + "-i", + inputPath, + "-vf", + zoomFilter, + "-c:v", + "libx264", + "-preset", + "ultrafast", + "-crf", + String(crf), + "-pix_fmt", + "yuv420p", + "-color_primaries", + "bt709", + "-color_trc", + "bt709", + "-colorspace", + "bt709", + "-movflags", + "+faststart", + "-r", + String(fps), + outputPath, + ], + { stdio: ["ignore", "pipe", "pipe"] }, + ); const stderrChunks: Buffer[] = []; ffmpeg.stderr?.on("data", (chunk: Buffer) => stderrChunks.push(chunk)); @@ -219,7 +399,7 @@ async function compositeFrames( const stderr = Buffer.concat(stderrChunks).toString().slice(-2000); rejectAll( new Error( - `Compositor ffmpeg exited with code ${code}${stderr ? `:\n${stderr}` : ""}`, + `Zoom-pass ffmpeg exited with code ${code}${stderr ? `:\n${stderr}` : ""}`, ), ); } @@ -228,6 +408,8 @@ async function compositeFrames( }); } +type OverlayLayer = "both" | "cursor" | "hud"; + async function renderOverlayFrame( frame: TimelineData["frames"][number], width: number, @@ -235,12 +417,13 @@ async function renderOverlayFrame( ctx: OverlayContext, cache: Map, hudCache: Map, + layer: OverlayLayer = "both", ): Promise { const cx = Math.round(frame.cursor.x * ctx.zoom * 10) / 10; const cy = Math.round(frame.cursor.y * ctx.zoom * 10) / 10; const scale = frame.cursor.scale; const hudKey = frame.hud ? frame.hud.labels.join("|") : ""; - const cacheKey = `${cx},${cy},${scale},${hudKey}`; + const cacheKey = `${layer}:${cx},${cy},${scale},${hudKey}`; const cached = cache.get(cacheKey); if (cached) return cached; @@ -252,7 +435,10 @@ async function renderOverlayFrame( const cursorVisible = icx >= -ctx.cursorWidth && icx < width && icy >= -ctx.cursorHeight && icy < height; - if (cursorVisible) { + const wantCursor = layer !== "hud"; + const wantHud = layer !== "cursor"; + + if (wantCursor && cursorVisible) { const cursorImg = scale !== 1 ? await ctx.getScaledCursor(scale) : ctx.cursorPng; const left = Math.max(0, icx); const top = Math.max(0, icy); @@ -262,7 +448,7 @@ async function renderOverlayFrame( } } - if (frame.hud && frame.hud.labels.length > 0) { + if (wantHud && frame.hud && frame.hud.labels.length > 0) { const hudOverlay = await renderHudOverlay( frame.hud.labels, width, @@ -346,7 +532,11 @@ async function renderHudOverlay( .replace(//g, ">"); - const svgOverlay = ` + const margin = Math.round(48 * zoom); + const maxHudWidth = Math.max(100, viewportWidth - margin * 2); + const renderedHudWidth = Math.min(hudWidth, maxHudWidth); + + const svgOverlay = ` `; const hudPng = await sharp(Buffer.from(svgOverlay)).png().toBuffer(); - const left = Math.round((viewportWidth - hudWidth) / 2); - const margin = Math.round(48 * zoom); + const left = Math.round((viewportWidth - renderedHudWidth) / 2); const top = hudConfig.position === "top" ? margin : viewportHeight - hudHeight - margin; const result: sharp.OverlayOptions = { input: hudPng, left, top }; diff --git a/packages/@webreel/core/src/cursor-motion.ts b/packages/@webreel/core/src/cursor-motion.ts index 609ef68..5b2ef8b 100644 --- a/packages/@webreel/core/src/cursor-motion.ts +++ b/packages/@webreel/core/src/cursor-motion.ts @@ -7,9 +7,17 @@ import type { RecordingContext } from "./actions.js"; * Returns milliseconds. Tuned so short moves feel quick but not * instant, and long cross-screen moves have enough frames to * appear smooth at the target capture rate. + * + * Constants are 2× the original upstream values. Before the + * tickDuplicate path-advance fix (see timeline.ts), the capture + * loop consumed path positions at half the rate NUM_STEPS assumed, + * so moves effectively rendered at 2× this duration. Upstream tuned + * the formula against that buggy rendering. With the fix in place, + * keeping the old constants made moves feel ~2× too fast. These + * scaled-up constants restore the perceived cadence. */ function moveDuration(distance: number): number { - return 180 + 16 * Math.sqrt(distance) + (Math.random() - 0.5) * 30; + return 360 + 32 * Math.sqrt(distance) + (Math.random() - 0.5) * 60; } /** @@ -99,7 +107,14 @@ export async function animateMoveTo( if (ctx.isRecording && ctx.timeline) { ctx.timeline.setCursorPath(positions); - await new Promise((r) => setTimeout(r, NUM_STEPS * CAPTURE_CYCLE_MS)); + // Wait until the capture loop has actually consumed the full path — i.e. + // until the cursor is visually at (toX, toY). This is hardware-independent + // and eliminates the gap between "next action fires" and "cursor arrived", + // which previously caused click feedback to appear delayed relative to the + // UI's response. A setTimeout-based wait can't match both the frameSlots=2 + // case (fast consumption, ~17.5 ms/step) and the frameSlots=1 case (slow + // consumption, ~35 ms/step) without over- or under-shooting on some hardware. + await ctx.timeline.waitForPathComplete(); await client.Input.dispatchMouseEvent({ type: "mouseMoved", @@ -127,7 +142,12 @@ export async function animateMoveTo( })()`, }); - await new Promise((r) => setTimeout(r, NUM_STEPS * CAPTURE_CYCLE_MS)); + // Wait for the capture loop to consume the path. Each capture cycle takes + // ~CAPTURE_CYCLE_MS and consumes TWO path entries (one via tick, one via + // tickDuplicate), so total time ≈ NUM_STEPS / 2 * CAPTURE_CYCLE_MS = + // NUM_STEPS * FRAME_MS. Using FRAME_MS also correctly scales to + // moveDuration() since NUM_STEPS = duration / FRAME_MS by construction. + await new Promise((r) => setTimeout(r, NUM_STEPS * FRAME_MS)); await client.Input.dispatchMouseEvent({ type: "mouseMoved", diff --git a/packages/@webreel/core/src/index.ts b/packages/@webreel/core/src/index.ts index 95d3f58..c7d78d6 100644 --- a/packages/@webreel/core/src/index.ts +++ b/packages/@webreel/core/src/index.ts @@ -42,3 +42,17 @@ export { compose, type ComposeOptions } from "./compositor.js"; export { ensureFfmpeg, FFMPEG_CACHE_DIR } from "./ffmpeg.js"; export { extractThumbnail, type SfxConfig } from "./media.js"; export { moveFileSync } from "./fs.js"; +export { + buildAutoZoomFilter, + computeCropForEvent, + generateZoomKeyframes, + unionBboxes, + type AutoZoomConfig, + type ZoomEvent, + type ZoomKeyframe, +} from "./autozoom.js"; +export { + installRevealObserver, + collectReveals, + type RevealObserverHandle, +} from "./revealObserver.js"; diff --git a/packages/@webreel/core/src/revealObserver.ts b/packages/@webreel/core/src/revealObserver.ts new file mode 100644 index 0000000..52e0c3a --- /dev/null +++ b/packages/@webreel/core/src/revealObserver.ts @@ -0,0 +1,146 @@ +import type { BoundingBox, CDPClient } from "./types.js"; + +export interface RevealObserverHandle { + id: number; +} + +// Installed BEFORE an interaction fires. Snapshots which elements are +// currently visible so we can later tell what's newly revealed, then hooks a +// MutationObserver to the document body subtree. Returns a numeric id used +// to identify this observer when we collect. +const INSTALL_SCRIPT = `(() => { + if (!window.__wrReveals) window.__wrReveals = {}; + if (!window.__wrRevealsNextId) window.__wrRevealsNextId = 0; + const id = ++window.__wrRevealsNextId; + const preVisible = new WeakSet(); + const preBounds = new WeakMap(); + // An element is "pre-visible" only if it's BOTH in layout AND actually + // rendered (not hidden by opacity:0, visibility:hidden, display:none). We + // use Element.checkVisibility when available because getBoundingClientRect + // returns a non-zero rect for opacity:0 elements — those are elements we + // DO want to detect as newly-revealed later. + const isActuallyVisible = (el) => { + if (typeof el.checkVisibility === 'function') { + return el.checkVisibility({ + opacityProperty: true, + visibilityProperty: true, + contentVisibilityAuto: true, + }); + } + const r = el.getBoundingClientRect(); + return r.width > 0 && r.height > 0; + }; + const all = document.body.querySelectorAll('*'); + for (const el of all) { + if (isActuallyVisible(el)) { + preVisible.add(el); + const r = el.getBoundingClientRect(); + preBounds.set(el, { x: r.left, y: r.top, w: r.width, h: r.height }); + } + } + const mutated = new Set(); + const observer = new MutationObserver((records) => { + for (const r of records) { + if (r.type === 'childList') { + for (const n of r.addedNodes) if (n.nodeType === 1) mutated.add(n); + } else if (r.type === 'attributes' && r.target.nodeType === 1) { + mutated.add(r.target); + } + } + }); + observer.observe(document.body, { + childList: true, + subtree: true, + attributes: true, + attributeFilter: ['class', 'style', 'hidden', 'aria-hidden', 'aria-expanded', 'open'], + }); + window.__wrReveals[id] = { observer, preVisible, preBounds, mutated }; + return id; +})()`; + +// Called AFTER the interaction + postDelay. Walks the mutated element set, +// decides which are newly visible or grew meaningfully, and returns their +// bounding boxes. Filters out our own recording overlay elements (prefixed +// with "__demo-"). Tiny mutations (< 100 px²) are dropped as noise. +const COLLECT_SCRIPT = `(id) => { + const state = window.__wrReveals && window.__wrReveals[id]; + if (!state) return []; + state.observer.disconnect(); + const MIN_AREA = 100; + const GROWTH_MARGIN = 4; + const reveals = []; + for (const el of state.mutated) { + if (!document.body.contains(el)) continue; + if (el.id && el.id.indexOf('__demo-') === 0) continue; + if (el.closest && el.closest('[id^="__demo-"]')) continue; + const r = el.getBoundingClientRect(); + if (r.width <= 0 || r.height <= 0) continue; + if (r.width * r.height < MIN_AREA) continue; + // Must be actually rendered now (not just laid-out-but-invisible). + if (typeof el.checkVisibility === 'function') { + if (!el.checkVisibility({ opacityProperty: true, visibilityProperty: true })) { + continue; + } + } + const wasVisible = state.preVisible.has(el); + if (!wasVisible) { + reveals.push({ x: r.left, y: r.top, width: r.width, height: r.height }); + continue; + } + const prev = state.preBounds.get(el); + if (!prev) continue; + const grewLeft = prev.x - r.left > GROWTH_MARGIN; + const grewTop = prev.y - r.top > GROWTH_MARGIN; + const grewRight = (r.left + r.width) - (prev.x + prev.w) > GROWTH_MARGIN; + const grewBottom = (r.top + r.height) - (prev.y + prev.h) > GROWTH_MARGIN; + if (grewLeft || grewTop || grewRight || grewBottom) { + reveals.push({ x: r.left, y: r.top, width: r.width, height: r.height }); + } + } + delete window.__wrReveals[id]; + return reveals; +}`; + +export async function installRevealObserver( + client: CDPClient, +): Promise { + try { + const { result } = await client.Runtime.evaluate({ + expression: INSTALL_SCRIPT, + returnByValue: true, + }); + if (typeof result.value === "number") return { id: result.value }; + return null; + } catch { + return null; + } +} + +function isBoundingBox(v: unknown): v is BoundingBox { + if (!v || typeof v !== "object") return false; + const b = v as Record; + return ( + typeof b.x === "number" && + typeof b.y === "number" && + typeof b.width === "number" && + typeof b.height === "number" + ); +} + +export async function collectReveals( + client: CDPClient, + handle: RevealObserverHandle, +): Promise { + try { + const { result } = await client.Runtime.evaluate({ + expression: `(${COLLECT_SCRIPT})(${handle.id})`, + returnByValue: true, + }); + if (Array.isArray(result.value)) return result.value.filter(isBoundingBox); + return []; + } catch { + return []; + } +} + +export const __TESTING__ = { INSTALL_SCRIPT, COLLECT_SCRIPT }; diff --git a/packages/@webreel/core/src/timeline.ts b/packages/@webreel/core/src/timeline.ts index 05c392e..af59b21 100644 --- a/packages/@webreel/core/src/timeline.ts +++ b/packages/@webreel/core/src/timeline.ts @@ -1,5 +1,6 @@ import { writeFileSync } from "node:fs"; import type { Point, SoundEvent } from "./types.js"; +import type { ZoomEvent } from "./autozoom.js"; import { TARGET_FPS, DEFAULT_CURSOR_SVG, @@ -44,6 +45,7 @@ export interface TimelineData { }; frames: FrameData[]; events: SoundEvent[]; + zoomEvents?: ZoomEvent[]; } export class InteractionTimeline { @@ -59,6 +61,7 @@ export class InteractionTimeline { private events: SoundEvent[] = []; private frameCount = 0; private tickResolvers: Array<() => void> = []; + private pathCompleteResolvers: Array<() => void> = []; private width: number; private height: number; @@ -143,6 +146,24 @@ export class InteractionTimeline { }); } + // Resolves when the current cursorPath is fully consumed. If no path is + // active, resolves immediately. Used by animateMoveTo to fire the next + // action (e.g. a click) exactly when the cursor arrives at its target — + // regardless of capture rate, tickDuplicate cadence, or hardware speed. + waitForPathComplete(): Promise { + if (this.cursorPath === null) return Promise.resolve(); + return new Promise((resolve) => { + this.pathCompleteResolvers.push(resolve); + }); + } + + private maybeResolvePathComplete(): void { + if (this.cursorPath !== null || this.pathCompleteResolvers.length === 0) return; + const resolvers = this.pathCompleteResolvers; + this.pathCompleteResolvers = []; + for (const resolve of resolvers) resolve(); + } + tick(): void { if (this.cursorPath && this.pathIndex < this.cursorPath.length) { const p = this.cursorPath[this.pathIndex++]; @@ -158,10 +179,32 @@ export class InteractionTimeline { const resolvers = this.tickResolvers; this.tickResolvers = []; for (const resolve of resolvers) resolve(); + + this.maybeResolvePathComplete(); } tickDuplicate(): void { + // Advance the cursor along the precomputed path on duplicate slots, so + // every 60fps output frame shows a unique cursor position even when the + // underlying screenshot is repeated. Without this, cursor motion runs at + // the capture rate (~28fps) in a 60fps container, producing visible stutter. + // + // Critical: do NOT fire tickResolvers here — those gate action timing + // (e.g. typeText inter-keystroke cadence via waitForNextTick) on REAL + // captured frames. Only the real `tick()` resolves them. + // `pathCompleteResolvers` DO fire here though — a path that exhausts on + // a duplicate slot is just as arrived as one that exhausts on a real tick. + if (this.cursorPath && this.pathIndex < this.cursorPath.length) { + const p = this.cursorPath[this.pathIndex++]; + this.currentCursor.x = p.x; + this.currentCursor.y = p.y; + if (this.pathIndex >= this.cursorPath.length) { + this.cursorPath = null; + } + } this.pushCurrentState(); + + this.maybeResolvePathComplete(); } private pushCurrentState(): void { diff --git a/packages/webreel/LICENSE b/packages/webreel/LICENSE new file mode 100644 index 0000000..8226d36 --- /dev/null +++ b/packages/webreel/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + +Copyright 2025 Vercel Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/packages/webreel/README.md b/packages/webreel/README.md index 68e5df0..b5fb0ae 100644 --- a/packages/webreel/README.md +++ b/packages/webreel/README.md @@ -1,13 +1,15 @@ -# webreel +# @lgariv/webreel + +> **Fork notice.** This is a fork of [vercel-labs/webreel](https://github.com/vercel-labs/webreel) with an added cinematic **autozoom** feature (session-based pan grouping, spatial sub-grouping, mutation-observer-based reveal detection). Original work © Vercel, licensed under Apache 2.0. All modifications are in [`lgariv-dn/webreel`](https://github.com/lgariv-dn/webreel). CLI that records scripted browser videos as MP4, GIF, or WebM files from JSON configs. -Define steps (clicks, key presses, drags, pauses) and webreel drives a headless Chrome instance, captures screenshots at ~60fps, adds cursor animation, keystroke overlays, and sound effects, and encodes the result with ffmpeg. +Define steps (clicks, key presses, drags, pauses) and webreel drives a headless Chrome instance, captures screenshots at ~60fps, adds cursor animation, keystroke overlays, sound effects, and an optional cinematic autozoom, then encodes the result with ffmpeg. ## Installation ```bash -npm install webreel +npm install @lgariv/webreel ``` ## Quick Start diff --git a/packages/webreel/package.json b/packages/webreel/package.json index c4af5c8..fdb2239 100644 --- a/packages/webreel/package.json +++ b/packages/webreel/package.json @@ -1,14 +1,14 @@ { - "name": "webreel", - "version": "0.1.4", - "description": "Record scripted browser demos as MP4, GIF, or WebM with cursor animation, keystroke overlays, and sound effects.", - "homepage": "https://webreel.dev", + "name": "@lgariv/webreel", + "version": "0.1.4-beta-20260418T165223Z", + "description": "Record scripted browser demos as MP4, GIF, or WebM with cursor animation, keystroke overlays, sound effects, and cinematic autozoom. Fork of vercel-labs/webreel.", + "homepage": "https://github.com/lgariv-dn/webreel", "repository": { "type": "git", - "url": "https://github.com/vercel-labs/webreel.git" + "url": "https://github.com/lgariv-dn/webreel.git" }, "bugs": { - "url": "https://github.com/vercel-labs/webreel/issues" + "url": "https://github.com/lgariv-dn/webreel/issues" }, "keywords": [ "video", @@ -41,7 +41,9 @@ "node": ">=18" }, "files": [ - "dist" + "dist", + "LICENSE", + "README.md" ], "scripts": { "clean": "rm -rf dist", @@ -51,7 +53,7 @@ "test": "vitest run" }, "dependencies": { - "@webreel/core": "workspace:*", + "@lgariv/webreel-core": "workspace:*", "commander": "^14.0.3", "jiti": "^2.6.1", "jsonc-parser": "^3.3.1" diff --git a/packages/webreel/src/commands/composite.ts b/packages/webreel/src/commands/composite.ts index d1c1363..269dc38 100644 --- a/packages/webreel/src/commands/composite.ts +++ b/packages/webreel/src/commands/composite.ts @@ -1,14 +1,14 @@ import { Command } from "commander"; import { readFileSync, existsSync, mkdirSync } from "node:fs"; import { resolve, dirname } from "node:path"; -import { compose, type TimelineData } from "@webreel/core"; +import { buildAutoZoomFilter, compose, type TimelineData } from "@lgariv/webreel-core"; import { loadWebreelConfig, resolveConfigPath, getConfigDir, filterVideosByName, } from "../lib/config.js"; -import { extractThumbnailIfConfigured } from "../lib/runner.js"; +import { extractThumbnailIfConfigured, normalizeAutoZoom } from "../lib/runner.js"; export const compositeCommand = new Command("composite") .description("Re-composite videos from stored raw recordings and timelines") @@ -49,7 +49,30 @@ export const compositeCommand = new Command("composite") mkdirSync(dirname(outputPath), { recursive: true }); console.log(`Compositing: ${video.name}`); - await compose(rawPath, timelineData, outputPath, { sfx: video.sfx }); + + const autoZoomCfg = normalizeAutoZoom(video.autoZoom); + const persistedZoomEvents = timelineData.zoomEvents ?? []; + const zoomFilter = + autoZoomCfg.enabled && persistedZoomEvents.length > 0 + ? buildAutoZoomFilter( + persistedZoomEvents, + { width: timelineData.width, height: timelineData.height }, + timelineData.zoom ?? 1, + timelineData.frames.length / timelineData.fps, + timelineData.fps, + autoZoomCfg, + ) + : null; + if (zoomFilter) { + console.log( + `Applying autozoom (${persistedZoomEvents.length} event${persistedZoomEvents.length === 1 ? "" : "s"})`, + ); + } + + await compose(rawPath, timelineData, outputPath, { + sfx: video.sfx, + zoomFilter: zoomFilter ?? undefined, + }); await extractThumbnailIfConfigured(video, outputPath); diff --git a/packages/webreel/src/commands/install.ts b/packages/webreel/src/commands/install.ts index 5318d58..dd52616 100644 --- a/packages/webreel/src/commands/install.ts +++ b/packages/webreel/src/commands/install.ts @@ -7,7 +7,7 @@ import { CHROME_CACHE_DIR, HEADLESS_SHELL_CACHE_DIR, FFMPEG_CACHE_DIR, -} from "@webreel/core"; +} from "@lgariv/webreel-core"; interface InstallStep { label: string; diff --git a/packages/webreel/src/lib/config.ts b/packages/webreel/src/lib/config.ts index c9fef3c..92c86ab 100644 --- a/packages/webreel/src/lib/config.ts +++ b/packages/webreel/src/lib/config.ts @@ -320,6 +320,7 @@ const KNOWN_VIDEO_KEYS = new Set([ "sfx", "defaultDelay", "clickDwell", + "autoZoom", "steps", ]); diff --git a/packages/webreel/src/lib/runner.ts b/packages/webreel/src/lib/runner.ts index e97ee17..b652d3e 100644 --- a/packages/webreel/src/lib/runner.ts +++ b/packages/webreel/src/lib/runner.ts @@ -5,6 +5,8 @@ import { type CDPClient, type BoundingBox, type OverlayTheme, + type AutoZoomConfig, + type ZoomEvent, RecordingContext, connectCDP, launchChrome, @@ -27,8 +29,13 @@ import { ensureFfmpeg, extractThumbnail, moveFileSync, + buildAutoZoomFilter, + unionBboxes, + installRevealObserver, + collectReveals, + type RevealObserverHandle, DEFAULT_VIEWPORT_SIZE, -} from "@webreel/core"; +} from "@lgariv/webreel-core"; import type { VideoConfig, Step, ElementTarget } from "./types.js"; export function formatStep(i: number, step: Step): string { @@ -109,6 +116,118 @@ export function randomPointInBox( }; } +export function normalizeAutoZoom(value: VideoConfig["autoZoom"]): AutoZoomConfig { + if (value === true) return { enabled: true }; + if (value === false || value === undefined) return { enabled: false }; + return { ...value, enabled: value.enabled ?? true }; +} + +async function captureZoomEvent( + client: CDPClient, + step: Step, + timeline: InteractionTimeline, + preEventCount: number, + fps: number, + previousBox: BoundingBox | null, + reveals: BoundingBox[] = [], +): Promise { + let box: BoundingBox | null = null; + + switch (step.action) { + case "navigate": + case "pause": + case "wait": + case "screenshot": + case "scroll": + return null; + + case "click": + case "moveTo": + case "hover": + case "select": { + if (step.selector) { + box = await findElementBySelector(client, step.selector, step.within); + } else if (step.text) { + box = await findElementByText(client, step.text, step.within); + } + break; + } + + case "type": { + if (step.selector) { + box = await findElementBySelector(client, step.selector, step.within); + } else if (previousBox) { + box = previousBox; + } + break; + } + + case "key": { + if (step.target) { + const sel = typeof step.target === "string" ? step.target : step.target.selector; + const within = typeof step.target === "string" ? undefined : step.target.within; + if (sel) box = await findElementBySelector(client, sel, within); + } else if (previousBox) { + box = previousBox; + } + break; + } + + case "drag": { + if (step.from.selector) { + box = await findElementBySelector(client, step.from.selector, step.from.within); + } else if (step.from.text) { + box = await findElementByText(client, step.from.text, step.from.within); + } + break; + } + } + + if (!box) return null; + + // For click/drag, union the click target's bbox with any elements that + // newly appeared or grew in size during the step (dropdowns, modals, + // tooltips). This makes the zoom frame the whole widget rather than just + // the trigger — matching Cursor's behavior for button→modal interactions. + if ((step.action === "click" || step.action === "drag") && reveals.length > 0) { + const unioned = unionBboxes([box, ...reveals]); + if (unioned) box = unioned; + } + + // Pick the timestamp from SoundEvents that fired DURING this step so the + // zoom anticipation is anchored on the actual interaction moment (not the + // step's end-of-postDelay time, which lags the click by 1–2 seconds for + // steps with cursor animation). + const newEvents = timeline.getEvents().slice(preEventCount); + const fallbackMs = (timeline.getFrameCount() / fps) * 1000; + let timeMs: number; + let holdUntilMs: number | undefined; + if (step.action === "click" || step.action === "drag") { + const firstClick = newEvents.find((e) => e.type === "click"); + timeMs = firstClick ? firstClick.timeMs : fallbackMs; + } else if (step.action === "type" || step.action === "key") { + // Anchor the approach on the FIRST event (click that lands on the input) + // so the camera is already zoomed in when typing begins. `holdUntilMs` + // extends the hold through the LAST keystroke, so the camera stays on + // the field for the entire typing span. + const firstEvent = newEvents[0]; + const lastEvent = newEvents[newEvents.length - 1]; + timeMs = firstEvent ? firstEvent.timeMs : fallbackMs; + if (lastEvent && lastEvent !== firstEvent) { + holdUntilMs = lastEvent.timeMs; + } + } else { + timeMs = fallbackMs; + } + + const { result } = await client.Runtime.evaluate({ + expression: "location.href", + returnByValue: true, + }); + const url = typeof result.value === "string" ? result.value : undefined; + return { timeMs, box, url, holdUntilMs }; +} + export async function extractThumbnailIfConfigured( config: Pick, outputPath: string, @@ -205,6 +324,8 @@ export async function runVideo( } let timeline: InteractionTimeline | null = null; + const autoZoomCfg = normalizeAutoZoom(config.autoZoom); + const zoomEvents: ZoomEvent[] = []; const outputPath = config.output ?? resolve(configDir, "videos", `${config.name}.mp4`); @@ -248,6 +369,18 @@ export async function runVideo( const step = config.steps[i]; if (verbose) console.log(formatStep(i, step)); + let preEventCount = 0; + let revealHandle: RevealObserverHandle | null = null; + if (shouldRecord && autoZoomCfg.enabled && timeline) { + preEventCount = timeline.getEvents().length; + // Observer installed only for interactions that can reveal UI. + // Collected after postDelay so any animations have had time to + // settle. + if (step.action === "click" || step.action === "drag") { + revealHandle = await installRevealObserver(client); + } + } + try { switch (step.action) { case "pause": @@ -284,7 +417,13 @@ export async function runVideo( case "type": { if (step.selector) { - const box = await resolveTarget(client, step); + // For type steps, `step.text` is the text to type — NOT a DOM + // matcher. Pass only the selector to resolveTarget so it doesn't + // try to find an element containing the typed text. + const box = await resolveTarget(client, { + selector: step.selector, + within: step.within, + }); const { x: tx, y: ty } = randomPointInBox(box); await clickAt(ctx, client, tx, ty); await client.Runtime.evaluate({ @@ -406,6 +545,35 @@ export async function runVideo( if (postDelay !== undefined && postDelay > 0) { await pause(postDelay); } + if (shouldRecord && autoZoomCfg.enabled && timeline) { + const fps = config.fps ?? timeline.toJSON().fps; + const previousBox = + zoomEvents.length > 0 ? zoomEvents[zoomEvents.length - 1].box : null; + const reveals: BoundingBox[] = revealHandle + ? await collectReveals(client, revealHandle) + : []; + if (process.env.WEBREEL_DEBUG_ZOOM && revealHandle) { + const summary = reveals + .map( + (r) => + `${r.x.toFixed(0)},${r.y.toFixed(0)} ${r.width.toFixed(0)}×${r.height.toFixed(0)}`, + ) + .join(" | "); + console.error( + `reveals (${step.action}): ${reveals.length}${summary ? " " + summary : ""}`, + ); + } + const ze = await captureZoomEvent( + client, + step, + timeline, + preEventCount, + fps, + previousBox, + reveals, + ); + if (ze) zoomEvents.push(ze); + } } catch (err) { throw new Error( `Step ${i} (${step.action}) failed at ${url}: ${err instanceof Error ? err.message : String(err)}`, @@ -421,6 +589,9 @@ export async function runVideo( if (timeline) { const timelineData = timeline.toJSON(); + if (zoomEvents.length > 0) { + timelineData.zoomEvents = zoomEvents; + } const metadataDir = resolve(configDir, ".webreel", "timelines"); mkdirSync(metadataDir, { recursive: true }); writeFileSync( @@ -437,7 +608,33 @@ export async function runVideo( ctx.setTimeline(null); mkdirSync(dirname(outputPath), { recursive: true }); console.log(`Compositing overlays...`); - await compose(rawVideoPath, timelineData, outputPath, { sfx: config.sfx }); + const zoomFilter = autoZoomCfg.enabled + ? buildAutoZoomFilter( + zoomEvents, + { width: timelineData.width, height: timelineData.height }, + timelineData.zoom ?? 1, + timelineData.frames.length / timelineData.fps, + timelineData.fps, + autoZoomCfg, + ) + : null; + if (zoomFilter) { + console.log( + `Applying autozoom (${zoomEvents.length} event${zoomEvents.length === 1 ? "" : "s"})`, + ); + if (verbose) { + for (const e of zoomEvents) { + const box = e.box; + console.log( + ` t=${(e.timeMs / 1000).toFixed(2)}s box=${box.x.toFixed(0)},${box.y.toFixed(0)} ${box.width.toFixed(0)}×${box.height.toFixed(0)}`, + ); + } + } + } + await compose(rawVideoPath, timelineData, outputPath, { + sfx: config.sfx, + zoomFilter: zoomFilter ?? undefined, + }); } await extractThumbnailIfConfigured(config, outputPath); diff --git a/packages/webreel/src/lib/types.ts b/packages/webreel/src/lib/types.ts index 2aed93c..5e788e6 100644 --- a/packages/webreel/src/lib/types.ts +++ b/packages/webreel/src/lib/types.ts @@ -169,8 +169,8 @@ export const VIEWPORT_PRESETS: Record "galaxy-s24": { width: 360, height: 780 }, }; -export type { SfxConfig } from "@webreel/core"; -import type { SfxConfig } from "@webreel/core"; +export type { SfxConfig, AutoZoomConfig } from "@lgariv/webreel-core"; +import type { SfxConfig, AutoZoomConfig } from "@lgariv/webreel-core"; export interface VideoConfig { name: string; @@ -188,6 +188,7 @@ export interface VideoConfig { sfx?: SfxConfig; defaultDelay?: number; clickDwell?: number; + autoZoom?: AutoZoomConfig | boolean; steps: Step[]; } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a024cef..e1c68e2 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -133,7 +133,7 @@ importers: packages/webreel: dependencies: - '@webreel/core': + '@lgariv/webreel-core': specifier: workspace:* version: link:../@webreel/core commander: diff --git a/skills/webreel/SKILL.md b/skills/webreel/SKILL.md index e7d9006..b0e08ed 100644 --- a/skills/webreel/SKILL.md +++ b/skills/webreel/SKILL.md @@ -162,6 +162,7 @@ Each entry in the `videos` map supports: | `clickDwell` | inherited | Override click dwell | | `fps` | `60` | Frame rate | | `quality` | `80` | Encoding quality (1-100) | +| `autoZoom` | `false` | Cinematic zoom into each action (bool or object) | | `steps` | required | Array of step objects | ### Videos map @@ -251,6 +252,63 @@ Customize cursor appearance and keystroke HUD: - `cursor.hotspot` - `"top-left"` (default) or `"center"` - `hud.position` - `"top"` or `"bottom"` +## Autozoom + +Autozoom cinematically zooms into each user action so small UI elements are readable on mobile-sized viewports. It runs as part of the existing compositing pass — no extra encoding — and uses the bounding box of each `click`, `type`, `hover`, `moveTo`, `drag`, or `select` target to decide where and when to zoom. + +Enable with defaults: + +```json +{ "autoZoom": true } +``` + +Or tune individual parameters: + +```json +{ + "autoZoom": { + "enabled": true, + "approachS": 0.5, + "settleBeforeS": 0.15, + "holdAfterS": 0.3, + "releaseS": 0.5, + "paddingRatio": 0.3, + "minZoomRatio": 0.6, + "skipZoomRatio": 0.75, + "sessionGapS": 4.0 + } +} +``` + +| Field | Default | Description | +| --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | `true` | Turn autozoom on/off without removing config | +| `approachS` | `0.5` | Seconds spent zooming IN (full → target) | +| `settleBeforeS` | `0.15` | Seconds the camera sits on the target BEFORE the action fires | +| `holdAfterS` | `0.3` | Seconds the camera holds AFTER the last action in a session | +| `releaseS` | `0.5` | Seconds spent zooming OUT (target → full) | +| `paddingRatio` | `0.3` | Fraction of bbox size added as padding around the target | +| `minZoomRatio` | `0.6` | Crop is never smaller than this fraction of the viewport (max zoom ~1.67x) | +| `skipZoomRatio` | `0.75` | Skip zoom when the computed crop would be this fraction or larger | +| `sessionGapS` | `4.0` | Events within this many seconds share one pan session (no zoom-out) | +| `minPanS` | `0.8` | Skip panning to an intermediate target if the pan would be shorter than this — prevents blink-and-miss transitions when actions fire in quick succession (e.g. a `select` right after typing) | + +**How it behaves:** + +- The camera starts zooming `approachS + settleBeforeS` seconds BEFORE each action, so it's already settled on the target when the click or keystroke fires. +- Multiple actions within `sessionGapS` are treated as one session — the camera pans between them instead of zooming all the way out and back in. +- Click/drag zoom targets automatically include any UI that appears as a result of the click (dropdown menus, modals, tooltips). A `MutationObserver` watches the DOM during each click step and unions newly-visible elements into the zoom bbox, so the camera frames the whole widget rather than just the trigger. +- Clicks that trigger a page navigation (URL change) are skipped; the page content changes unpredictably, and the zoom timing would be off. +- Full-width targets (crop ≥ `skipZoomRatio`) are skipped to avoid imperceptible zooms. + +**When NOT to use autozoom:** + +- Page-scroll-only demos (no discrete click targets). +- Videos whose interactions are all `key` presses without a `target` (no bbox to zoom to). +- Videos recorded at already-small viewports where the content is already readable. + +**Pair with `zoom` for best results on small viewports:** the static `zoom` field upscales the captured viewport for readability; `autoZoom` then cinematically zooms FURTHER into each action. + ## Common patterns ### Shared steps via include diff --git a/skills/webreel/examples.md b/skills/webreel/examples.md index cfdd1f1..0db304a 100644 --- a/skills/webreel/examples.md +++ b/skills/webreel/examples.md @@ -321,3 +321,47 @@ Set the `output` field to a `.webm` extension. } } ``` + +## Autozoom + +Cinematically zoom into each click, type, or hover. The camera eases in before the action, holds through it, then releases. Consecutive actions within `sessionGapS` share a single zoom session (camera pans between them instead of zooming out). + +Enable with defaults: + +```json +{ + "$schema": "https://webreel.dev/schema/v1.json", + "videos": { + "autozoom": { + "url": "./web/index.html", + "viewport": { "width": 1920, "height": 1080 }, + "zoom": 2, + "waitFor": ".card", + "autoZoom": true, + "defaultDelay": 500, + "steps": [ + { "action": "click", "selector": "#name" }, + { "action": "type", "text": "Jane Doe", "selector": "#name" }, + { "action": "click", "selector": "#email" }, + { "action": "type", "text": "jane@acme.com", "selector": "#email" }, + { "action": "select", "selector": "#role", "value": "engineer" }, + { "action": "click", "selector": "#save", "delay": 1200 } + ] + } + } +} +``` + +Override specific timing or thresholds with an object: + +```json +{ + "autoZoom": { + "enabled": true, + "approachS": 1.2, + "holdAfterS": 0.8, + "paddingRatio": 0.4, + "sessionGapS": 3.5 + } +} +```