fix: support postMessage options overload and handle transferable clone failures in Firefox

[crisryantan]

Closes #3077

Problem

When testing an application that uses iframe communication with MessagePort transfers, Firefox throws DataCloneError: MessagePort object could not be cloned under Hammerhead's proxy. This breaks any application that relies on MessageChannel-based communication between a parent page and iframes.

The root cause is in MessageSandbox.postMessage: the method wraps the original message in a Hammerhead envelope before calling fastApply. Firefox's structured clone algorithm cannot reconcile the wrapped envelope object with the MessagePort transferables in the transfer list — the ports reference the original message, not the new envelope.

Additionally, the modern postMessage(message, { targetOrigin, transfer }) overload is not supported. When the second argument is an object, the call is silently dropped because typeof targetUrl !== 'string' evaluates to true and the method returns null.

This does not affect Chrome because TestCafe uses native automation (CDP) for Chrome, which bypasses Hammerhead entirely.

Solution

1. Support the postMessage(message, options) overload

The postMessage method now detects when the second argument is an object (rather than a string targetOrigin) and delegates to a new _postMessageWithOptionsOverload handler that extracts targetOrigin and transfer from the options object.

2. Graceful fallback for structured clone failures

Both _postMessageWrapped and _postMessageWithOptionsOverload now wrap the fastApply call in a try/catch. When the wrapped message fails to be structured-cloned (as happens in Firefox with MessagePort transfers), the fallback sends the original unwrapped message with targetOrigin: '*'. This preserves the transfer semantics while still routing through the proxy.

3. Updated receiving side to handle unwrapped messages

• _onWindowMessage: Messages that don't carry the Hammerhead envelope (MessageType.User or MessageType.Service) are now passed through to the original listener. Since these arrive via the browser's native postMessage, the browser has already validated the target origin.
• MessageEvent.data getter: Only unwraps messages that explicitly carry the MessageType.User type. Raw messages from the fallback path are returned as-is.

Tests

• Added test for postMessage(message, { targetOrigin }) options overload
• Added test for postMessage(message, { targetOrigin, transfer }) with MessagePort
• Updated the existing "object as targetOrigin" test to verify that messages are now correctly delivered via the options overload path

Made with Cursor

[github-actions]

Thank you for your contribution to TestCafe. When a member of the TestCafe team becomes available, they will review this PR.