feat(webmcp-polyfill): serialize image tool results - #260
Draft
MiguelsPizza wants to merge 2 commits into
Draft
Conversation
Tools registered on document.modelContext can now return image values:
the serialized MCP-style {type: 'image', data, mimeType} form, or a
browser-source {type: 'image', value, mimeType?} form carrying a Blob,
HTMLCanvasElement, or HTMLImageElement. The source shape mirrors the
Prompt API's LanguageModelToolResultContent {type, value}; sources are
encoded with Web APIs (Blob.arrayBuffer, canvas.toBlob) and serialized
to the base64 data form before executeTool() returns (see
webmachinelearning/webmcp#41 and webmachinelearning/webmcp#86).
Adds @mcp-b/webmcp-types image value types and a Playwright runtime
contract suite that runs against the polyfill, stock Chrome, and native
WebMCP Chromium builds. Browser-source serialization is optional for
native runtimes until spec text exists; those tests skip via runtime
capability detection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cover a canvas-backed image value with a requested JPEG MIME type and an array-wrapped image value (image detection applies to the top-level result only), mirroring the new wpt_internal subtests in the Chromium CL. Verified green against both the polyfill and the native build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the WebMCP image-value feature work out of local
mainonto its own branch. Tools registered ondocument.modelContextcan return direct image values, either as serialized{ type: "image", data, mimeType }payloads or browser-source{ type: "image", value, mimeType? }payloads backed byBlob,HTMLCanvasElement, orHTMLImageElement.This also adds the matching
@mcp-b/webmcp-typesimage value types and a focused Playwright runtime contract for the polyfill/native image-value behavior.Why
This is unrelated feature work and should not be mixed into the release-prep commits currently sitting on local
main. The implementation follows the image/tool-result direction discussed in WebMCP: webmachinelearning/webmcp#41 and webmachinelearning/webmcp#86.Validation
pnpm --filter @mcp-b/webmcp-types typecheckpnpm --filter @mcp-b/webmcp-polyfill typecheckpnpm --filter mcp-e2e-tests exec playwright test tests/runtime-contract-image-values.spec.tsNote: the focused e2e run requires local workspace package builds first in a fresh worktree because the test app imports built package entry points.