view() tool blocked on upstream: OWUI Python toolkit tools cannot return model-visible images

[rndmcnlly]

Context

Lathe would benefit from a view() tool that lets the agent visually perceive an image on the sandbox filesystem — useful when driving a headless browser, inspecting generated charts, iterating on visual output, etc. The sandbox can produce the image; the problem is getting it into the model's context window.

The upstream gap

OWUI has a clean convention for human-facing rich output from tools: HTMLResponse with Content-Disposition: inline renders an iframe embed, and the model gets a status string. There is no symmetric convention for model-facing visual output — a tool producing an image that the model needs to see on its next turn.

This is tracked upstream as:

• open-webui/open-webui#22591 — feat: Tool return convention for model-visible image content

What partially works today

The OWUI codebase has two tool execution paths with different image handling:

1. Legacy function-calling path (chat_completion_tools_handler): tool_result_files with type: "image" are emitted as socket events to the frontend for display, but are not injected into the model's next context. The model receives only a text summary like "Image file read successfully.".

2. Native tool-calling path (Responses API / OR-aligned, around middleware.py:4248): For MCP tools returning {"type": "image", ...} content items, image data URIs are added as input_image parts in function_call_output items — meaning the model can see them. However, this only applies to MCP tools on the native path.

Python toolkit tools (which is what Lathe is) return plain strings. There is no return type or convention that causes OWUI to inject an image as an image_url content part in the model's next turn.

Related upstream discussions

• open-webui/open-webui#22467 — MCP annotations for user-only output (the inverse routing problem)
• open-webui/open-webui#22590 — Image transcription for non-vision models (same gap, different workaround angle)

Together these describe the full routing matrix {human, model} × {text, rich} — three of four cells are served; the model-facing visual cell is empty.

What this means for Lathe

A view(path) tool is blocked until OWUI provides a return convention that Python toolkit tools can use to deliver image data to the model. No workaround within Lathe's architecture (single-file toolkit, no OWUI storage dependency) can cleanly bridge this gap.

When the upstream feature lands, implementing view() in Lathe should be straightforward: download the image bytes from the sandbox via the toolbox API, base64-encode them, and return them using whatever convention OWUI establishes.

Action

Watch open-webui/open-webui#22591. No Lathe changes needed until the upstream convention is defined.

[rndmcnlly]

Experimental findings (v0.11.0-dev, not committed)

Tested a prototype where read() detects raster image extensions and returns data:image/<mime>;base64,... instead of text lines. The OWUI process_tool_result chain works on the current turn: data:image/ prefix is detected → moved to tool_result_files → injected as input_image in function_call_output → extracted into a synthetic user message with image_url parts → model sees the image and describes it accurately.

However, it poisons the conversation. The input_image parts are persisted in the DB inside function_call_output.output. On subsequent turns, OWUI reloads the chat history and re-serializes these into the message chain. Anthropic's API rejects the resulting payload with 400 Invalid Anthropic Messages API request (expected string, received array). Every turn after the image tool call fails.

The stripping logic (middleware.py:4329-4337) removes input_image from frontend_output sent to the socket, but the DB save path receives the unstripped output. This means the one-shot perception works but the conversation is unrecoverable without manually editing the chat history.

This confirms the upstream dependency: the ad-hoc data:image/ return path was designed for MCP tools in the native function-calling flow and doesn't handle persistence/replay correctly. A proper upstream convention (open-webui/open-webui#22591) would need to address both the injection and the persistence lifecycle.

[rndmcnlly]

Reproduced in shipped OWUI 0.9.6 (live prod), with the precise root cause

Following up on the dev-prototype finding above: this now reproduces against a shipped OWUI 0.9.6 instance (chat.adamsmith.as) using a minimal throwaway toolkit (a single see_canary_image() returning a synthetic data:image/png;base64,... swatch), not the read() prototype. So the poisoning is a property of the shipped image-return pipeline, not of any lathe-specific code.

The exact failure

Turn 1: tool returns the image, model sees it, replies correctly ("red"). Turn 2: a new user message triggers a request rebuild from stored history, and the provider returns:

400 Invalid Anthropic Messages API request
"Invalid input: expected string, received array"
path: messages[2].content[0].content

Where the base64 actually persists

It is not in the DB content string. serialize_output renders only text parts, so content is the clean summary ("see_canary_image: Image file read successfully."). The base64 survives in the assistant message's structured output array:

{
  "type": "function_call_output",
  "output": [
    {"type": "input_text",  "text": "see_canary_image: Image file read successfully."},
    {"type": "input_image", "image_url": "data:image/png;base64,iVBOR..."}
  ]
}

This output array is persisted to the chat and is what later turns rebuild from.

Root cause: two conversion paths, only one carries the image fix

OWUI flattens multimodal tool results into a string + hoisted user-message image only in the in-loop continuation path, immediately after generate_chat_completion (v0.9.6 middleware.py:4845-4875):

# Chat Completions providers don't support multimodal tool messages.
# Extract images into a user message.
for message in tool_messages:
    if message.get('role') == 'tool' and isinstance(message.get('content'), list):
        ...
        message['content'] = ''.join(text_parts)   # flatten to string
...
# images re-appended as a synthetic user message with image_url parts

The history-reload path has no such flattening. When a fresh turn rebuilds the request, process_messages_with_output (middleware.py:2192) calls convert_output_to_messages(message['output'], raw=True, ...) (:2207) on the persisted output array, which still contains input_image. The result is a role:"tool" message whose content is the [input_text, input_image] array — sent verbatim to the provider.

So the existing fix is incomplete: the multimodal-tool-message flattening at :4845-4875 needs to also apply on the history-reload path (e.g. inside process_messages_with_output / convert_output_to_messages when raw=True), or input_image parts must be stripped/hoisted there too.

Why it looks intermittent

The OWUI model id routes through OpenRouter, which load-balances the upstream Anthropic provider. The failing turn's tool-call id is prefixed toolu_bdrk_ (AWS Bedrock). Bedrock's Messages schema strictly requires tool-result content to be a string; the direct Anthropic API is lenient and accepts the [text, image] array. So the same conversation can succeed or 400 depending on which upstream OpenRouter picks. A naive single-turn probe that happens to land on the lenient route (or that stays inside the in-loop path) will falsely appear to pass — worth noting for anyone testing a fix.

Bottom line

Confirms the upstream dependency (open-webui/open-webui#22591) and pins a concrete, fixable gap: the image-flattening that already exists for the live generation loop is missing from the stored-history conversion path. Until that's addressed, a view() / image-read() tool poisons every subsequent turn on strict providers.