Send read-tool images to vision models

TODO.md

@@ -0,0 +1,17 @@
+# Multimodal read follow-up
+
+Status: WIP follow-up for PR #479. Not required to complete the current provider-wire implementation.
+
+## Live image validation
+
+Test every catalog model declared with `input = ["text", "image"]` using an actual credential for its provider.
+
+- [ ] Generate a provider/model matrix from the effective catalog.
+- [ ] Use the same small JPEG, PNG, GIF, and WebP fixtures where each provider supports them.
+- [ ] Confirm the `read` result reaches the model as visual content, not only as text metadata.
+- [ ] Cover direct API, subscription/OAuth, gateway, and OpenAI-compatible transports.
+- [ ] Record provider-specific limits, rejected formats, required headers, and payload differences.
+- [ ] Add regression tests and compatibility metadata for every discovered exception.
+- [ ] Verify text-only models still receive the explicit omission marker.
+
+Run credentialed validation manually; do not store API keys, OAuth tokens, image payloads, or sensitive provider responses in the repository or CI artifacts.

dev-notes/architecture/index.md

@@ -29,6 +29,7 @@ For the practical frontend contract, see [Building a Custom TUI](../custom-tui.m
 - [Phase 3: Pure Agent Loop](./phase-3-agent-loop.md)
 - [Phase 4: AgentHarness](./phase-4-agent-harness.md)
 - [Phase 5: Built-in Coding Tools](./phase-5-coding-tools.md)
+- [Multimodal read-tool results](./multimodal-read-results.md)
 - [Phase 6: Non-interactive Print-mode CLI](./phase-6-print-mode-cli.md)
 - [Phase 7: Session Tree and JSONL Persistence](./phase-7-session-tree.md)
 - [Phase 8: Coding Session Wrapper](./phase-8-coding-session.md)

dev-notes/architecture/multimodal-read-results.md

@@ -0,0 +1,40 @@
+# Multimodal read-tool results
+
+Tau's `read` tool now detects supported images from file magic and returns them
+as canonical `ImageContent` blocks alongside its short text note. Attachments
+are capped at 5 MB, and animated PNG files are not treated as supported static
+PNG attachments. The agent loop already preserves ordered
+tool-result content, so image data remains provider-neutral until the `tau_ai`
+serialization boundary.
+
+Each provider adapter maps image blocks to its wire format:
+
+- Anthropic nests base64 image blocks in `tool_result.content`.
+- OpenAI Responses and Codex use `input_image` blocks in function-call output.
+- OpenAI Chat Completions and Mistral keep the textual tool result and attach
+  images in a following user message.
+- Gemini 3 uses multimodal `functionResponse.parts`; older Gemini models receive
+  a separate user image message.
+
+Runtime provider configuration derives image support from the selected model's
+catalog `input` metadata. Every provider config, including the distinct
+`OpenAICodexProviderConfig`, preserves this metadata through runtime creation.
+The sparse OpenAI Codex, OpenCode Go, OpenCode Zen, and GitHub Copilot catalog
+entries now declare input modalities for every model, matching Pi's generated
+provider catalog. Text-only models receive an explicit omission marker, which
+avoids invalid provider requests and makes the missing visual context visible to
+the model. GitHub Copilot image requests also include its required
+`Copilot-Vision-Request: true` header.
+
+The image base64 payload moved from tool-result `details` into `content`. This
+prevents duplicate session storage and lets all frontends continue rendering the
+small text note without exposing base64 data.
+
+## Validation
+
+```bash
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run mypy
+```

dev-notes/kimi-k3-model-catalog.md

@@ -7,9 +7,10 @@ Tau's built-in `kimi-code` provider now exposes Kimi K3 through the model ID
 - environment variable: `KIMI_CODE_API_KEY`
 - saved credential name: `kimi-code`
 
-Kimi documents a context window of up to 1,048,576 tokens for eligible plans.
-The catalog records that maximum so Tau's context budgeting can use it; the API
-may reject requests beyond the user's plan entitlement.
+Kimi documents native visual understanding and a context window of up to
+1,048,576 tokens for eligible plans. The catalog therefore marks K3 as accepting
+text and image input and records that maximum so Tau's context budgeting can use
+it; the API may reject requests beyond the user's plan entitlement.
 
 K3 currently accepts only `max` reasoning effort. Tau maps its `xhigh` thinking
 level to the API value `max` and excludes all other thinking levels for this

src/tau_ai/anthropic.py

@@ -11,6 +11,7 @@
 from tau_agent.messages import (
     AgentMessage,
     AssistantMessage,
+    ImageContent,
     TextContent,
     ThinkingContent,
     ToolResultMessage,
@@ -30,6 +31,12 @@
     ProviderThinkingDeltaEvent,
     ProviderToolCallEvent,
 )
+from tau_ai.content import (
+    NON_VISION_TOOL_IMAGE_PLACEHOLDER,
+    NON_VISION_USER_IMAGE_PLACEHOLDER,
+    messages_have_images,
+    text_and_images,
+)
 from tau_ai.env import AnthropicConfig
 from tau_ai.events import AssistantMessageEvent
 from tau_ai.http import create_async_client
@@ -112,13 +119,20 @@ async def iterator() -> AsyncIterator[ProviderEvent]:
                 thinking_budget_tokens=self._config.thinking_budget_tokens,
                 thinking_effort=self._config.thinking_effort,
                 thinking_mode=self._config.thinking_mode,
+                supports_images=self._config.supports_images,
             )
             headers = {
                 "anthropic-version": ANTHROPIC_VERSION,
                 "content-type": "application/json",
                 **(dict(self._config.headers or {})),
                 **auth_headers,
             }
+            if (
+                self._config.provider_name == "github-copilot"
+                and self._config.supports_images
+                and messages_have_images(messages)
+            ):
+                headers["Copilot-Vision-Request"] = "true"
             if self._config.bearer_auth:
                 headers.setdefault("Authorization", f"Bearer {api_key}")
             else:
@@ -344,6 +358,7 @@ def _build_messages_payload(
     thinking_budget_tokens: int | None = None,
     thinking_effort: str | None = None,
     thinking_mode: str = "budget",
+    supports_images: bool = False,
 ) -> dict[str, JSONValue]:
     resolved_max_tokens = max_tokens or DEFAULT_MAX_TOKENS
     if thinking_budget_tokens is not None:
@@ -360,7 +375,9 @@ def _build_messages_payload(
             if oauth_system_prompt
             else system
         ),
-        "messages": [_anthropic_message(message) for message in messages],
+        "messages": [
+            _anthropic_message(message, supports_images=supports_images) for message in messages
+        ],
     }
     if thinking_mode == "disabled":
         payload["thinking"] = {"type": "disabled"}
@@ -377,9 +394,20 @@ def _build_messages_payload(
     return payload
 
 
-def _anthropic_message(message: AgentMessage) -> dict[str, JSONValue]:
+def _anthropic_message(message: AgentMessage, *, supports_images: bool) -> dict[str, JSONValue]:
     if isinstance(message, UserMessage):
-        return {"role": "user", "content": message.text}
+        text, images = text_and_images(
+            message.content,
+            supports_images=supports_images,
+            image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER,
+        )
+        if not images:
+            return {"role": "user", "content": text}
+        user_content: list[JSONValue] = []
+        if text:
+            user_content.append({"type": "text", "text": text})
+        user_content.extend(_anthropic_image(image) for image in images)
+        return {"role": "user", "content": user_content}
     if isinstance(message, AssistantMessage):
         content: list[JSONValue] = []
         for block in message.content:
@@ -404,18 +432,38 @@ def _anthropic_message(message: AgentMessage) -> dict[str, JSONValue]:
                 )
         return {"role": "assistant", "content": content}
     if isinstance(message, ToolResultMessage):
+        text, images = text_and_images(
+            message.content,
+            supports_images=supports_images,
+            image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER,
+        )
+        result_content: list[JSONValue] = []
+        if text:
+            result_content.append({"type": "text", "text": text})
+        result_content.extend(_anthropic_image(image) for image in images)
         return {
             "role": "user",
             "content": [
                 {
                     "type": "tool_result",
                     "tool_use_id": message.tool_call_id,
-                    "content": message.text,
+                    "content": result_content,
                     "is_error": bool(message.is_error),
                 }
             ],
         }
-    return _anthropic_message(message_to_user(message))
+    return _anthropic_message(message_to_user(message), supports_images=supports_images)
+
+
+def _anthropic_image(image: ImageContent) -> dict[str, JSONValue]:
+    return {
+        "type": "image",
+        "source": {
+            "type": "base64",
+            "media_type": image.mime_type,
+            "data": image.data,
+        },
+    }
 
 
 def _anthropic_tool(tool: AgentTool) -> dict[str, JSONValue]:

src/tau_ai/content.py

@@ -0,0 +1,39 @@
+"""Shared helpers for provider serialization of multimodal message content."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from tau_agent.messages import ImageContent, TextContent, ToolResultMessage, UserMessage
+
+NON_VISION_USER_IMAGE_PLACEHOLDER = "(image omitted: model does not support images)"
+NON_VISION_TOOL_IMAGE_PLACEHOLDER = "(tool image omitted: model does not support images)"
+
+
+def messages_have_images(messages: Sequence[object]) -> bool:
+    """Return whether user or tool-result context contains image blocks."""
+    return any(
+        isinstance(message, (UserMessage, ToolResultMessage))
+        and not isinstance(message.content, str)
+        and any(isinstance(block, ImageContent) for block in message.content)
+        for message in messages
+    )
+
+
+def text_and_images(
+    content: str | Sequence[TextContent | ImageContent],
+    *,
+    supports_images: bool,
+    image_placeholder: str,
+) -> tuple[str, list[ImageContent]]:
+    """Return visible text and sendable images, downgrading unsupported images."""
+    if isinstance(content, str):
+        return content, []
+
+    text = "".join(block.text for block in content if isinstance(block, TextContent))
+    images = [block for block in content if isinstance(block, ImageContent)]
+    if supports_images:
+        return text, images
+    if images:
+        text = f"{text}\n{image_placeholder}" if text else image_placeholder
+    return text, []

src/tau_ai/env.py

@@ -39,6 +39,7 @@ class OpenAICompatibleConfig:
     max_retry_delay_seconds: float = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRY_DELAY_SECONDS
     api: str = "openai-completions"
     max_tokens: int | None = None
+    supports_images: bool = False
     reasoning_effort: str | None = None
     reasoning_effort_parameter: str = "reasoning_effort"
     thinking_format: str = "openai"
@@ -61,6 +62,7 @@ class AnthropicConfig:
     max_retries: int = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRIES
     max_retry_delay_seconds: float = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRY_DELAY_SECONDS
     max_tokens: int | None = None
+    supports_images: bool = False
     thinking_budget_tokens: int | None = None
     thinking_effort: str | None = None
     thinking_mode: str = "budget"

src/tau_ai/google.py

@@ -10,6 +10,7 @@
 from tau_agent.messages import (
     AgentMessage,
     AssistantMessage,
+    ImageContent,
     TextContent,
     ThinkingContent,
     ToolResultMessage,
@@ -28,6 +29,11 @@
     ProviderThinkingDeltaEvent,
     ProviderToolCallEvent,
 )
+from tau_ai.content import (
+    NON_VISION_TOOL_IMAGE_PLACEHOLDER,
+    NON_VISION_USER_IMAGE_PLACEHOLDER,
+    text_and_images,
+)
 from tau_ai.env import OpenAICompatibleConfig
 from tau_ai.events import AssistantMessageEvent
 from tau_ai.http import create_async_client
@@ -93,6 +99,7 @@ async def iterator() -> AsyncIterator[ProviderEvent]:
                 tools=tools,
                 reasoning_effort=self._config.reasoning_effort,
                 max_tokens=self._config.max_tokens,
+                supports_images=self._config.supports_images,
             )
             url = (
                 f"{self._config.base_url.rstrip('/')}/models/"
@@ -265,10 +272,15 @@ def _build_google_payload(
     tools: list[AgentTool],
     reasoning_effort: str | None,
     max_tokens: int | None,
+    supports_images: bool = False,
 ) -> dict[str, JSONValue]:
     config: dict[str, JSONValue] = {}
     payload: dict[str, JSONValue] = {
-        "contents": [_message_to_google(message) for message in messages],
+        "contents": [
+            item
+            for message in messages
+            for item in _messages_to_google(message, model=model, supports_images=supports_images)
+        ],
     }
     if system:
         payload["systemInstruction"] = {"parts": [{"text": system}]}
@@ -348,9 +360,20 @@ def _is_gemma4_model(model: str) -> bool:
     return "gemma-4" in model.lower() or "gemma4" in model.lower()
 
 
-def _message_to_google(message: AgentMessage) -> dict[str, JSONValue]:
+def _messages_to_google(
+    message: AgentMessage, *, model: str, supports_images: bool
+) -> list[dict[str, JSONValue]]:
     if isinstance(message, UserMessage):
-        return {"role": "user", "parts": [{"text": message.text}]}
+        text, images = text_and_images(
+            message.content,
+            supports_images=supports_images,
+            image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER,
+        )
+        user_parts: list[JSONValue] = []
+        if text:
+            user_parts.append({"text": text})
+        user_parts.extend(_google_image(image) for image in images)
+        return [{"role": "user", "parts": user_parts}]
     if isinstance(message, AssistantMessage):
         parts: list[JSONValue] = []
         for block in message.content:
@@ -372,16 +395,44 @@ def _message_to_google(message: AgentMessage) -> dict[str, JSONValue]:
                 if block.thought_signature is not None:
                     part["thoughtSignature"] = block.thought_signature
                 parts.append(part)
-        return {"role": "model", "parts": parts or [{"text": ""}]}
+        return [{"role": "model", "parts": parts or [{"text": ""}]}]
     if isinstance(message, ToolResultMessage):
+        text, images = text_and_images(
+            message.content,
+            supports_images=supports_images,
+            image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER,
+        )
         response: dict[str, JSONValue] = {
             "name": message.tool_name,
-            "response": {"output" if not message.is_error else "error": message.text},
+            "response": {"output" if not message.is_error else "error": text},
         }
         if message.tool_call_id:
             response["id"] = message.tool_call_id
-        return {"role": "user", "parts": [{"functionResponse": response}]}
-    return _message_to_google(message_to_user(message))
+        image_parts: list[JSONValue] = [_google_image(image) for image in images]
+        if image_parts and _supports_multimodal_function_response(model):
+            response["parts"] = image_parts
+        converted: list[dict[str, JSONValue]] = [
+            {"role": "user", "parts": [{"functionResponse": response}]}
+        ]
+        if image_parts and not _supports_multimodal_function_response(model):
+            separate_image_parts: list[JSONValue] = [
+                {"text": "Tool result image:"},
+                *image_parts,
+            ]
+            converted.append({"role": "user", "parts": separate_image_parts})
+        return converted
+    return _messages_to_google(
+        message_to_user(message), model=model, supports_images=supports_images
+    )
+
+
+def _google_image(image: ImageContent) -> dict[str, JSONValue]:
+    return {"inlineData": {"mimeType": image.mime_type, "data": image.data}}
+
+
+def _supports_multimodal_function_response(model: str) -> bool:
+    normalized = model.lower()
+    return not normalized.startswith("gemini-") or normalized.startswith("gemini-3")
 
 
 def _tool_to_google(tool: AgentTool) -> dict[str, JSONValue]: