diff --git a/TODO.md b/TODO.md new file mode 100644 index 000000000..326f86d07 --- /dev/null +++ b/TODO.md @@ -0,0 +1,17 @@ +# Multimodal read follow-up + +Status: WIP follow-up for PR #479. Not required to complete the current provider-wire implementation. + +## Live image validation + +Test every catalog model declared with `input = ["text", "image"]` using an actual credential for its provider. + +- [ ] Generate a provider/model matrix from the effective catalog. +- [ ] Use the same small JPEG, PNG, GIF, and WebP fixtures where each provider supports them. +- [ ] Confirm the `read` result reaches the model as visual content, not only as text metadata. +- [ ] Cover direct API, subscription/OAuth, gateway, and OpenAI-compatible transports. +- [ ] Record provider-specific limits, rejected formats, required headers, and payload differences. +- [ ] Add regression tests and compatibility metadata for every discovered exception. +- [ ] Verify text-only models still receive the explicit omission marker. + +Run credentialed validation manually; do not store API keys, OAuth tokens, image payloads, or sensitive provider responses in the repository or CI artifacts. diff --git a/dev-notes/architecture/index.md b/dev-notes/architecture/index.md index 5bf373b35..11d710e91 100644 --- a/dev-notes/architecture/index.md +++ b/dev-notes/architecture/index.md @@ -29,6 +29,7 @@ For the practical frontend contract, see [Building a Custom TUI](../custom-tui.m - [Phase 3: Pure Agent Loop](./phase-3-agent-loop.md) - [Phase 4: AgentHarness](./phase-4-agent-harness.md) - [Phase 5: Built-in Coding Tools](./phase-5-coding-tools.md) +- [Multimodal read-tool results](./multimodal-read-results.md) - [Phase 6: Non-interactive Print-mode CLI](./phase-6-print-mode-cli.md) - [Phase 7: Session Tree and JSONL Persistence](./phase-7-session-tree.md) - [Phase 8: Coding Session Wrapper](./phase-8-coding-session.md) diff --git a/dev-notes/architecture/multimodal-read-results.md b/dev-notes/architecture/multimodal-read-results.md new file mode 100644 index 000000000..6b2598995 --- /dev/null +++ b/dev-notes/architecture/multimodal-read-results.md @@ -0,0 +1,40 @@ +# Multimodal read-tool results + +Tau's `read` tool now detects supported images from file magic and returns them +as canonical `ImageContent` blocks alongside its short text note. Attachments +are capped at 5 MB, and animated PNG files are not treated as supported static +PNG attachments. The agent loop already preserves ordered +tool-result content, so image data remains provider-neutral until the `tau_ai` +serialization boundary. + +Each provider adapter maps image blocks to its wire format: + +- Anthropic nests base64 image blocks in `tool_result.content`. +- OpenAI Responses and Codex use `input_image` blocks in function-call output. +- OpenAI Chat Completions and Mistral keep the textual tool result and attach + images in a following user message. +- Gemini 3 uses multimodal `functionResponse.parts`; older Gemini models receive + a separate user image message. + +Runtime provider configuration derives image support from the selected model's +catalog `input` metadata. Every provider config, including the distinct +`OpenAICodexProviderConfig`, preserves this metadata through runtime creation. +The sparse OpenAI Codex, OpenCode Go, OpenCode Zen, and GitHub Copilot catalog +entries now declare input modalities for every model, matching Pi's generated +provider catalog. Text-only models receive an explicit omission marker, which +avoids invalid provider requests and makes the missing visual context visible to +the model. GitHub Copilot image requests also include its required +`Copilot-Vision-Request: true` header. + +The image base64 payload moved from tool-result `details` into `content`. This +prevents duplicate session storage and lets all frontends continue rendering the +small text note without exposing base64 data. + +## Validation + +```bash +uv run pytest +uv run ruff check . +uv run ruff format --check . +uv run mypy +``` diff --git a/dev-notes/kimi-k3-model-catalog.md b/dev-notes/kimi-k3-model-catalog.md index 2120e36ce..8de37c926 100644 --- a/dev-notes/kimi-k3-model-catalog.md +++ b/dev-notes/kimi-k3-model-catalog.md @@ -7,9 +7,10 @@ Tau's built-in `kimi-code` provider now exposes Kimi K3 through the model ID - environment variable: `KIMI_CODE_API_KEY` - saved credential name: `kimi-code` -Kimi documents a context window of up to 1,048,576 tokens for eligible plans. -The catalog records that maximum so Tau's context budgeting can use it; the API -may reject requests beyond the user's plan entitlement. +Kimi documents native visual understanding and a context window of up to +1,048,576 tokens for eligible plans. The catalog therefore marks K3 as accepting +text and image input and records that maximum so Tau's context budgeting can use +it; the API may reject requests beyond the user's plan entitlement. K3 currently accepts only `max` reasoning effort. Tau maps its `xhigh` thinking level to the API value `max` and excludes all other thinking levels for this diff --git a/src/tau_ai/anthropic.py b/src/tau_ai/anthropic.py index 338ab1f70..da969d35b 100644 --- a/src/tau_ai/anthropic.py +++ b/src/tau_ai/anthropic.py @@ -11,6 +11,7 @@ from tau_agent.messages import ( AgentMessage, AssistantMessage, + ImageContent, TextContent, ThinkingContent, ToolResultMessage, @@ -30,6 +31,12 @@ ProviderThinkingDeltaEvent, ProviderToolCallEvent, ) +from tau_ai.content import ( + NON_VISION_TOOL_IMAGE_PLACEHOLDER, + NON_VISION_USER_IMAGE_PLACEHOLDER, + messages_have_images, + text_and_images, +) from tau_ai.env import AnthropicConfig from tau_ai.events import AssistantMessageEvent from tau_ai.http import create_async_client @@ -112,6 +119,7 @@ async def iterator() -> AsyncIterator[ProviderEvent]: thinking_budget_tokens=self._config.thinking_budget_tokens, thinking_effort=self._config.thinking_effort, thinking_mode=self._config.thinking_mode, + supports_images=self._config.supports_images, ) headers = { "anthropic-version": ANTHROPIC_VERSION, @@ -119,6 +127,12 @@ async def iterator() -> AsyncIterator[ProviderEvent]: **(dict(self._config.headers or {})), **auth_headers, } + if ( + self._config.provider_name == "github-copilot" + and self._config.supports_images + and messages_have_images(messages) + ): + headers["Copilot-Vision-Request"] = "true" if self._config.bearer_auth: headers.setdefault("Authorization", f"Bearer {api_key}") else: @@ -344,6 +358,7 @@ def _build_messages_payload( thinking_budget_tokens: int | None = None, thinking_effort: str | None = None, thinking_mode: str = "budget", + supports_images: bool = False, ) -> dict[str, JSONValue]: resolved_max_tokens = max_tokens or DEFAULT_MAX_TOKENS if thinking_budget_tokens is not None: @@ -360,7 +375,9 @@ def _build_messages_payload( if oauth_system_prompt else system ), - "messages": [_anthropic_message(message) for message in messages], + "messages": [ + _anthropic_message(message, supports_images=supports_images) for message in messages + ], } if thinking_mode == "disabled": payload["thinking"] = {"type": "disabled"} @@ -377,9 +394,20 @@ def _build_messages_payload( return payload -def _anthropic_message(message: AgentMessage) -> dict[str, JSONValue]: +def _anthropic_message(message: AgentMessage, *, supports_images: bool) -> dict[str, JSONValue]: if isinstance(message, UserMessage): - return {"role": "user", "content": message.text} + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER, + ) + if not images: + return {"role": "user", "content": text} + user_content: list[JSONValue] = [] + if text: + user_content.append({"type": "text", "text": text}) + user_content.extend(_anthropic_image(image) for image in images) + return {"role": "user", "content": user_content} if isinstance(message, AssistantMessage): content: list[JSONValue] = [] for block in message.content: @@ -404,18 +432,38 @@ def _anthropic_message(message: AgentMessage) -> dict[str, JSONValue]: ) return {"role": "assistant", "content": content} if isinstance(message, ToolResultMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER, + ) + result_content: list[JSONValue] = [] + if text: + result_content.append({"type": "text", "text": text}) + result_content.extend(_anthropic_image(image) for image in images) return { "role": "user", "content": [ { "type": "tool_result", "tool_use_id": message.tool_call_id, - "content": message.text, + "content": result_content, "is_error": bool(message.is_error), } ], } - return _anthropic_message(message_to_user(message)) + return _anthropic_message(message_to_user(message), supports_images=supports_images) + + +def _anthropic_image(image: ImageContent) -> dict[str, JSONValue]: + return { + "type": "image", + "source": { + "type": "base64", + "media_type": image.mime_type, + "data": image.data, + }, + } def _anthropic_tool(tool: AgentTool) -> dict[str, JSONValue]: diff --git a/src/tau_ai/content.py b/src/tau_ai/content.py new file mode 100644 index 000000000..429344eed --- /dev/null +++ b/src/tau_ai/content.py @@ -0,0 +1,39 @@ +"""Shared helpers for provider serialization of multimodal message content.""" + +from __future__ import annotations + +from collections.abc import Sequence + +from tau_agent.messages import ImageContent, TextContent, ToolResultMessage, UserMessage + +NON_VISION_USER_IMAGE_PLACEHOLDER = "(image omitted: model does not support images)" +NON_VISION_TOOL_IMAGE_PLACEHOLDER = "(tool image omitted: model does not support images)" + + +def messages_have_images(messages: Sequence[object]) -> bool: + """Return whether user or tool-result context contains image blocks.""" + return any( + isinstance(message, (UserMessage, ToolResultMessage)) + and not isinstance(message.content, str) + and any(isinstance(block, ImageContent) for block in message.content) + for message in messages + ) + + +def text_and_images( + content: str | Sequence[TextContent | ImageContent], + *, + supports_images: bool, + image_placeholder: str, +) -> tuple[str, list[ImageContent]]: + """Return visible text and sendable images, downgrading unsupported images.""" + if isinstance(content, str): + return content, [] + + text = "".join(block.text for block in content if isinstance(block, TextContent)) + images = [block for block in content if isinstance(block, ImageContent)] + if supports_images: + return text, images + if images: + text = f"{text}\n{image_placeholder}" if text else image_placeholder + return text, [] diff --git a/src/tau_ai/env.py b/src/tau_ai/env.py index 4929a8da5..da623fb46 100644 --- a/src/tau_ai/env.py +++ b/src/tau_ai/env.py @@ -39,6 +39,7 @@ class OpenAICompatibleConfig: max_retry_delay_seconds: float = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRY_DELAY_SECONDS api: str = "openai-completions" max_tokens: int | None = None + supports_images: bool = False reasoning_effort: str | None = None reasoning_effort_parameter: str = "reasoning_effort" thinking_format: str = "openai" @@ -61,6 +62,7 @@ class AnthropicConfig: max_retries: int = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRIES max_retry_delay_seconds: float = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRY_DELAY_SECONDS max_tokens: int | None = None + supports_images: bool = False thinking_budget_tokens: int | None = None thinking_effort: str | None = None thinking_mode: str = "budget" diff --git a/src/tau_ai/google.py b/src/tau_ai/google.py index f978954af..c0b16f06f 100644 --- a/src/tau_ai/google.py +++ b/src/tau_ai/google.py @@ -10,6 +10,7 @@ from tau_agent.messages import ( AgentMessage, AssistantMessage, + ImageContent, TextContent, ThinkingContent, ToolResultMessage, @@ -28,6 +29,11 @@ ProviderThinkingDeltaEvent, ProviderToolCallEvent, ) +from tau_ai.content import ( + NON_VISION_TOOL_IMAGE_PLACEHOLDER, + NON_VISION_USER_IMAGE_PLACEHOLDER, + text_and_images, +) from tau_ai.env import OpenAICompatibleConfig from tau_ai.events import AssistantMessageEvent from tau_ai.http import create_async_client @@ -93,6 +99,7 @@ async def iterator() -> AsyncIterator[ProviderEvent]: tools=tools, reasoning_effort=self._config.reasoning_effort, max_tokens=self._config.max_tokens, + supports_images=self._config.supports_images, ) url = ( f"{self._config.base_url.rstrip('/')}/models/" @@ -265,10 +272,15 @@ def _build_google_payload( tools: list[AgentTool], reasoning_effort: str | None, max_tokens: int | None, + supports_images: bool = False, ) -> dict[str, JSONValue]: config: dict[str, JSONValue] = {} payload: dict[str, JSONValue] = { - "contents": [_message_to_google(message) for message in messages], + "contents": [ + item + for message in messages + for item in _messages_to_google(message, model=model, supports_images=supports_images) + ], } if system: payload["systemInstruction"] = {"parts": [{"text": system}]} @@ -348,9 +360,20 @@ def _is_gemma4_model(model: str) -> bool: return "gemma-4" in model.lower() or "gemma4" in model.lower() -def _message_to_google(message: AgentMessage) -> dict[str, JSONValue]: +def _messages_to_google( + message: AgentMessage, *, model: str, supports_images: bool +) -> list[dict[str, JSONValue]]: if isinstance(message, UserMessage): - return {"role": "user", "parts": [{"text": message.text}]} + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER, + ) + user_parts: list[JSONValue] = [] + if text: + user_parts.append({"text": text}) + user_parts.extend(_google_image(image) for image in images) + return [{"role": "user", "parts": user_parts}] if isinstance(message, AssistantMessage): parts: list[JSONValue] = [] for block in message.content: @@ -372,16 +395,44 @@ def _message_to_google(message: AgentMessage) -> dict[str, JSONValue]: if block.thought_signature is not None: part["thoughtSignature"] = block.thought_signature parts.append(part) - return {"role": "model", "parts": parts or [{"text": ""}]} + return [{"role": "model", "parts": parts or [{"text": ""}]}] if isinstance(message, ToolResultMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER, + ) response: dict[str, JSONValue] = { "name": message.tool_name, - "response": {"output" if not message.is_error else "error": message.text}, + "response": {"output" if not message.is_error else "error": text}, } if message.tool_call_id: response["id"] = message.tool_call_id - return {"role": "user", "parts": [{"functionResponse": response}]} - return _message_to_google(message_to_user(message)) + image_parts: list[JSONValue] = [_google_image(image) for image in images] + if image_parts and _supports_multimodal_function_response(model): + response["parts"] = image_parts + converted: list[dict[str, JSONValue]] = [ + {"role": "user", "parts": [{"functionResponse": response}]} + ] + if image_parts and not _supports_multimodal_function_response(model): + separate_image_parts: list[JSONValue] = [ + {"text": "Tool result image:"}, + *image_parts, + ] + converted.append({"role": "user", "parts": separate_image_parts}) + return converted + return _messages_to_google( + message_to_user(message), model=model, supports_images=supports_images + ) + + +def _google_image(image: ImageContent) -> dict[str, JSONValue]: + return {"inlineData": {"mimeType": image.mime_type, "data": image.data}} + + +def _supports_multimodal_function_response(model: str) -> bool: + normalized = model.lower() + return not normalized.startswith("gemini-") or normalized.startswith("gemini-3") def _tool_to_google(tool: AgentTool) -> dict[str, JSONValue]: diff --git a/src/tau_ai/mistral.py b/src/tau_ai/mistral.py index 71a07c730..2a0444c1d 100644 --- a/src/tau_ai/mistral.py +++ b/src/tau_ai/mistral.py @@ -11,6 +11,7 @@ from tau_agent.messages import ( AgentMessage, AssistantMessage, + ImageContent, ThinkingContent, ToolResultMessage, UserMessage, @@ -28,6 +29,11 @@ ProviderThinkingDeltaEvent, ProviderToolCallEvent, ) +from tau_ai.content import ( + NON_VISION_TOOL_IMAGE_PLACEHOLDER, + NON_VISION_USER_IMAGE_PLACEHOLDER, + text_and_images, +) from tau_ai.env import OpenAICompatibleConfig from tau_ai.events import AssistantMessageEvent from tau_ai.http import create_async_client @@ -90,6 +96,7 @@ def _stream_provider_events( tools=tools, reasoning_effort=self._config.reasoning_effort, max_tokens=self._config.max_tokens, + supports_images=self._config.supports_images, ) return self._stream( model=model, @@ -305,13 +312,14 @@ def _build_mistral_payload( tools: list[AgentTool], reasoning_effort: str | None, max_tokens: int | None, + supports_images: bool = False, ) -> dict[str, JSONValue]: payload: dict[str, JSONValue] = { "model": model, "stream": True, "messages": [ *_system_messages(system), - *[_message_to_mistral(message) for message in messages], + *_messages_to_mistral(messages, supports_images=supports_images), ], } if max_tokens is not None: @@ -330,6 +338,70 @@ def _system_messages(system: str) -> list[dict[str, JSONValue]]: return [{"role": "system", "content": system}] if system else [] +def _messages_to_mistral( + messages: list[AgentMessage], *, supports_images: bool +) -> list[dict[str, JSONValue]]: + converted: list[dict[str, JSONValue]] = [] + pending_tool_images: list[ImageContent] = [] + for message in messages: + if pending_tool_images and not isinstance(message, ToolResultMessage): + converted.append(_mistral_tool_image_message(pending_tool_images)) + pending_tool_images = [] + if isinstance(message, UserMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER, + ) + if images: + content: list[JSONValue] = [] + if text: + content.append({"type": "text", "text": text}) + content.extend( + { + "type": "image_url", + "image_url": f"data:{image.mime_type};base64,{image.data}", + } + for image in images + ) + converted.append({"role": "user", "content": content}) + else: + converted.append({"role": "user", "content": text}) + continue + if isinstance(message, ToolResultMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER, + ) + converted.append( + { + "role": "tool", + "tool_call_id": message.tool_call_id, + "name": message.tool_name, + "content": text or ("(see attached image)" if images else "(no tool output)"), + } + ) + pending_tool_images.extend(images) + continue + converted.append(_message_to_mistral(message)) + if pending_tool_images: + converted.append(_mistral_tool_image_message(pending_tool_images)) + return converted + + +def _mistral_tool_image_message(images: list[ImageContent]) -> dict[str, JSONValue]: + content: list[JSONValue] = [{"type": "text", "text": "Attached image(s) from tool result:"}] + content.extend( + { + "type": "image_url", + "image_url": f"data:{image.mime_type};base64,{image.data}", + } + for image in images + ) + return {"role": "user", "content": content} + + def _message_to_mistral(message: AgentMessage) -> dict[str, JSONValue]: if isinstance(message, UserMessage): return {"role": "user", "content": message.text} diff --git a/src/tau_ai/openai_codex.py b/src/tau_ai/openai_codex.py index aa4bf042b..7cd623a6b 100644 --- a/src/tau_ai/openai_codex.py +++ b/src/tau_ai/openai_codex.py @@ -13,6 +13,7 @@ from tau_agent.messages import ( AgentMessage, AssistantMessage, + ImageContent, TextContent, ThinkingContent, ToolResultMessage, @@ -31,6 +32,11 @@ ProviderThinkingDeltaEvent, ProviderToolCallEvent, ) +from tau_ai.content import ( + NON_VISION_TOOL_IMAGE_PLACEHOLDER, + NON_VISION_USER_IMAGE_PLACEHOLDER, + text_and_images, +) from tau_ai.env import ( DEFAULT_OPENAI_COMPATIBLE_MAX_RETRIES, DEFAULT_OPENAI_COMPATIBLE_MAX_RETRY_DELAY_SECONDS, @@ -71,6 +77,7 @@ class OpenAICodexConfig: originator: str = "tau" reasoning_effort: str | None = None reasoning_summary: str = "auto" + supports_images: bool = False provider_name: str = "OpenAI Codex" # The Codex catalog filters models by the official client's compatibility # version. This is the oldest known version that advertises GPT-5.6. @@ -161,6 +168,7 @@ async def iterator() -> AsyncIterator[ProviderEvent]: tools=tools, reasoning_effort=self._config.reasoning_effort, reasoning_summary=self._config.reasoning_summary, + supports_images=self._config.supports_images, ) url = _resolve_codex_url(self._config.base_url) @@ -358,13 +366,14 @@ def _build_codex_payload( tools: list[AgentTool], reasoning_effort: str | None = None, reasoning_summary: str = "auto", + supports_images: bool = False, ) -> dict[str, JSONValue]: payload: dict[str, JSONValue] = { "model": model, "store": False, "stream": True, "instructions": system or "You are a helpful assistant.", - "input": _messages_to_responses_input(messages), + "input": _messages_to_responses_input(messages, supports_images=supports_images), "text": {"verbosity": "low"}, "include": ["reasoning.encrypted_content"], "tool_choice": "auto", @@ -380,17 +389,23 @@ def _build_codex_payload( return payload -def _messages_to_responses_input(messages: list[AgentMessage]) -> list[JSONValue]: +def _messages_to_responses_input( + messages: list[AgentMessage], *, supports_images: bool = False +) -> list[JSONValue]: items: list[JSONValue] = [] assistant_index = 0 for message in messages: if isinstance(message, UserMessage): - items.append( - { - "role": "user", - "content": [{"type": "input_text", "text": message.text}], - } + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER, ) + content: list[JSONValue] = [] + if text: + content.append({"type": "input_text", "text": text}) + content.extend(_codex_input_image(image) for image in images) + items.append({"role": "user", "content": content}) elif isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, ThinkingContent) and block.thinking_signature: @@ -430,16 +445,38 @@ def _messages_to_responses_input(messages: list[AgentMessage]) -> list[JSONValue items.append(item) elif isinstance(message, ToolResultMessage): call_id, _item_id = _split_tool_call_id(message.tool_call_id) + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER, + ) + output: JSONValue + if images: + output_parts: list[JSONValue] = [] + if text: + output_parts.append({"type": "input_text", "text": text}) + output_parts.extend(_codex_input_image(image) for image in images) + output = output_parts + else: + output = text or "(no tool output)" items.append( { "type": "function_call_output", "call_id": call_id, - "output": message.text, + "output": output, } ) return items +def _codex_input_image(image: ImageContent) -> dict[str, JSONValue]: + return { + "type": "input_image", + "detail": "auto", + "image_url": f"data:{image.mime_type};base64,{image.data}", + } + + def _tool_to_codex(tool: AgentTool) -> dict[str, JSONValue]: return { "type": "function", diff --git a/src/tau_ai/openai_compatible.py b/src/tau_ai/openai_compatible.py index dd1e5d0e8..7e68e1fb6 100644 --- a/src/tau_ai/openai_compatible.py +++ b/src/tau_ai/openai_compatible.py @@ -19,6 +19,7 @@ from tau_agent.messages import ( AgentMessage, AssistantMessage, + ImageContent, ThinkingContent, ToolResultMessage, Usage, @@ -37,6 +38,12 @@ ProviderThinkingDeltaEvent, ProviderToolCallEvent, ) +from tau_ai.content import ( + NON_VISION_TOOL_IMAGE_PLACEHOLDER, + NON_VISION_USER_IMAGE_PLACEHOLDER, + messages_have_images, + text_and_images, +) from tau_ai.env import OpenAICompatibleConfig from tau_ai.events import AssistantMessageEvent from tau_ai.http import create_async_client @@ -147,12 +154,14 @@ def _stream_chat_completions( compat=self._config.compat, max_tokens=self._config.max_tokens, include_reasoning_effort_none=self._config.include_reasoning_effort_none, + supports_images=self._config.supports_images, ) return self._stream( model=model, url=f"{self._config.base_url.rstrip('/')}/chat/completions", payload=payload, parser_factory=_ChatStreamParser, + has_images=(self._config.supports_images and messages_have_images(messages)), signal=signal, ) @@ -173,12 +182,14 @@ def _stream_responses( tools=tools, reasoning_effort=self._config.reasoning_effort, max_tokens=self._config.max_tokens, + supports_images=self._config.supports_images, ) return self._stream( model=model, url=f"{self._config.base_url.rstrip('/')}/responses", payload=payload, parser_factory=_ResponsesStreamParser, + has_images=(self._config.supports_images and messages_have_images(messages)), signal=signal, ) @@ -189,6 +200,7 @@ def _stream( url: str, payload: Mapping[str, JSONValue], parser_factory: Callable[[], _StreamParser], + has_images: bool = False, signal: CancellationToken | None = None, ) -> AsyncIterator[ProviderEvent]: """Run the shared streaming POST + retry envelope for a given endpoint. @@ -204,6 +216,8 @@ async def iterator() -> AsyncIterator[ProviderEvent]: api_key = self._config.api_key request_url = url headers = dict(self._config.headers or {}) + if self._config.provider_name == "github-copilot" and has_images: + headers["Copilot-Vision-Request"] = "true" if self._config.credential_resolver is not None: auth = await self._config.credential_resolver() api_key = auth.api_key @@ -666,6 +680,7 @@ def _build_chat_payload( compat: Mapping[str, JSONValue] | None = None, max_tokens: int | None = None, include_reasoning_effort_none: bool = False, + supports_images: bool = False, ) -> dict[str, JSONValue]: resolved_compat = dict(compat or {}) supports_store = bool(resolved_compat.get("supportsStore", True)) @@ -679,7 +694,7 @@ def _build_chat_payload( "stream": True, "messages": [ _system_message(system), - *[_message_to_openai(message) for message in messages], + *_messages_to_openai_chat(messages, supports_images=supports_images), ], } if supports_usage: @@ -757,6 +772,7 @@ def _build_responses_payload( tools: list[AgentTool], reasoning_effort: str | None = None, max_tokens: int | None = None, + supports_images: bool = False, ) -> dict[str, JSONValue]: payload: dict[str, JSONValue] = { "model": model, @@ -766,7 +782,7 @@ def _build_responses_payload( # path usable for zero-data-retention orgs, which reject ``store: true``. "store": False, "instructions": system, - "input": _messages_to_responses_input(messages), + "input": _messages_to_responses_input(messages, supports_images=supports_images), } if max_tokens is not None: payload["max_output_tokens"] = max_tokens @@ -792,12 +808,24 @@ def _normalize_responses_effort(reasoning_effort: str | None) -> str | None: def _messages_to_responses_input( - messages: list[AgentMessage], + messages: list[AgentMessage], *, supports_images: bool = False ) -> list[JSONValue]: items: list[JSONValue] = [] for message in messages: if isinstance(message, UserMessage): - items.append({"role": "user", "content": message.text}) + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER, + ) + if images: + content: list[JSONValue] = [] + if text: + content.append({"type": "input_text", "text": text}) + content.extend(_openai_input_image(image) for image in images) + items.append({"role": "user", "content": content}) + else: + items.append({"role": "user", "content": text}) elif isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, ThinkingContent) and block.thinking_signature: @@ -819,16 +847,38 @@ def _messages_to_responses_input( } ) elif isinstance(message, ToolResultMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER, + ) + output: JSONValue + if images: + output_parts: list[JSONValue] = [] + if text: + output_parts.append({"type": "input_text", "text": text}) + output_parts.extend(_openai_input_image(image) for image in images) + output = output_parts + else: + output = text or "(no tool output)" items.append( { "type": "function_call_output", "call_id": message.tool_call_id, - "output": message.text, + "output": output, } ) return items +def _openai_input_image(image: ImageContent) -> dict[str, JSONValue]: + return { + "type": "input_image", + "detail": "auto", + "image_url": f"data:{image.mime_type};base64,{image.data}", + } + + def _tool_to_responses(tool: AgentTool) -> dict[str, JSONValue]: return { "type": "function", @@ -948,6 +998,70 @@ def _system_message(system: str) -> dict[str, JSONValue]: return {"role": "system", "content": system} +def _messages_to_openai_chat( + messages: list[AgentMessage], *, supports_images: bool +) -> list[dict[str, JSONValue]]: + converted: list[dict[str, JSONValue]] = [] + pending_tool_images: list[ImageContent] = [] + for message in messages: + if pending_tool_images and not isinstance(message, ToolResultMessage): + converted.append(_openai_tool_image_message(pending_tool_images)) + pending_tool_images = [] + if isinstance(message, UserMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_USER_IMAGE_PLACEHOLDER, + ) + if images: + content: list[JSONValue] = [] + if text: + content.append({"type": "text", "text": text}) + content.extend( + { + "type": "image_url", + "image_url": {"url": f"data:{image.mime_type};base64,{image.data}"}, + } + for image in images + ) + converted.append({"role": "user", "content": content}) + else: + converted.append({"role": "user", "content": text}) + continue + if isinstance(message, ToolResultMessage): + text, images = text_and_images( + message.content, + supports_images=supports_images, + image_placeholder=NON_VISION_TOOL_IMAGE_PLACEHOLDER, + ) + converted.append( + { + "role": "tool", + "tool_call_id": message.tool_call_id, + "name": message.tool_name, + "content": text or ("(see attached image)" if images else "(no tool output)"), + } + ) + pending_tool_images.extend(images) + continue + converted.append(_message_to_openai(message)) + if pending_tool_images: + converted.append(_openai_tool_image_message(pending_tool_images)) + return converted + + +def _openai_tool_image_message(images: list[ImageContent]) -> dict[str, JSONValue]: + content: list[JSONValue] = [{"type": "text", "text": "Attached image(s) from tool result:"}] + content.extend( + { + "type": "image_url", + "image_url": {"url": f"data:{image.mime_type};base64,{image.data}"}, + } + for image in images + ) + return {"role": "user", "content": content} + + def _message_to_openai(message: AgentMessage) -> dict[str, JSONValue]: if isinstance(message, UserMessage): return {"role": "user", "content": message.text} diff --git a/src/tau_coding/data/catalog.toml b/src/tau_coding/data/catalog.toml index 6eeae8fee..0abea1108 100644 --- a/src/tau_coding/data/catalog.toml +++ b/src/tau_coding/data/catalog.toml @@ -510,6 +510,30 @@ cost = { input = 1, output = 6, cacheRead = 0.1, cacheWrite = 1.25 } thinking_level_map = { off = "none", xhigh = "xhigh" } unsupported_thinking_levels = ["minimal"] +[providers.model_metadata."gpt-5.5"] +input = ["text", "image"] +thinking_level_map = { xhigh = "xhigh" } + +[providers.model_metadata."gpt-5.4"] +input = ["text", "image"] +thinking_level_map = { xhigh = "xhigh" } + +[providers.model_metadata."gpt-5.4-mini"] +input = ["text", "image"] +thinking_level_map = { xhigh = "xhigh" } + +[providers.model_metadata."gpt-5.3-codex"] +input = ["text", "image"] +thinking_level_map = { xhigh = "xhigh" } + +[providers.model_metadata."gpt-5.3-codex-spark"] +input = ["text"] +thinking_level_map = { xhigh = "xhigh" } + +[providers.model_metadata."gpt-5.2"] +input = ["text", "image"] +thinking_level_map = { xhigh = "xhigh" } + [[providers]] name = "anthropic" display_name = "Anthropic" @@ -3877,7 +3901,7 @@ kimi-for-coding = 262144 [providers.model_metadata.k3] name = "Kimi K3" reasoning = true -input = ["text"] +input = ["text", "image"] context_window = 1048576 compat = { supportsStore = false, supportsDeveloperRole = false, supportsReasoningEffort = true, maxTokensField = "max_tokens", supportsStrictMode = false } cost = { input = 0, output = 0, cacheRead = 0, cacheWrite = 0 } @@ -6572,6 +6596,45 @@ thinking_parameter = "reasoning_effort" "qwen3.7-max" = 1000000 "qwen3.7-plus" = 1000000 +[providers.model_metadata."deepseek-v4-flash"] +input = ["text"] + +[providers.model_metadata."deepseek-v4-pro"] +input = ["text"] + +[providers.model_metadata."glm-5.1"] +input = ["text"] + +[providers.model_metadata."glm-5.2"] +input = ["text"] + +[providers.model_metadata."kimi-k2.6"] +input = ["text", "image"] + +[providers.model_metadata."kimi-k2.7-code"] +input = ["text", "image"] + +[providers.model_metadata."mimo-v2.5"] +input = ["text", "image"] + +[providers.model_metadata."mimo-v2.5-pro"] +input = ["text"] + +[providers.model_metadata."minimax-m2.7"] +input = ["text"] + +[providers.model_metadata."minimax-m3"] +input = ["text", "image"] + +[providers.model_metadata."qwen3.6-plus"] +input = ["text", "image"] + +[providers.model_metadata."qwen3.7-max"] +input = ["text"] + +[providers.model_metadata."qwen3.7-plus"] +input = ["text", "image"] + [[providers]] name = "opencode" display_name = "OpenCode Zen" @@ -6587,26 +6650,93 @@ thinking_levels = ["off", "minimal", "low", "medium", "high", "xhigh"] thinking_default = "medium" thinking_parameter = "reasoning_effort" +[providers.model_metadata."big-pickle"] +input = ["text"] + +[providers.model_metadata."deepseek-v4-flash"] +input = ["text"] + +[providers.model_metadata."deepseek-v4-flash-free"] +input = ["text"] + +[providers.model_metadata."deepseek-v4-pro"] +input = ["text"] + +[providers.model_metadata."glm-5"] +input = ["text"] + +[providers.model_metadata."glm-5.1"] +input = ["text"] + +[providers.model_metadata."glm-5.2"] +input = ["text"] + [providers.model_metadata."gpt-5.4"] api = "openai-responses" +input = ["text", "image"] [providers.model_metadata."gpt-5.4-mini"] api = "openai-responses" +input = ["text", "image"] [providers.model_metadata."gpt-5.5"] api = "openai-responses" +input = ["text", "image"] [providers.model_metadata."gpt-5.5-pro"] api = "openai-responses" +input = ["text", "image"] [providers.model_metadata."gpt-5.6-luna"] api = "openai-responses" +input = ["text", "image"] [providers.model_metadata."gpt-5.6-sol"] api = "openai-responses" +input = ["text", "image"] [providers.model_metadata."gpt-5.6-terra"] api = "openai-responses" +input = ["text", "image"] + +[providers.model_metadata."grok-4.5"] +input = ["text", "image"] + +[providers.model_metadata."grok-build-0.1"] +input = ["text", "image"] + +[providers.model_metadata."kimi-k2.5"] +input = ["text", "image"] + +[providers.model_metadata."kimi-k2.6"] +input = ["text", "image"] + +[providers.model_metadata."kimi-k2.7-code"] +input = ["text", "image"] + +[providers.model_metadata."mimo-v2.5-free"] +input = ["text", "image"] + +[providers.model_metadata."minimax-m2.5"] +input = ["text"] + +[providers.model_metadata."minimax-m2.7"] +input = ["text"] + +[providers.model_metadata."minimax-m3"] +input = ["text", "image"] + +[providers.model_metadata."nemotron-3-ultra-free"] +input = ["text"] + +[providers.model_metadata."north-mini-code-free"] +input = ["text"] + +[providers.model_metadata."qwen3.5-plus"] +input = ["text", "image"] + +[providers.model_metadata."qwen3.6-plus"] +input = ["text", "image"] [[providers]] name = "github-copilot" @@ -6627,48 +6757,100 @@ headers = { User-Agent = "GitHubCopilotChat/0.35.0", Editor-Version = "vscode/1. [providers.model_metadata."claude-fable-5"] api = "openai-completions" +input = ["text", "image"] [providers.model_metadata."claude-haiku-4.5"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-opus-4.5"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-opus-4.6"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-opus-4.7"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-opus-4.8"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-sonnet-4"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-sonnet-4.5"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-sonnet-4.6"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."claude-sonnet-5"] api = "anthropic-messages" +input = ["text", "image"] [providers.model_metadata."gemini-2.5-pro"] api = "openai-completions" +input = ["text", "image"] [providers.model_metadata."gemini-3-flash-preview"] api = "openai-completions" +input = ["text", "image"] [providers.model_metadata."gemini-3.1-pro-preview"] api = "openai-completions" +input = ["text", "image"] [providers.model_metadata."gemini-3.5-flash"] api = "openai-completions" +input = ["text", "image"] [providers.model_metadata."gpt-4.1"] api = "openai-completions" +input = ["text", "image"] + +[providers.model_metadata."gpt-5-mini"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.2"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.2-codex"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.3-codex"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.4"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.4-mini"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.4-nano"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.5"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.6-luna"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.6-sol"] +input = ["text", "image"] + +[providers.model_metadata."gpt-5.6-terra"] +input = ["text", "image"] [providers.model_metadata."kimi-k2.7-code"] api = "openai-completions" +input = ["text", "image"] + +[providers.model_metadata."mai-code-1-flash-picker"] +input = ["text"] diff --git a/src/tau_coding/provider_config.py b/src/tau_coding/provider_config.py index f850bea45..b66528f98 100644 --- a/src/tau_coding/provider_config.py +++ b/src/tau_coding/provider_config.py @@ -263,6 +263,7 @@ class OpenAICodexProviderConfig: default_model: str = "gpt-5.5" context_windows: dict[str, int] = field(default_factory=dict) headers: dict[str, str] = field(default_factory=dict) + model_metadata: dict[str, ProviderModelMetadata] = field(default_factory=dict) timeout_seconds: float = DEFAULT_OPENAI_COMPATIBLE_TIMEOUT_SECONDS max_retries: int = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRIES max_retry_delay_seconds: float = DEFAULT_OPENAI_COMPATIBLE_MAX_RETRY_DELAY_SECONDS @@ -279,6 +280,7 @@ def __post_init__(self) -> None: max_retry_delay_seconds=self.max_retry_delay_seconds, ) _validate_context_windows(self.context_windows) + _validate_model_metadata(self.models, self.model_metadata) _validate_thinking_config( thinking_levels=self.thinking_levels, thinking_models=self.thinking_models, @@ -299,6 +301,9 @@ def to_json(self) -> dict[str, Any]: "default_model": self.default_model, "context_windows": dict(self.context_windows), "headers": dict(self.headers), + "model_metadata": { + model: metadata.to_json() for model, metadata in self.model_metadata.items() + }, "timeout_seconds": self.timeout_seconds, "max_retries": self.max_retries, "max_retry_delay_seconds": self.max_retry_delay_seconds, @@ -413,6 +418,8 @@ def provider_config_from_entry(entry: ProviderCatalogEntry) -> ProviderConfig: models=entry.models, default_model=entry.default_model, context_windows=context_windows, + headers=dict(entry.headers), + model_metadata=model_metadata, thinking_levels=entry.thinking_levels, thinking_models=entry.thinking_models, thinking_default=entry.thinking_default, @@ -754,6 +761,10 @@ def _merge_provider_config(existing: ProviderConfig, incoming: ProviderConfig) - max_retries=existing.max_retries, max_retry_delay_seconds=existing.max_retry_delay_seconds, context_windows={**incoming.context_windows, **existing.context_windows}, + model_metadata=_merge_provider_model_metadata( + incoming.model_metadata, + existing.model_metadata, + ), thinking_levels=( existing.thinking_levels if existing.thinking_levels is not None @@ -1330,6 +1341,8 @@ def _metadata_supports_thinking_level( metadata: ProviderModelMetadata, level: ThinkingLevel, ) -> bool: + if metadata.reasoning is None and not metadata.thinking_level_map: + return True return _thinking_level_map_supports(metadata.thinking_level_map, level) @@ -1416,6 +1429,12 @@ def _model_max_tokens(provider: ProviderConfig, model: str | None = None) -> int return metadata.max_tokens if metadata is not None else None +def provider_model_supports_images(provider: ProviderConfig, model: str | None = None) -> bool: + selected_model = model or provider.default_model + metadata = _metadata_for_model(provider, selected_model) + return metadata is not None and "image" in metadata.input + + def provider_default_thinking_level( provider: ProviderConfig, *, @@ -1487,6 +1506,7 @@ def openai_compatible_config_from_provider( timeout_seconds=provider.timeout_seconds, max_retries=provider.max_retries, max_retry_delay_seconds=provider.max_retry_delay_seconds, + supports_images=provider_model_supports_images(provider, selected_model), reasoning_effort=reasoning_effort, reasoning_effort_parameter=provider.thinking_parameter or "reasoning_effort", thinking_format=_thinking_format(provider, selected_model), @@ -1522,6 +1542,7 @@ def anthropic_config_from_provider( timeout_seconds=provider.timeout_seconds, max_retries=provider.max_retries, max_retry_delay_seconds=provider.max_retry_delay_seconds, + supports_images=provider_model_supports_images(provider, selected_model), thinking_budget_tokens=thinking_budget_tokens, thinking_effort=_reasoning_effort_from_anthropic_provider( provider, @@ -1797,7 +1818,7 @@ def _provider_from_json(data: object) -> ProviderConfig: thinking_defaults=thinking_defaults, ) if provider_type == "openai-codex": - _reject_catalog_only_legacy_metadata(compat, model_metadata) + _reject_codex_legacy_compat(compat) return OpenAICodexProviderConfig( name=name, base_url=base_url, @@ -1807,6 +1828,7 @@ def _provider_from_json(data: object) -> ProviderConfig: default_model=default_model, context_windows=context_windows, headers=headers, + model_metadata=model_metadata, timeout_seconds=timeout_seconds, max_retries=max_retries, max_retry_delay_seconds=max_retry_delay_seconds, @@ -1971,12 +1993,9 @@ def _validate_json_value(value: object, field_name: str) -> None: raise ProviderConfigError(f"{field_name} must be JSON-compatible") -def _reject_catalog_only_legacy_metadata( - compat: dict[str, Any], - model_metadata: dict[str, ProviderModelMetadata], -) -> None: - if compat or model_metadata: - raise ProviderConfigError("OpenAI Codex legacy provider metadata is not supported") +def _reject_codex_legacy_compat(compat: dict[str, Any]) -> None: + if compat: + raise ProviderConfigError("OpenAI Codex legacy provider compat is not supported") def _validate_thinking_defaults(thinking_defaults: dict[str, ThinkingLevel]) -> None: diff --git a/src/tau_coding/provider_runtime.py b/src/tau_coding/provider_runtime.py index d941cfc05..678bfeafc 100644 --- a/src/tau_coding/provider_runtime.py +++ b/src/tau_coding/provider_runtime.py @@ -33,6 +33,7 @@ ProviderConfigError, anthropic_config_from_provider, openai_compatible_config_from_provider, + provider_model_supports_images, provider_thinking_levels, validate_provider_model, ) @@ -98,6 +99,7 @@ def create_model_provider( model=model, thinking_level=thinking_level, ), + supports_images=provider_model_supports_images(provider, model), ) ) if isinstance(provider, OpenAICompatibleProviderConfig): @@ -134,11 +136,12 @@ def create_model_provider( base_url=compatible_config.base_url, headers=compatible_config.headers, timeout_seconds=compatible_config.timeout_seconds, + provider_name=compatible_config.provider_name, max_retries=compatible_config.max_retries, max_retry_delay_seconds=compatible_config.max_retry_delay_seconds, - provider_name=compatible_config.provider_name, bearer_auth=True, credential_resolver=compatible_config.credential_resolver, + supports_images=compatible_config.supports_images, ) return AnthropicProvider(anthropic_config) if selected_api == "google-generative-ai": diff --git a/src/tau_coding/tools.py b/src/tau_coding/tools.py index f20e516e8..16dc193c5 100644 --- a/src/tau_coding/tools.py +++ b/src/tau_coding/tools.py @@ -12,7 +12,6 @@ import asyncio import difflib import json -import mimetypes import os import signal import tempfile @@ -22,7 +21,7 @@ from time import monotonic from typing import Any -from tau_agent.messages import TextContent +from tau_agent.messages import ImageContent, TextContent from tau_agent.tools import ( AgentTool, AgentToolResult, @@ -33,7 +32,8 @@ DEFAULT_MAX_OUTPUT_BYTES = 50 * 1024 DEFAULT_MAX_OUTPUT_LINES = 2_000 -SUPPORTED_IMAGE_MIME_TYPES = {"image/jpeg", "image/png", "image/gif", "image/webp"} +DEFAULT_MAX_IMAGE_BYTES = 5 * 1024 * 1024 +PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n" UTF8_BOM = "\ufeff" @@ -174,15 +174,32 @@ async def execute( mime_type = _detect_supported_image_mime_type(path) if mime_type is not None: + image_bytes = path.stat().st_size + image_details: dict[str, JSONValue] = { + "path": str(path), + "mime_type": mime_type, + "bytes": image_bytes, + } + if image_bytes > DEFAULT_MAX_IMAGE_BYTES: + return AgentToolResult( + content=[ + TextContent( + text=( + f"Read image file [{mime_type}]\n" + f"[Image omitted: {format_size(image_bytes)} exceeds the " + f"{format_size(DEFAULT_MAX_IMAGE_BYTES)} attachment limit.]" + ) + ) + ], + details=image_details, + ) data = path.read_bytes() return AgentToolResult( - content=[TextContent(text=f"Read image file [{mime_type}]")], - details={ - "path": str(path), - "mime_type": mime_type, - "bytes": len(data), - "image_base64": _base64_text(data), - }, + content=[ + TextContent(text=f"Read image file [{mime_type}]"), + ImageContent(data=_base64_text(data), mime_type=mime_type), + ], + details=image_details, ) text = path.read_text(encoding="utf-8") @@ -246,7 +263,8 @@ async def execute( name="read", description=( "Read the contents of a file. Supports text files and images (jpg, png, gif, webp). " - "Images are returned as base64 metadata. For text files, output is truncated to " + "Images are sent to vision-capable models as attachments. For text files, output is " + "truncated to " f"{DEFAULT_MAX_OUTPUT_LINES} lines or {DEFAULT_MAX_OUTPUT_BYTES // 1024}KB " "(whichever is hit first). Use offset/limit for large files. When you need the " "full file, continue with offset until complete." @@ -1006,8 +1024,34 @@ def _no_change_error(path: str, total_edits: int) -> str: def _detect_supported_image_mime_type(path: Path) -> str | None: - mime_type, _encoding = mimetypes.guess_type(path) - return mime_type if mime_type in SUPPORTED_IMAGE_MIME_TYPES else None + with path.open("rb") as file: + header = file.read(64 * 1024) + if header.startswith(b"\xff\xd8\xff"): + return "image/jpeg" + if header.startswith(PNG_SIGNATURE): + return None if _is_animated_png(header) else "image/png" + if header.startswith((b"GIF87a", b"GIF89a")): + return "image/gif" + if header.startswith(b"RIFF") and header[8:12] == b"WEBP": + return "image/webp" + + return None + + +def _is_animated_png(data: bytes) -> bool: + offset = len(PNG_SIGNATURE) + while offset + 8 <= len(data): + chunk_length = int.from_bytes(data[offset : offset + 4], "big") + chunk_type = data[offset + 4 : offset + 8] + if chunk_type == b"acTL": + return True + if chunk_type == b"IDAT": + return False + next_offset = offset + 12 + chunk_length + if next_offset <= offset or next_offset > len(data): + return False + offset = next_offset + return False def _base64_text(data: bytes) -> str: diff --git a/tests/test_coding_tools.py b/tests/test_coding_tools.py index 1ff239366..9fc08af9c 100644 --- a/tests/test_coding_tools.py +++ b/tests/test_coding_tools.py @@ -5,6 +5,7 @@ import pytest +from tau_agent import ImageContent from tau_coding import ( create_bash_tool, create_coding_tools, @@ -68,6 +69,51 @@ async def test_read_tool_reads_file_with_offset_and_limit(tmp_path: Path) -> Non assert isinstance(result.details["truncation"], dict) +@pytest.mark.anyio +async def test_read_tool_returns_images_as_model_content(tmp_path: Path) -> None: + image_data = b"\x89PNG\r\n\x1a\nnot-real" + path = tmp_path / "diagram.png" + path.write_bytes(image_data) + tool = create_read_tool(cwd=tmp_path) + + result = await tool.execute("test-call", {"path": "diagram.png"}) + + assert result.text == "Read image file [image/png]" + image = next(block for block in result.content if isinstance(block, ImageContent)) + assert image.mime_type == "image/png" + assert image.data == "iVBORw0KGgpub3QtcmVhbA==" + assert result.details == { + "path": str(path), + "mime_type": "image/png", + "bytes": len(image_data), + } + + +@pytest.mark.anyio +async def test_read_tool_detects_images_by_content_not_extension(tmp_path: Path) -> None: + path = tmp_path / "diagram.txt" + path.write_bytes(b"\x89PNG\r\n\x1a\nnot-real") + tool = create_read_tool(cwd=tmp_path) + + result = await tool.execute("test-call", {"path": "diagram.txt"}) + + assert any(isinstance(block, ImageContent) for block in result.content) + + +@pytest.mark.anyio +async def test_read_tool_omits_oversized_images(tmp_path: Path) -> None: + path = tmp_path / "large.png" + with path.open("wb") as file: + file.write(b"\x89PNG\r\n\x1a\n") + file.truncate(5 * 1024 * 1024 + 1) + tool = create_read_tool(cwd=tmp_path) + + result = await tool.execute("test-call", {"path": "large.png"}) + + assert "exceeds the 5.0MB attachment limit" in result.text + assert not any(isinstance(block, ImageContent) for block in result.content) + + @pytest.mark.anyio async def test_read_tool_treats_zero_offset_as_start_of_file(tmp_path: Path) -> None: path = tmp_path / "notes.txt" diff --git a/tests/test_multimodal_provider_payloads.py b/tests/test_multimodal_provider_payloads.py new file mode 100644 index 000000000..9a7767f0d --- /dev/null +++ b/tests/test_multimodal_provider_payloads.py @@ -0,0 +1,124 @@ +from tau_agent import ImageContent, TextContent, ToolResultMessage +from tau_ai.anthropic import _build_messages_payload +from tau_ai.google import _build_google_payload +from tau_ai.mistral import _build_mistral_payload +from tau_ai.openai_codex import _build_codex_payload +from tau_ai.openai_compatible import _build_chat_payload, _build_responses_payload + + +def _image_result() -> ToolResultMessage: + return ToolResultMessage( + tool_call_id="call-1", + tool_name="read", + content=[ + TextContent(text="Read image file [image/png]"), + ImageContent(data="aW1hZ2U=", mime_type="image/png"), + ], + ) + + +def test_anthropic_embeds_image_in_tool_result() -> None: + payload = _build_messages_payload( + model="claude", + system="system", + messages=[_image_result()], + tools=[], + supports_images=True, + ) + + result = payload["messages"][0]["content"][0] + assert result["content"][1] == { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": "aW1hZ2U=", + }, + } + + +def test_openai_responses_embeds_image_in_function_output() -> None: + payload = _build_responses_payload( + model="gpt", + system="system", + messages=[_image_result()], + tools=[], + supports_images=True, + ) + + result = payload["input"][0] + assert result["output"][1] == { + "type": "input_image", + "detail": "auto", + "image_url": "data:image/png;base64,aW1hZ2U=", + } + + +def test_codex_embeds_image_in_function_output() -> None: + payload = _build_codex_payload( + model="codex", + system="system", + messages=[_image_result()], + tools=[], + supports_images=True, + ) + + result = payload["input"][0] + assert result["output"][1]["type"] == "input_image" + + +def test_openai_chat_attaches_tool_images_in_followup_user_message() -> None: + payload = _build_chat_payload( + model="vision", + system="system", + messages=[_image_result()], + tools=[], + supports_images=True, + ) + + assert [message["role"] for message in payload["messages"]] == ["system", "tool", "user"] + assert payload["messages"][2]["content"][1]["type"] == "image_url" + + +def test_google_embeds_image_in_gemini_3_function_response() -> None: + payload = _build_google_payload( + model="gemini-3-pro", + system="system", + messages=[_image_result()], + tools=[], + reasoning_effort=None, + max_tokens=None, + supports_images=True, + ) + + response = payload["contents"][0]["parts"][0]["functionResponse"] + assert response["parts"][0] == {"inlineData": {"mimeType": "image/png", "data": "aW1hZ2U="}} + + +def test_mistral_attaches_tool_images_in_followup_user_message() -> None: + payload = _build_mistral_payload( + model="pixtral", + system="system", + messages=[_image_result()], + tools=[], + reasoning_effort=None, + max_tokens=None, + supports_images=True, + ) + + assert [message["role"] for message in payload["messages"]] == ["system", "tool", "user"] + assert payload["messages"][2]["content"][1]["type"] == "image_url" + + +def test_non_vision_models_receive_placeholder_instead_of_image() -> None: + payload = _build_responses_payload( + model="text-only", + system="system", + messages=[_image_result()], + tools=[], + supports_images=False, + ) + + result = payload["input"][0] + assert result["output"].endswith("(tool image omitted: model does not support images)") + assert "aW1hZ2U=" not in str(payload) diff --git a/tests/test_provider_catalog.py b/tests/test_provider_catalog.py index f55546a77..dd553f3a4 100644 --- a/tests/test_provider_catalog.py +++ b/tests/test_provider_catalog.py @@ -161,6 +161,100 @@ def test_builtin_catalog_separates_openai_api_and_codex_context_limits() -> None assert codex.model_metadata["gpt-5.6-sol"].context_window == 272_000 +@pytest.mark.parametrize( + ("provider_name", "vision_models"), + [ + ( + "openai-codex", + { + "gpt-5.6-sol", + "gpt-5.6-terra", + "gpt-5.6-luna", + "gpt-5.5", + "gpt-5.4", + "gpt-5.4-mini", + "gpt-5.3-codex", + "gpt-5.2", + }, + ), + ( + "opencode-go", + { + "kimi-k2.6", + "kimi-k2.7-code", + "mimo-v2.5", + "minimax-m3", + "qwen3.6-plus", + "qwen3.7-plus", + }, + ), + ( + "opencode", + { + "gpt-5.4", + "gpt-5.4-mini", + "gpt-5.5", + "gpt-5.5-pro", + "gpt-5.6-luna", + "gpt-5.6-sol", + "gpt-5.6-terra", + "grok-4.5", + "grok-build-0.1", + "kimi-k2.5", + "kimi-k2.6", + "kimi-k2.7-code", + "mimo-v2.5-free", + "minimax-m3", + "qwen3.5-plus", + "qwen3.6-plus", + }, + ), + ( + "github-copilot", + { + "claude-fable-5", + "claude-haiku-4.5", + "claude-opus-4.5", + "claude-opus-4.6", + "claude-opus-4.7", + "claude-opus-4.8", + "claude-sonnet-4", + "claude-sonnet-4.5", + "claude-sonnet-4.6", + "claude-sonnet-5", + "gemini-2.5-pro", + "gemini-3-flash-preview", + "gemini-3.1-pro-preview", + "gemini-3.5-flash", + "gpt-4.1", + "gpt-5-mini", + "gpt-5.2", + "gpt-5.2-codex", + "gpt-5.3-codex", + "gpt-5.4", + "gpt-5.4-mini", + "gpt-5.4-nano", + "gpt-5.5", + "gpt-5.6-luna", + "gpt-5.6-sol", + "gpt-5.6-terra", + "kimi-k2.7-code", + }, + ), + ], +) +def test_sparse_provider_catalogs_declare_model_input_modalities( + provider_name: str, vision_models: set[str] +) -> None: + provider = builtin_provider_entry(provider_name) + + assert provider is not None + assert set(provider.model_metadata) == set(provider.models) + assert { + model for model, metadata in provider.model_metadata.items() if "image" in metadata.input + } == vision_models + + def test_builtin_catalog_oauth_and_opencode_auth_methods() -> None: codex = builtin_provider_entry("openai-codex") copilot = builtin_provider_entry("github-copilot") @@ -317,7 +411,7 @@ def test_builtin_catalog_golden_kimi_entries() -> None: k3 = coding.model_metadata["k3"] assert k3.name == "Kimi K3" assert k3.reasoning is True - assert k3.input == ("text",) + assert k3.input == ("text", "image") assert k3.context_window == 1_048_576 assert k3.thinking_level_map == { "off": None, diff --git a/tests/test_provider_config.py b/tests/test_provider_config.py index 8f9321666..0bb82a8c8 100644 --- a/tests/test_provider_config.py +++ b/tests/test_provider_config.py @@ -21,6 +21,7 @@ openai_compatible_config_from_provider, provider_default_thinking_level, provider_has_usable_credentials, + provider_model_supports_images, provider_settings_from_json, provider_thinking_levels, provider_thinking_unavailable_reason, @@ -73,6 +74,23 @@ def test_load_provider_settings_missing_file_uses_openai_default(tmp_path: Path) assert settings.get_provider("huggingface").api_key_env == "HF_TOKEN" +def test_builtin_codex_preserves_model_input_capabilities() -> None: + codex = ProviderSettings().get_provider("openai-codex") + + for model in ( + "gpt-5.6-sol", + "gpt-5.6-terra", + "gpt-5.6-luna", + "gpt-5.5", + "gpt-5.4", + "gpt-5.4-mini", + "gpt-5.3-codex", + "gpt-5.2", + ): + assert provider_model_supports_images(codex, model) + assert not provider_model_supports_images(codex, "gpt-5.3-codex-spark") + + def test_builtin_openai_declares_model_scoped_thinking_capabilities() -> None: settings = ProviderSettings() openai = settings.get_provider("openai") @@ -683,6 +701,25 @@ def test_set_default_provider_model_rejects_model_not_declared_for_provider() -> set_default_provider_model(settings, provider_name="local", model="llama") +def test_runtime_config_uses_selected_model_image_capability( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("CUSTOM_API_KEY", "secret") + provider = OpenAICompatibleProviderConfig( + name="custom", + api_key_env="CUSTOM_API_KEY", + models=("text", "vision"), + default_model="text", + model_metadata={ + "text": ProviderModelMetadata(input=("text",)), + "vision": ProviderModelMetadata(input=("text", "image")), + }, + ) + + assert not openai_compatible_config_from_provider(provider, model="text").supports_images + assert openai_compatible_config_from_provider(provider, model="vision").supports_images + + def test_openai_compatible_config_from_provider_uses_configured_env_var( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/tests/test_provider_runtime.py b/tests/test_provider_runtime.py index 34dd0c662..8f1afc488 100644 --- a/tests/test_provider_runtime.py +++ b/tests/test_provider_runtime.py @@ -26,6 +26,27 @@ def test_create_model_provider_returns_openai_codex_provider(tmp_path) -> None: assert isinstance(provider, OpenAICodexProvider) +def test_create_model_provider_uses_codex_model_image_capability(tmp_path) -> None: + store = FileCredentialStore(tmp_path / "credentials.json") + config = provider_config_from_catalog_entry("openai-codex") + + vision_provider = create_model_provider( + config, + credential_store=store, + model="gpt-5.6-sol", + ) + text_provider = create_model_provider( + config, + credential_store=store, + model="gpt-5.3-codex-spark", + ) + + assert isinstance(vision_provider, OpenAICodexProvider) + assert isinstance(text_provider, OpenAICodexProvider) + assert vision_provider._config.supports_images is True + assert text_provider._config.supports_images is False + + def test_create_model_provider_uses_anthropic_oauth_runtime_auth(tmp_path) -> None: store = FileCredentialStore(tmp_path / "credentials.json") store.set_oauth( diff --git a/tests/test_session.py b/tests/test_session.py index 6bd47643b..72954651c 100644 --- a/tests/test_session.py +++ b/tests/test_session.py @@ -6,6 +6,7 @@ from tau_agent import ( AssistantMessage, CustomMessage, + ImageContent, TextContent, ThinkingContent, ToolResultMessage, @@ -88,6 +89,29 @@ def test_assistant_and_tool_result_round_trip_canonical_blocks() -> None: assert entry_from_json_line(entry_to_json_line(result)) == result +def test_tool_result_image_round_trips_jsonl() -> None: + entry = MessageEntry( + id="image", + message=ToolResultMessage( + tool_call_id="call-1", + tool_name="read", + content=[ + TextContent(text="Read image file [image/png]"), + ImageContent(data="aW1hZ2U=", mime_type="image/png"), + ], + ), + ) + + line = entry_to_json_line(entry) + + assert entry_from_json_line(line) == entry + assert json.loads(line)["message"]["content"][1] == { + "type": "image", + "data": "aW1hZ2U=", + "mimeType": "image/png", + } + + def test_structured_thinking_message_round_trips_jsonl() -> None: entry = MessageEntry( id="a", diff --git a/tests/test_tau_ai.py b/tests/test_tau_ai.py index 9fd767961..f55cc422c 100644 --- a/tests/test_tau_ai.py +++ b/tests/test_tau_ai.py @@ -8,6 +8,7 @@ AgentTool, AgentToolResult, AssistantMessage, + ImageContent, SimpleCancellationToken, TextContent, ThinkingContent, @@ -2715,3 +2716,82 @@ def handler(_request: httpx.Request) -> httpx.Response: assert usage.reasoning is None assert usage.cache_read == 0 assert usage.total_tokens == 12 + + +@pytest.mark.anyio +async def test_github_copilot_sends_vision_header_for_tool_result_images() -> None: + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response( + 200, + text='data: {"type":"response.completed","response":{"status":"completed"}}\n\n', + headers={"content-type": "text/event-stream"}, + ) + + message = ToolResultMessage( + tool_call_id="call-1", + tool_name="read", + content=[ImageContent(data="aW1hZ2U=", mime_type="image/png")], + ) + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client: + provider = OpenAICompatibleProvider( + OpenAICompatibleConfig( + api_key="test-key", + base_url="https://copilot.test", + api="openai-responses", + provider_name="github-copilot", + supports_images=True, + ), + client=client, + ) + await _collect( + provider.stream_response( + model="gpt-5.4", + system="You are Tau.", + messages=[message], + tools=[], + ) + ) + + assert requests[0].headers["Copilot-Vision-Request"] == "true" + + +@pytest.mark.anyio +async def test_github_copilot_anthropic_sends_vision_header() -> None: + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response( + 200, + text='data: {"type":"message_stop"}\n\n', + headers={"content-type": "text/event-stream"}, + ) + + message = ToolResultMessage( + tool_call_id="call-1", + tool_name="read", + content=[ImageContent(data="aW1hZ2U=", mime_type="image/png")], + ) + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client: + provider = AnthropicProvider( + AnthropicConfig( + api_key="test-key", + base_url="https://copilot.test", + provider_name="github-copilot", + supports_images=True, + ), + client=client, + ) + await _collect( + provider.stream_response( + model="claude-sonnet-4.6", + system="You are Tau.", + messages=[message], + tools=[], + ) + ) + + assert requests[0].headers["Copilot-Vision-Request"] == "true" diff --git a/website/content/guides/providers-and-models.md b/website/content/guides/providers-and-models.md index c10c14ac9..7697d6d4b 100644 --- a/website/content/guides/providers-and-models.md +++ b/website/content/guides/providers-and-models.md @@ -78,7 +78,9 @@ configured fallback. Live limits can vary by account or rollout and may change independently of Tau. A discovery failure is non-fatal: Tau reports it in `/session` and continues with the fallback. Direct OpenAI API sessions retain the context limits documented on -the API model page. The `gpt-5.6` alias, which routes to GPT-5.6 Sol, is only +the API model page. Vision-capable Codex models retain their image-input +metadata separately from these runtime context limits, allowing image files read +by Tau to reach the model. The `gpt-5.6` alias, which routes to GPT-5.6 Sol, is only available through the direct OpenAI API; Codex subscription users should select the explicit `gpt-5.6-sol` model instead. @@ -119,8 +121,9 @@ different endpoints, and charge against different billing plans: | `moonshotai` | Pay-as-you-go key from the [Kimi Open Platform](https://platform.kimi.ai/console/api-keys) | `kimi-k2.7-code` | `https://api.moonshot.ai/v1` | `MOONSHOT_API_KEY` | | `kimi-code` | Subscription key from the [Kimi Code console](https://www.kimi.com/code/console) | `k3` or rolling `kimi-for-coding` alias | `https://api.kimi.com/coding/v1` | `KIMI_CODE_API_KEY` | -Kimi K3 uses the `k3` model ID and supports up to a 1,048,576-token context -window on eligible plans. Its reasoning effort is currently fixed at `max`, +Kimi K3 uses the `k3` model ID, accepts text and image input, and supports up to +a 1,048,576-token context window on eligible plans. Its reasoning effort is +currently fixed at `max`, which Tau exposes as the `xhigh` thinking level. Start a new session when switching to K3 so the previous model's context cache is not re-prefilled. See [Kimi's model documentation](https://www.kimi.com/code/docs/en/kimi-code/models) diff --git a/website/content/reference/tools.md b/website/content/reference/tools.md index 98169a175..af47b211c 100644 --- a/website/content/reference/tools.md +++ b/website/content/reference/tools.md @@ -33,7 +33,11 @@ Reads a file from disk. For text files, `read` returns UTF-8 content, applies `offset`/`limit`, and truncates to at most 2,000 lines or 50 KB (whichever comes first), appending a hint like `[42 more lines in file. Use offset=101 to continue.]`. Supported -images (JPEG, PNG, GIF, WebP) are returned as base64 with metadata. +images (JPEG, PNG, GIF, WebP) are detected from their file content and sent to +vision-capable models as image attachments. Attachments are limited to 5 MB. +When the active model does not accept images, Tau sends an explicit +omission notice instead of silently dropping the image or submitting an invalid +request. Fails when `path` is missing/invalid, the file doesn't exist, the path is a directory, `offset` is past the end, or the file is neither UTF-8 text nor a