Skip to content

Send read-tool images to vision models#479

Draft
alejandro-ao wants to merge 4 commits into
mainfrom
feat/multimodal-read-results
Draft

Send read-tool images to vision models#479
alejandro-ao wants to merge 4 commits into
mainfrom
feat/multimodal-read-results

Conversation

@alejandro-ao

@alejandro-ao alejandro-ao commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Description

Send supported images read from disk to vision-capable models as canonical multimodal tool results.

  • return ImageContent from the built-in read tool
  • serialize tool-result images for Anthropic, OpenAI Responses/Codex, OpenAI Chat Completions, Google, and Mistral
  • replace images with an explicit omission marker for text-only models
  • detect image formats from file magic and cap attachments at 5 MB
  • persist image content through session JSONL without duplicating base64 in tool details
  • preserve OpenAI Codex model metadata through runtime provider creation
  • declare input modalities for every OpenAI Codex, OpenCode Go, OpenCode Zen, and GitHub Copilot model, matching Pi's provider catalog
  • send GitHub Copilot's required Copilot-Vision-Request header when context contains images
  • correct Kimi K3 catalog metadata to enable its documented native visual understanding

User experience

Agents using vision-capable models—including Kimi K3, GPT-5.6 Sol through OpenAI Codex, OpenCode-hosted vision models, and GitHub Copilot vision models—can inspect screenshots, diagrams, and other supported project images instead of receiving only image metadata. Text-only models clearly report that visual context was omitted rather than failing the provider request or silently losing it.

Issue

No linked issue. Addresses the missing provider integration noted in the original image-read implementation.

Manual validation

  1. Select a vision-capable model such as kimi-code:k3 or openai-codex:gpt-5.6-sol.
  2. Start Tau in a project containing a JPEG, PNG, GIF, or WebP file smaller than 5 MB.
  3. Ask Tau to read and describe the image.
  4. Confirm the provider receives the image and the model describes its contents.
  5. Repeat with a text-only model and confirm the tool result includes (tool image omitted: model does not support images).

Checks

  • uv run pytest
  • uv run ruff check .
  • uv run ruff format --check .
  • uv run mypy
  • uv build
  • hugo --minify --destination /tmp/tau-multimodal-read-site

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant