Image input support

[aa956]

What is the best way to implement image input?

As I'm looking at the scripts generating SVG files using late image input would be really nice to have.

• MCP? But it will write base64 image into the "type": "text" of request, so instead of MCP late can just use command line base64 for that, model will not get the image encoded into vision tokens if base64 string will be sent as text input.
• Some other way that I'm missing?

[aa956]

Example of the behavior I'd like to have during late sessions:

[aa956]

Here's the python image input request example:

#!/usr/bin/env python3

import base64
import requests

OPENAI_BASE_URL = "http://localhost:9280"
OPENAI_MODEL="qwen3.6-27b-q4-c128k"
OPENAI_API_KEY = ""

# base64 encode the file
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

# Example image from
# https://en.wikipedia.org/wiki/Computer_network_diagram#/media/File:Sample-network-diagram.png
image_path = "./examples/Sample-network-diagram.png"

# base64 string
base64_image = encode_image(image_path)

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {OPENAI_API_KEY}"
}

# mimetype image/jpeg does not matter for llama.cpp llama-server
# it only reads bytes from request, so we can send PNG too.

payload = {
    "model": OPENAI_MODEL,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image."
          },
          {
            "type": "image_url",
            "image_url": {
              "url": f"data:image/jpeg;base64,{base64_image}"
            }
          }
        ]
      }
    ],
    "max_tokens": 16384
}

response = requests.post(f"{OPENAI_BASE_URL}/v1/chat/completions", headers=headers, json=payload)

if response.status_code == 200:
    result = response.json()
    print(result["choices"][0]["message"]["content"])
else:
    print(f"Error posting request, status code {response.status_code}")

$ uv run python post-image-example.py 
This image is a schematic diagram illustrating a basic
computer network topology. It depicts how various hardware
components are connected to share resources and access the internet.

Here is a breakdown of the components from left to right:

- **Client Computers:** On the far left, there are three
  desktop computers, each consisting of a tower case and a monitor.
- **Switch:** These three computers are connected via
  black lines to a square device labeled "Switch."
- **Server:** The Switch is connected to a large, vertical
  tower in the center labeled "Server." This acts as
  the central hub for the local network.
- **Printer:** Connected to the top of the
  Server is a device labeled "Printer."
- **Router:** To the right of the Server,
  there is a device labeled "Router."
- **The Internet:** The Router is connected via a jagged line
  to a cloud shape labeled "The Internet," indicating the gateway
  to the outside world.

In summary, the diagram shows a local network where
three user computers connect through a switch to a central server,
which manages a printer and provides internet access through a router.

[giveen]

@aa956 so the codebase uses Bubbletea as its TUI framework and it does not currently support images.
I can see where this would be great for editing things like websites to ensure that things are lined up right or the coder chose the right image, its not feasible with the current TUI.

[aa956]

@giveen, idea is not to show images to user but to upload them for checking by multimodal model.

Use case:

Working on a script generating SVG.
Model that can natively process images can directly view file from the working directory as image. Something like that:

• Run test of the currently updated code that produces SVG (e.g. bash tool python script.py /example/data.yaml > /example/diagram.svg)
• Convert SVG to PNG (bash tool rsvg-convert diagram.svg > diagram.png or inkscape)
• Post PNG to /v1/chat/completions in image_url
• Review the result

It's a very low priority but probably useful feature to have if project worked on generates images somehow (UI that can be screenshotted and reviewed by a model or like my case now, script generating SVG drawings).

For now I'm just using chat UI for that (dumping entire project's sources in a single Markdown file there together with image):

[mlhher]

I will look into it. The issue is not necessarily passing the image to the model that is more or less trivial (as your own python script shows the oai api handles that).

[aa956]

I see that it's implemented in 74bf449

But how to force the SupportsVision()?

Cannot get it to work for any local model.

Edit:
Looks like this checks for /props, so will not work with proxies, only llama-server directly.

Edit2:
It's working with direct llama-server access, thank you!
Combined with playwright MCP means that I can use late-cli for basic web tasks and so on.

[aa956]

Improvement idea: make it user configurable, as there are only 2 models now (main and subagent), then 2 switches in environment variables are enough?

It'll be simpler than parsing e.g. /v1/models for all possible providers that's not standardized, e.g.:

• openrouter uses architecture - input_modalities list in /v1/models responses and
• Mistral uses capabilities - vision boolean

[mlhher]

Other APIs do not seem to explicitly advertise model support. This is also the reason why the token querying only supports llama-server other servers do not seem to support it. Any other API seems to rely on us "just trying" and then returning an error if the model does not support vision. The /props endpoint does a lot of work here.

Might have to, as you suggested, go the user-configurable route.

[giveen]

@mlhher kinda long this line, i was thinking about building out a image input, but it would need to be able to detect that the local model has the mmproj .

So a user would go "Heh look at /mnt/storage/path/image.png this is how I want my UI to look", and maybe have an environmental variable of VISION_SUBAGENT

[mlhher]

If you mean a subagent solely for vision tasks I am expecting it to run into a similar problem as the researcher subagent. If the orchestrator already has the toolset available (read files, execute bash, view an image) to it it will likely default to lazy and do the task itself instead of orchestrating a subagent to do it (assuming smaller models). Compare to writing code where it physically cannot do the task. This is pretty much the reason why I haven't merged it yet (testing with Qwen-AgentWorld). Taking away the orchestrators ability to do the task (reading files, viewing an image) is not really a solution that works outside of writing code as reading code and viewing an image are likely (always) required to synthesize a coherent plan and adjust along the way if details change.

If I misunderstood please explain.

[giveen]

nope, you hit it right, how do we get an LLM to look at it.
Possible a command "/vision" then this would activate the SUBAGENT_VISION model router switch, then "/vision Examine IMAGE XYZ AND DESCRIBE", it will output the vision models description and drop it into a vision_description.md document. going back to my dropped "/model" command that would go back to the primary orchestrator/subagent.

[mlhher]

I was about to say I don't think having a /vision one would be helpful, not because of the feature itself I do think it is helpful but I think users would simply not use. Thinking about it we might have something like /task though where users can simply schedule subagents themselves as currently only the orchestrator can schedule subagents. Might need to evaluate this long term though I don't think this is something for the immediate time.