Image Input (Multi-modal models)

[Legerdo]

Describe the feature

hello.
llamafile seems to have image input functions such as jpg/png/gif/bmp.

Example)
llamafile -ngl 9999 --temp 0
--image ~/Pictures/lemurs.jpg
-m llava-v1.5-7b-Q4_K.gguf
--mmproj llava-v1.5-7b-mmproj-Q4_0.gguf
-e -p '### User: What do you see?\n### Assistant: '
--no-display-prompt 2>/dev/null

Is it possible to implement this feature in the future?
Or is there some problem that makes it impossible?

[amakropoulos]

hi, thanks for the request!
that should be feasible.
how would you like to use it / see it inside Unity?

[Legerdo]

Eventually, I would like to add the ability to describe to the user what the NPC character's camera (eyes) sees.

I haven't tested this against the local vision model yet, so I don't know to what extent it's possible, but it would be interesting if it were!

[amakropoulos]

I implemented most of the functionality in this branch: feature/multimodal_models
and I afterwards figured out that multimodal support has been dropped from the llama.cpp server and not brought back for the last months: ggml-org/llama.cpp#8010 😞

[N0RAIR]

It seems its coming back reading the comments in llama.cpp/issues/8010
Will there be a further adaptation here? It would be great to be able to pass a texture to the model with a string asking something about the texture that was passed to it. If possible, array of textures.

[amakropoulos]

Yes it will need adaptations, I will do them once it comes back.