Support Image Analyzation Task Type

[lukasdotcom]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

---

Feature request

Is your feature request related to a problem? Please describe.
In nextcloud/server#53763 an analyze image task type was added with support in openai integration. There should also be support for a local model built into llm2.

Describe the solution you'd like
Using llama_cpp python library directly you can use multi modal models such as SmolVLM. This is some example code of how to call the model (note for SmolVLM you need two files for the model):

messages = [
  {
      "role": "user",
      "content": [
          {"type": "image_url", "image_url": {
              "url": image_to_base64_data_uri(image_data)
          }},
          {"type": "text", "text": question}
      ]
  },
]
chat_handler = Llava15ChatHandler(clip_model_path="/models/mmproj-model-f16.gguf")
llm = Llama(
  model_path="/models/ggml-model-q4_k.gguf",
  chat_handler=chat_handler,
  n_ctx=2048,
)
response = llm.create_chat_completion(messages=messages)
output_text = response["choices"][0]["message"]['content']

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.