Use CodeCompanion.nvim with OpenWebUI

[eldios]

Hello,
I'm trying to have CodeCompanion work OpenWebUI[*] as a frontend for OLLAMA.

You can see below that sadly the endpoints are "slightly" different then what you'd expect so it's not working from me.

https://docs.openwebui.com/getting-started/advanced-topics/api-endpoints/

Any idea or anybody successfully had it working already?

I'm open to write an adapter dedicated to it if you think it may help.

Thank you.

[*]
https://github.com/open-webui/open-webui
https://openwebui.com

[olimorris]

Unless I'm missing something...CodeCompanion is a frontend for Ollama and other adapters. However, instead of using the browser like OpenWebUI, it uses Neovim.

[eldios]

hey @olimorris not sure how to read your comment, sorry :)

Let me rephrase my case: I'm using OpenWebUI as a frontend to OLLAMA and OLLAMA is NOT exposed at all, so the only way I have to reach OLLAMA is through OpenWEBUI.

all my other adapters work, but the OLLAMA one doesn't cause the endpoints in OpenWEBUI are just slightly different than the default ones for OLLAMA, thus I THINK it will need a different adapter to work.

Let me know if that makes more sense.
Thank you

[olimorris]

Have you read the docs? I've spent considerable time and effort covering these scenarios.

[eldios]

I did but I haven't seen anything related. The only one that seemed related is using the openai-compatible adapter but it uses different endpoints. If you could kindly point me in the direction you're thinking do cover my case, I'll re-read again.

[olimorris]

You may need to create your own adapter so I'd advise reading that guide. Also, a great tip is to use the fetch slash command to share both that guide and the docs of OpenWebUI and ask the LLM to help you.

[lll9p]

@eldios @olimorris Hello! I have written an Open-WebUI adapter (by making some simple modifications to the Ollama adapter). It is currently available for basic use (not thoroughly tested). It is possible to add some features specific to Open-WebUI on this basis.

You can use it as follows:

open_webui = function()
   return require("codecompanion.adapters").extend(require "myconfigs.open_webui", {...})
end

Below is the code:

Code of open_webui adapter

local config = require "codecompanion.config"
local curl = require "plenary.curl"
local log = require "codecompanion.utils.log"
local utils = require "codecompanion.utils.adapters"

---Get a list of available OpenWebui models
---@params self CodeCompanion.Adapter
---@params opts? table
---@return table
local function get_models(self, opts)
   local adapter = require("codecompanion.adapters").resolve(self)
   if not adapter then
      log:error "Could not resolve OpenWebui adapter in the `get_models` function"
      return {}
   end

   adapter:get_env_vars()
   local url = adapter.env_replaced.url

   local headers = {
      ["content-type"] = "application/json",
   }
   if adapter.env_replaced.api_key then
      headers["Authorization"] = "Bearer " .. adapter.env_replaced.api_key
   end

   local ok, response = pcall(function()
      return curl.get(url .. "/api/models", {
         sync = true,
         headers = headers,
         insecure = config.adapters.opts.allow_insecure,
         proxy = config.adapters.opts.proxy,
      })
   end)
   if not ok then
      log:error("Could not get the OpenWebui models from " .. url .. "/api/models.\nError: %s", response)
      return {}
   end

   local ok, json = pcall(vim.json.decode, response.body)
   if not ok then
      log:error("Could not parse the response from " .. url .. "/api/models")
      return {}
   end

   local models = {}
   for _, model in ipairs(json.data) do
      table.insert(models, model.id)
   end

   if opts and opts.last then
      return models[1]
   end
   return models
end

---@class OpenWebui.Adapter: CodeCompanion.Adapter
return {
   name = "OpenWebui",
   formatted_name = "OpenWebui",
   roles = {
      llm = "assistant",
      user = "user",
   },
   opts = {
      stream = true,
   },
   features = {
      text = true,
      tokens = true,
      vision = false,
   },
   url = "${url}/api/chat/completions",
   env = {
      url = "http://localhost:8080",
      api_key = "OPENWEBUI_API_KEY",
   },
   headers = {
      ["Content-Type"] = "application/json",
      Authorization = "Bearer ${api_key}",
   },
   handlers = {
      ---@param self CodeCompanion.Adapter
      ---@return boolean
      setup = function(self)
         self.parameters.stream = false
         if self.opts and self.opts.stream then
            self.parameters.stream = true
         end

         return true
      end,

      ---Set the parameters
      ---@param self CodeCompanion.Adapter
      ---@param params table
      ---@param messages table
      ---@return table
      form_parameters = function(self, params, messages)
         return params
      end,

      ---Set the format of the role and content for the messages from the chat buffer
      ---@param self CodeCompanion.Adapter
      ---@param messages table Format is: { { role = "user", content = "Your prompt here" } }
      ---@return table
      form_messages = function(self, messages)
         messages = utils.merge_messages(messages)
         return { messages = messages }
      end,

      ---Returns the number of tokens generated from the LLM
      ---@param self CodeCompanion.Adapter
      ---@param data table The data from the LLM
      ---@return number|nil
      tokens = function(self, data)
         if data and data ~= "" then
            local data_mod = utils.clean_streamed_data(data)
            local ok, json = pcall(vim.json.decode, data_mod, { luanil = { object = true } })

            if not ok then
               return
            end

            if json.eval_count then
               log:debug("Done! %s", json.eval_count)
               return json.eval_count
            end
         end
      end,

      ---Output the data from the API ready for insertion into the chat buffer
      ---@param self CodeCompanion.Adapter
      ---@param data table The streamed JSON data from the API, also formatted by the format_data callback
      ---@return table|nil
      chat_output = function(self, data)
         local output = {}

         if data and data ~= "" then
            if not self.opts.stream then
               data = data.body
            end
            local data_mod = utils.clean_streamed_data(data)
            local ok, json = pcall(vim.json.decode, data_mod, { luanil = { object = true } })

            if ok and json.choices and #json.choices > 0 then
               local choice = json.choices[1]

               if choice.finish_reason then
                  local reason = choice.finish_reason
                  if reason ~= "stop" then
                     return {
                        status = "error",
                        output = "The stream was stopped due to: " .. reason,
                     }
                  end
               end

               local delta = (self.opts and self.opts.stream) and choice.delta or choice.message

               if delta then
                  if delta.role then
                     output.role = delta.role
                  else
                     output.role = "system"
                  end

                  -- Some providers may return empty content
                  if delta.content then
                     output.content = delta.content
                  else
                     output.content = ""
                  end

                  return {
                     status = "success",
                     output = output,
                  }
               end
            end
         end

         return nil
      end,

      ---Output the data from the API ready for inlining into the current buffer
      ---@param self CodeCompanion.Adapter
      ---@param data table The streamed JSON data from the API, also formatted by the format_data handler
      ---@param context table Useful context about the buffer to inline to
      ---@return table|nil
      inline_output = function(self, data, context)
         if data and data ~= "" then
            if not self.opts.stream then
               data = data.body
            end
            data = utils.clean_streamed_data(data)
            local ok, json = pcall(vim.json.decode, data, { luanil = { object = true } })

            if ok then
               --- Some third-party OpenAI forwarding services may have a return package with an empty json.choices.
               if not json.choices or #json.choices == 0 then
                  return
               end

               local choice = json.choices[1]
               local delta = (self.opts and self.opts.stream) and choice.delta or choice.message
               if delta.content then
                  return delta.content
               end
            end
         end
      end,

      ---Function to run when the request has completed. Useful to catch errors
      ---@param self CodeCompanion.Adapter
      ---@param data table
      ---@return nil
      on_exit = function(self, data)
         if data.status >= 400 then
            log:error("Error: %s", data.body)
         end
      end,
   },
   schema = {
      model = {
         order = 1,
         mapping = "parameters",
         type = "enum",
         desc = "ID of the model to use.",
         default = function(self)
            return get_models(self, { last = true })
         end,
         choices = function(self)
            return get_models(self)
         end,
      },
      temperature = {
         order = 2,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 0.8,
         desc = "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both.",
         validate = function(n)
            return n >= 0 and n <= 2, "Must be between 0 and 2"
         end,
      },
      num_ctx = {
         order = 3,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 2048,
         desc = "The maximum number of tokens that the language model can consider at once. This determines the size of the input context window, allowing the model to take into account longer text passages for generating responses. Adjusting this value can affect the model's performance and memory usage.",
         validate = function(n)
            return n > 0, "Must be a positive number"
         end,
      },
      mirostat = {
         order = 4,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 0,
         desc = "Enable Mirostat sampling for controlling perplexity. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0)",
         validate = function(n)
            return n == 0 or n == 1 or n == 2, "Must be 0, 1, or 2"
         end,
      },
      mirostat_eta = {
         order = 5,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 0.1,
         desc = "Influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. (Default: 0.1)",
         validate = function(n)
            return n > 0, "Must be a positive number"
         end,
      },
      mirostat_tau = {
         order = 6,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 5.0,
         desc = "Controls the balance between coherence and diversity of the output. A lower value will result in more focused and coherent text. (Default: 5.0)",
         validate = function(n)
            return n > 0, "Must be a positive number"
         end,
      },
      repeat_last_n = {
         order = 7,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 64,
         desc = "Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)",
         validate = function(n)
            return n >= -1, "Must be -1 or greater"
         end,
      },
      repeat_penalty = {
         order = 8,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 1.1,
         desc = "Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.1)",
         validate = function(n)
            return n >= 0, "Must be a non-negative number"
         end,
      },
      seed = {
         order = 9,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 0,
         desc = "Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. (Default: 0)",
         validate = function(n)
            return n >= 0, "Must be a non-negative number"
         end,
      },
      stop = {
         order = 10,
         mapping = "parameters.options",
         type = "string",
         optional = true,
         default = nil,
         desc = "Sets the stop sequences to use. When this pattern is encountered the LLM will stop generating text and return. Multiple stop patterns may be set by specifying multiple separate stop parameters in a modelfile.",
         validate = function(s)
            return s:len() > 0, "Cannot be an empty string"
         end,
      },
      tfs_z = {
         order = 11,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 1.0,
         desc = "Tail free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting. (default: 1)",
         validate = function(n)
            return n >= 0, "Must be a non-negative number"
         end,
      },
      num_predict = {
         order = 12,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = -1,
         desc = "Maximum number of tokens to predict when generating text. (Default: -1, -1 = infinite generation, -2 = fill context)",
         validate = function(n)
            return n >= -2, "Must be -2 or greater"
         end,
      },
      top_k = {
         order = 13,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 40,
         desc = "Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative. (Default: 40)",
         validate = function(n)
            return n >= 0, "Must be a non-negative number"
         end,
      },
      top_p = {
         order = 14,
         mapping = "parameters.options",
         type = "number",
         optional = true,
         default = 0.9,
         desc = "Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)",
         validate = function(n)
            return n >= 0 and n <= 1, "Must be between 0 and 1"
         end,
      },
   },
}

[eldios]

nice, will definitely test! I was also in the process of writing one by doing EXACTLY the same 😆

https://github.com/eldios/codecompanion.nvim

here's my shot at it in the main branch of my fork. Will definitely fork yours and add there then cause I was having a hard time making the basic functionality work.

Thanks @lll9p ! ❤️

[eldios]

confirmed it works! 🥳 🥳 🥳 what features are missing? 🤔 I'll use and test this on my side and keep you posted if I see any issue!
Thanks again @lll9p 🙌🏻

[lll9p]

@eldios The open-webui has many features, which can actually be integrated. For example: for some private reference documents or code libraries, they can be added to the Knowledge Base, and then enhance the model's coding ability through searching. I don't have much time to test it out, so you could try and see how to implement it. 😊

[eldios]

though it wouldn't make sense to implement them in a random order if nobody is using them. I'd probably start with the ones I/you/people use.
So I'm open to suggestions. The knowledge/tools one is an interesting one.

[lll9p]

@eldios
According API OF OPENWEBUI , I can add Collections and files in the open_webui adapters directly, there should be a better way to do it, for example mapping parameters in schema, and that get_collections and get_files functions should be implemented.

      form_parameters = function(self, params, messages)
         params.files = {
            { type = "collection", id = "collection-id-xxx" },
            {"type": "file", "id": "file-id-xxx"},
         }
         return params
      end,