Offline LLMs

[Hrishikesh007788]

@gventuri Is there a way to connect pandasai with LLM in our local machine completely offline? And is there is way to serve the local LLM in a server such that anyone can access it?

[philchass]

Same question, I can't find the way to do it

[richfly]

Are you trying with LM Studio?

[mspronesti]

@Hrishikesh007788 Just import the local LLM you want to use via langchain and pass it to a SmartDataframe. It will be internally converted by pandas-ai. Have a look at the docs for more (scroll down to the last section).

Example:

from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "gpt2" 
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=100)
llm = HuggingFacePipeline(pipeline=pipe)

df = SmartDataframe(..., config={"llm": llm})

Hope this helps.

[lordpba]

use ollama, should work (as I know under Linux and Mac actually)

example (after installed ollama on local machine):

from langchain_community.llms import Ollama
llm= Ollama(model='mistral', temperature=0.1)