List of Free Local LLM Models - Up and Running

[psymbio]

Is your feature request related to a problem? Please describe.
Can I get a list of free local LLM models I can run without the PRO subscription.

Describe the solution you'd like
A list of the args I can pass into HfApiModel() (most likely) so that I don't run into this error:

You have exceeded your monthly included credits for Inference Providers. Subscribe to PRO to get 20x more monthly
included credits.
[Step 17: Duration 0.30 seconds| Input tokens: 53,107 | Output tokens: 4,313]

Is this not possible with the current options.
I've tried DeepSeek model, which I thought would be free - but turns out even that is connected to the Together API. How can I get a local free LLM model up and running with smolagents?

Maybe this code sample can be provided in the README section.

[HKanoje]

@psymbio I faced the same problem when i was doing the hugging face AI agents course . So I figured out out this way:
You can use ollama + LiteLLM instead of HfApiModel() to Run model locally and and for free without any limits. Below Code is how you can load the model run. Check out models on ollama website. there are plenty of options. All for Free.

from smolagents import CodeAgent, LiteLLMModel

model = LiteLLMModel(
    model_id="ollama_chat/qwen2.5:7b", #Can try diffrent model here I am using qwen2.5 7B model
    api_base="http://127.0.0.1:11434",
    num_ctx=8192,
)

agent = CodeAgent(tools=[], model=model, add_base_tools=True)

agent.run("Could you give me the 118th number in the Fibonacci sequence?")

[aymeric-roucher]

The solution described by @HKanoje is good! If you think it's worth it, please open a PR ion the docs wherever you faced the problem (in agents course?) to add this as a note like "if you run into credit limits, you can switch to local inference using ...".

[HKanoje]

@aymeric-roucher Okay got it. Will do ASAP. Just a quick question should i make note everywhere HfApiModel() is used or just initial agent will do?

[HKanoje]

Hey @aymeric-roucher , just letting you know that I’ve opened a PR huggingface/agents-course#327 with the note on using Ollama + LiteLLM as a local alternative to HfApiModel.

[albertvillanova]

Thanks, @HKanoje, for the PR in agents-course.
Closing the issue here.