Fine-tuned model does not support tools (Base Model supports tools)

[mosi-tech]

Fine-tuned model doesn't support tools even when starting with tool supported model (meta-llama/Llama-3.2-1B-Instruct). Has anyone encounterd this? Below are the steps I followed.

ollama._types.ResponseError: registry.ollama.ai/library/finetuned_llama:latest does not support tools

#!/bin/bash

mlx_lm.fuse \
    --model meta-llama/Llama-3.2-1B-Instruct \
    --adapter-path adapters \
    --save-path models/fused_model \
    --export-gguf    

# Modelfile

FROM ./models/fused_model

ollama create finetuned_llama -f Modelfile

from ollama import ChatResponse, chat
import json
import functions

toolSpecs = []
toolSpecFilePath = "training/data/specs/specs_array.json"
with open(toolSpecFilePath, 'r') as file:
    toolSpecs = json.load(file)

messages = [
   {'role': 'user', 'content': "What's the weather in Paris"}
]

response: ChatResponse = chat(
  'finetuned_llama',
  messages=messages,
  tools=toolSpecs,
)

#Just print the response for now
if response.message.tool_calls:
  print(response.message.tool_calls)

[mosi-tech]

Just some observations.

WORKS
Running using mlx_lm.server and then requesting via curl works (accepts tools)

DOESN'T WORK

1. Running with mlx_lm.generate and passing message. I am not sure if generate supports tools.

2. Packaging the model to ollama;

[awni]

Can you share the request that works for mlx_lm.server and the request that doesn't work for mlx_lm.generate?

As for ollama, does the base model work with tool use? If so, maybe there is a configuration parameter that needs to be set when we export.

[mosi-tech]

I am able to get mlx_lm.generate to work using stringified prompt

What remains is Ollama packaging so I can use it in open-webui.

[awni]

As for ollama, does the base model work with tool use? If so, maybe there is a configuration parameter that needs to be set when we export.

?

[mosi-tech]

As for ollama, does the base model work with tool use? If so, maybe there is a configuration parameter that needs to be set when we export.

Yes. On running the base model (meta-llama/Llama-3.2-1B-Instruct) with mlx_m.server, it does work with tools.

[awni]

Does the base model work with ollama?

[mosi-tech]

That's good question. The answer is NO.

#!/bin/bash

mlx_lm.fuse \
    --model meta-llama/Llama-3.2-1B-Instruct \
    --save-path models/model_base \
    --export-gguf 

Modelfile

# Modelfile

FROM ./models/model_base

ollama create base_llama3.2 -f Modelfile

And still the same error. So, this is a ollama problem.
llama._types.ResponseError: registry.ollama.ai/library/base_llama3.2:latest does not support tools

[mosi-tech]

Copying base model's template and adding to TEMPLATE field to Modelfile worked.

FROM ./models/fused_model
TEMPLATE """
<base_model_template>
"""

[awni]

Great, I will close this then. Glad it is resolved.