driaforall/Dria-Agent-a-3B

Dria-Agent-α-3B

Introduction

Dria-Agent-α are series of large language models trained on top of the Qwen2.5-Coder series, specifically on top of the Qwen/Qwen2.5-Coder-3B-Instruct and Qwen/Qwen2.5-Coder-7B-Instruct models to be used in agentic applications. These models are the first instalment of agent-focused LLMs (hence the α in the naming) we hope to improve with better and more elaborate techniques in subsequent releases.

Dria-Agent-α employs Pythonic function calling, which is LLMs using blocks of Python code to interact with provided tools and output actions. This method was inspired by many previous work, including but not limited to DynaSaur, RLEF, ADAS and CAMEL. This way of function calling has a few advantages over traditional JSON-based function calling methods:

1. One-shot Parallel Multiple Function Calls: The model can can utilise many synchronous processes in one chat turn to arrive to a solution, which would require other function calling models multiple turns of conversation.
2. Free-form Reasoning and Actions: The model provides reasoning traces freely in natural language and the actions in between ```python ``` blocks, as it already tends to do without special prompting or tuning. This tries to mitigate the possible performance loss caused by imposing specific formats on LLM outputs discussed in Let Me Speak Freely?
3. On-the-fly Complex Solution Generation: The solution provided by the model is essentially a Python program with the exclusion of some "risky" builtins like exec, eval and compile (see full list in Quickstart below). This enables the model to implement custom complex logic with conditionals and synchronous pipelines (using the output of one function in the next function's arguments) which would not be possible with the current JSON-based function calling methods (as far as we know).

Quickstart

import json
from typing import Any, Dict, List
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "driaforall/Dria-Agent-a-3B"
model = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Please use our provided prompt for best performance
SYSTEM_PROMPT = """
You are an expert AI assistant that specializes in providing Python code to solve the task/problem at hand provided by the user.

You can use Python code freely, including the following available functions:

<|functions_schema|>
{{functions_schema}}
<|end_functions_schema|>

The following dangerous builtins are restricted for security:
- exec
- eval
- execfile
- compile
- importlib
- input
- exit

Think step by step and provide your reasoning, outside of the function calls.
You can write Python code and use the available functions. Provide all your python code in a SINGLE markdown code block like the following:

```python
result = example_function(arg1, "string")
result2 = example_function2(result, arg2)
```

DO NOT use print() statements AT ALL. Avoid mutating variables whenever possible.
""".strip()


get_sample_data = """
def check_availability(day: str, start_time: str, end_time: str) -> bool:
    \"\"\" 
    Check if a time slot is available on a given day.

    Args:
    - day: The day to check in YYYY-MM-DD format
    - start_time: Start time in HH:MM format
    - end_time: End time in HH:MM format

    Returns:
    - True if slot is available, False otherwise
    \"\"\"
    pass

def make_appointment(day: str, start_time: str, end_time: str) -> dict:
    \"\"\" 
    Make an appointment for a given time slot.

    Args:
    - day: The day to make appointment in YYYY-MM-DD format
    - start_time: Start time in HH:MM format
    - end_time: End time in HH:MM format
    - title: The title of the appointment

    Returns:
    - A dictionary with the appointment details and if it's made or not.
        dict keys:
            - day (str): The day the appointment is on, in YYYY-MM-DD format
            - start_time (str): Start time in HH:MM format
            - end_time (str): End time in HH:MM format
            - appointment_made (bool): Whether the appointment is successfully made or not. 
    \"\"\"
    pass

def add_to_reminders(reminder_text: str) -> bool:
    \"\"\" 
    Add a text to reminders.

    Args: 
    - reminder_text: The text to add to reminders

    Returns:
    - Whether the reminder was successfully created or not.
    \"\"\"
    pass
"""

# Helper function to create the system prompt for our model
def format_prompt(tools: str):
    return SYSTEM_PROMPT.format(functions_schema=tools)

system_prompt = SYSTEM_PROMPT.replace("{{functions_schema}}", get_sample_data)

USER_QUERY = """
Can you check if I have tomorrow 10:00-12:00 available and make an appointment for a meeting 
with my thesis supervisor if so? If you made the appointment, please add it to my reminders.
"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": USER_QUERY},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=2048
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

The output should be something like:

I'll help you check availability and make the appointment. Let's break this down into steps:

1. First, we need to check if tomorrow's 10:00-12:00 time slot is available using check_availability()
2. If available, we'll make the appointment using make_appointment()
3. Finally, we'll add the appointment to reminders if it was successfully made

Here's the code to accomplish this:

```python
from datetime import datetime, timedelta

# Calculate tomorrow's date
tomorrow = (datetime.now() + timedelta(days=1)).strftime("%Y-%m-%d")

# Check availability for 10:00-12:00
available = check_availability(tomorrow, "10:00", "12:00")

# If available, make the appointment
appointment_details = make_appointment(
    day=tomorrow,
    start_time="10:00",
    end_time="12:00",
    title="Meeting with thesis supervisor"
)

# Add appointment to reminders if it was successfully made
if appointment_details["appointment_made"]:
    reminder_text = f"Meeting with thesis supervisor on {tomorrow} from 10:00-12:00"
    add_to_reminders(reminder_text)
```

This code will:
1. Calculate tomorrow's date in YYYY-MM-DD format
2. Check if the 10:00-12:00 slot is available
3. If available, make the appointment with the specified details
4. If the appointment is successfully made, add a reminder to the system

The code handles all error cases implicitly through the boolean returns of the functions. If any step fails, the subsequent steps won't execute, preventing partial or invalid appointments.

Evaluation & Performance

We evaluate the model on the following benchmarks:

1. Berkeley Function Calling Leaderboard (BFCL)
2. MMLU-Pro
3. Dria-Pythonic-Agent-Benchmark (DPAB): The benchmark we curated with a synthetic data generation +model-based validation + filtering and manual selection to evaluate LLMs on their Pythonic function calling ability, spanning multiple scenarios and tasks. More detailed information about the benchmark and the Github repo will be released soon.

Below are the BFCL results: evaluation results for Qwen2.5-Coder-3B-Instruct, Dria-Agent-α-3B and gpt-4o-2024-11-20

Metric	Qwen/Qwen2.5-3B-Instruct	Dria-Agent-a-3B	gpt-4o-2024-11-20 (Prompt)
Non-Live Simple AST	75.50%	75.08%	79.42%
Non-Live Multiple AST	90.00%	93.00%	95.50%
Non-Live Parallel AST	80.00%	85.00%	94.00%
Non-Live Parallel Multiple AST	78.50%	79.00%	83.50%
Non-Live Simple Exec	82.07%	87.57%	100.00%
Non-Live Multiple Exec	86.00%	85.14%	94.00%
Non-Live Parallel Exec	82.00%	90.00%	86.00%
Non-Live Parallel Multiple Exec	80.00%	88.00%	77.50%
Live Simple AST	68.22%	70.16%	83.72%
Live Multiple AST	66.00%	67.14%	79.77%
Live Parallel AST	62.50%	50.00%	87.50%
Live Parallel Multiple AST	66.67%	70.83%	70.83%
Relevance Detection	88.89%	100.00%	83.33%

and the MMLU-Pro and DPAB results:

Benchmark Name	Qwen2.5-Coder-3B-Instruct	Dria-Agent-α-3B
MMLU-Pro	35.2 (Self Reported)	29.8*
DPAB (Pythonic, Strict)	24	51

*Note: The model tends to use Pythonic function calling for a lot of the test cases in STEM-related fields (math, physics, chemistry, etc.) in the MMLU-Pro benchmark, which isn't captured by the evaluation framework and scripts provided in their Github repository. We haven't modified the script for evaluation, and leave it for the future iterations of this model. However, by performing qualitative analysis on the model responses, we suspect that the model's score will increase instead of suffering a ~6% decrease.