Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Complex datatypes like List, Dict or Pydantic Objects are not understood by Phidata while registering tools with Gemini. Works for OpenAI. #1759

Open
gauravdhiman opened this issue Jan 11, 2025 · 4 comments
Open

Complex datatypes like List, Dict or Pydantic Objects are not understood by Phidata while registering tools with Gemini. Works for OpenAI. #1759

gauravdhiman opened this issue Jan 11, 2025 · 4 comments

Comments

@gauravdhiman
Copy link
Contributor

I have a tool with below interface:

from phi.tools import Toolkit
from pydantic import BaseModel, Field

class Sentence(BaseModel):
    voice: str = Field(..., description="Voice name", examples=["af_sarah", "am_michael"])
    text: str

class Sentences(BaseModel):
    sentences: List[Sentence]

class TTSToolkit(Toolkit):
    ....
    ....
    def text_to_speech(self, sentences: Sentences, output_filename: str = "output.wav") -> str:
        """
        Converts a list of text sentences to a single audio file with given filename.

        Args:
            sentences (List[Dict[str, str]]): Stringified list of dictionaries, where each dictionary contains
                "voice" (str) and "text" (str) keys. voice key can have one of these values: af_sarah, am_michael alternatively.
            output_path (str, optional): The path to save the output audio file. Defaults to "output.wav".
        """

In agent, if I use Gemini model like this, it fails but if I use OpenAI model, it works. No other change, just change in model.

I am using this Gemini provider class:

from phi.model.google import Gemini

Gemini model fails while determining tool definition with below err:

    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[sentences].properties: should be non-empty for OBJECT type

For debugging, I printed this:

print(f"Adding function {name} from {tool.name} to model. Function: {func}")

I printed this within this if condition, and this is what it prints:

Adding function text_to_speech from TTSToolkit to model. Function: name='text_to_speech' description=None parameters={'type': 'object', 'properties': {}, 'required': []} strict=None entrypoint=<bound method TTSToolkit.text_to_speech of <TTSToolkit name=TTSToolkit functions=['text_to_speech']>> sanitize_arguments=True show_result=False stop_after_tool_call=False pre_hook=None post_hook=None
@gauravdhiman gauravdhiman changed the title Complex datatypes like List, Dict are not understood by Phidata when while registering tools with Gemini. Works for OpenAI. Complex datatypes like List, Dict or Pydantic Objects are not understood by Phidata when while registering tools with Gemini. Works for OpenAI. Jan 11, 2025
@gauravdhiman gauravdhiman changed the title Complex datatypes like List, Dict or Pydantic Objects are not understood by Phidata when while registering tools with Gemini. Works for OpenAI. Complex datatypes like List, Dict or Pydantic Objects are not understood by Phidata while registering tools with Gemini. Works for OpenAI. Jan 11, 2025
@gauravdhiman
Copy link
Contributor Author

gauravdhiman commented Jan 13, 2025 •
edited
Loading

If I make the sentences param as List[Dict[str, str]], in that case I get this err with Gemini:

Traceback (most recent call last):
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Protocol message Schema has no "propertyNames" field.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/podcast_agent.py", line 38, in <module>
    response = agent.run(
               ^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/phi/agent/agent.py", line 2070, in run
    return next(resp)
           ^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/phi/agent/agent.py", line 1765, in _run
    self.update_model()
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/phi/agent/agent.py", line 522, in update_model
    self.model.add_tool(tool=tool, agent=self)
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/phi/model/google/gemini.py", line 400, in add_tool
    function_declaration = self._build_function_declaration(func)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/phi/model/google/gemini.py", line 356, in _build_function_declaration
    return FunctionDeclaration(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/google/generativeai/types/content_types.py", line 559, in __init__
    self._proto = protos.FunctionDeclaration(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/message.py", line 728, in __init__
    pb_value = marshal.to_proto(pb_type, value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/marshal.py", line 235, in to_proto
    pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 45, in to_proto
    return self._wrapper(value)._pb
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/message.py", line 728, in __init__
    pb_value = marshal.to_proto(pb_type, value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/marshal.py", line 233, in to_proto
    return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/marshal.py", line 235, in to_proto
    pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 45, in to_proto
    return self._wrapper(value)._pb
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/message.py", line 728, in __init__
    pb_value = marshal.to_proto(pb_type, value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/marshal.py", line 235, in to_proto
    pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 45, in to_proto
    return self._wrapper(value)._pb
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/playgrounds/kokoro-onnx-tts/.venv-kokoro-onnx-tts/lib/python3.12/site-packages/proto/message.py", line 724, in __init__
    raise ValueError(
ValueError: Unknown field for Schema: propertyNames

I feel the issue is in this function which is supposed to covert the params to Gemini compatible params. Its not doing that properly for all cases. Its not going recursively down, like params could be List of List of Dict of str to number. This method just look at top level, means it will work for simple cases like List[str] or List[int] et, not for complex data types. I think it should recursive approach.

@ysolanky
Copy link
Contributor

Hey @gauravdhiman, thank you for bringing this up! I’m currently working on a fix. Could you please share the debug logs when using your Custom Tool with OpenAI? I’d like to review the tool calls made by the model.

@gauravdhiman
Copy link
Contributor Author

Here is the complete run log, when the tool signature is this def text_to_speech(self, script: List[Dict[str, str]], output_path: str, output_filename: Optional[str] = "output.wav") -> str::

❯ python -m src.backend.api.ai.agents.podcast_agent.py                                           
Setting espeak library to /Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/espeakng_loader/libespeak-ng.dylib
DEBUG    Debug logs enabled                                                                                                                          
DEBUG    *********** Agent Run Start: 66160711-dcb9-4d27-93a3-dab136382fe6 ***********                                                               
Traceback (most recent call last):
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Parameter to CopyFrom() must be instance of same class: expected <class 'Schema'> got <class 'dict'>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Protocol message Schema has no "propertyNames" field.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Protocol message Schema has no "propertyNames" field.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/src/backend/api/ai/agents/podcast_agent.py", line 47, in <module>
    response = agent.run(
               ^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/phi/agent/agent.py", line 2070, in run
    return next(resp)
           ^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/phi/agent/agent.py", line 1765, in _run
    self.update_model()
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/phi/agent/agent.py", line 522, in update_model
    self.model.add_tool(tool=tool, agent=self)
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/phi/model/google/gemini.py", line 395, in add_tool
    function_declaration = self._build_function_declaration(func)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/phi/model/google/gemini.py", line 352, in _build_function_declaration
    return FunctionDeclaration(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/google/generativeai/types/content_types.py", line 558, in __init__
    self._proto = protos.FunctionDeclaration(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/message.py", line 728, in __init__
    pb_value = marshal.to_proto(pb_type, value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/marshal.py", line 235, in to_proto
    pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 45, in to_proto
    return self._wrapper(value)._pb
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/message.py", line 728, in __init__
    pb_value = marshal.to_proto(pb_type, value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/marshal.py", line 233, in to_proto
    return {k: self.to_proto(recursive_type, v) for k, v in value.items()}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/marshal.py", line 235, in to_proto
    pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 45, in to_proto
    return self._wrapper(value)._pb
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/message.py", line 728, in __init__
    pb_value = marshal.to_proto(pb_type, value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/marshal.py", line 235, in to_proto
    pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/marshal/rules/message.py", line 45, in to_proto
    return self._wrapper(value)._pb
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/gauravdhiman/projects/python/ai/phidata-projects/social-media-agents/.venv-social-media-agents/lib/python3.12/site-packages/proto/message.py", line 724, in __init__
    raise ValueError(
ValueError: Unknown field for Schema: propertyNames

@ysolanky
Copy link
Contributor

@gauravdhiman can you please share the debug logs for the Agent with a pydantic class in the tool definition running with OpenAIChat?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gauravdhiman @ysolanky