Gemini tool use: Unable to submit request because schema specified other fields alongside any_of

[joselpart]

Note: This issue may be related to issues #1076 and #535. However, the fix introduced in PR #1143 for the latter issue doesn't seem to have addressed the issue I'm reporting here.

Environment

Python version: 3.12.3

langchain 0.3.27
langchain-core 0.3.76
langchain-google-genai 2.1.12
langchain-text-splitters 0.3.11
langgraph 0.6.7
langgraph-checkpoint 2.1.1
langgraph-prebuilt 0.6.4
langgraph-sdk 0.2.8
langsmith 0.4.29

Issue

I was recently experimenting with the material from the course "Deep Agents with LangGraph", but using Gemini 2.0 Flash instead of the model used in the course. I quickly hit a bump when trying to call a tool that uses a Union type hint:

langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 Unable to submit request because calculator functionDeclaration parameters.a schema specified other fields alongside any_of. When using any_of, it must be the only field set. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling

Code to reproduce the issue

import asyncio

from dotenv import load_dotenv
from typing import Literal, Union

from langchain_core.tools import tool
from langchain.chat_models import init_chat_model
from langgraph.prebuilt import create_react_agent
from langgraph.graph.state import CompiledStateGraph


@tool
def calculator(
    operation: Literal["add","subtract","multiply","divide"],
    a: Union[int, float],
    b: Union[int, float],
) -> Union[int, float]:
    """Define a two-input calculator tool.

    Arg:
        operation (str): The operation to perform ('add', 'subtract', 'multiply', 'divide').
        a (float or int): The first number.
        b (float or int): The second number.
        
    Returns:
        result (float or int): the result of the operation
    Example
        Divide: result   = a / b
        Subtract: result = a - b
    """
    if operation == 'divide' and b == 0:
        return {"error": "Division by zero is not allowed."}

    # Perform calculation
    if operation == 'add':
        result = a + b
    elif operation == 'subtract':
        result = a - b
    elif operation == 'multiply':
        result = a * b
    elif operation == 'divide':
        result = a / b
    else: 
        result = "unknown operation"
    return result


SYSTEM_PROMPT = (
    "You are a helpful arithmetic assistant who is an expert "
    "at using a calculator."
)


def create_agent() -> CompiledStateGraph:
    llm = init_chat_model("gemini-2.0-flash", model_provider="google_genai")
    tools = [calculator]

    return create_react_agent(
        llm,
        tools,
        prompt=SYSTEM_PROMPT,
    ).with_config({"recursion_limit": 20})


async def main():
    # Create agent
    agent = create_agent()

    # Chat loop
    while True:
        query = input("User: ")
        if query == "quit":
            break
        
        messages = [
            {
                "role": "user",
                "content": query,
            }
        ]
        response = await agent.ainvoke({"messages": messages})
        for message in response['messages']:
            print(message)
            print("=" * 50)


if __name__ == "__main__":
    # Load environment variables
    load_dotenv()

    # Run main program
    asyncio.run(main())

As hinted at the top, the issue seems to stem from how the schema is being constructed, resulting in an incompatible schema for the Gemini API whenever the Union type hint is used for an input argument. If I replace the Union type hint for the input tool arguments with just float, then it works:

$ uv run main.py 
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1758898610.978872   81131 alts_credentials.cc:93] ALTS creds ignored. Not running on GCP and untrusted ALTS is not enabled.
User: what is 1 + 1?
E0000 00:00:1758898616.423722   81131 alts_credentials.cc:93] ALTS creds ignored. Not running on GCP and untrusted ALTS is not enabled.
content='what is 1 + 1?' additional_kwargs={} response_metadata={} id='ea5b38ec-6cac-4e19-bcdd-dc67f056a055'
==================================================
content='' additional_kwargs={'function_call': {'name': 'calculator', 'arguments': '{"b": 1, "operation": "add", "a": 1}'}} response_metadata={'prompt_feedback': {'block_reason': 0, 'safety_ratings': []}, 'finish_reason': 'STOP', 'model_name': 'gemini-2.0-flash', 'safety_ratings': [], 'grounding_metadata': {}} id='run--5f6e28fb-3249-4c77-9549-e56e7ea6b601-0' tool_calls=[{'name': 'calculator', 'args': {'b': 1, 'operation': 'add', 'a': 1}, 'id': '4a74cbdb-fab0-443c-9d7a-7bb72dc38700', 'type': 'tool_call'}] usage_metadata={'input_tokens': 135, 'output_tokens': 7, 'total_tokens': 142, 'input_token_details': {'cache_read': 0}}
==================================================
content='2.0' name='calculator' id='55c3f762-fd80-4453-b9fd-135df8bd6bde' tool_call_id='4a74cbdb-fab0-443c-9d7a-7bb72dc38700'
==================================================
content='1 + 1 = 2' additional_kwargs={} response_metadata={'prompt_feedback': {'block_reason': 0, 'safety_ratings': []}, 'finish_reason': 'STOP', 'model_name': 'gemini-2.0-flash', 'safety_ratings': [], 'grounding_metadata': {}} id='run--7ae85c29-6af6-4050-b2b8-457951572baa-0' usage_metadata={'input_tokens': 145, 'output_tokens': 8, 'total_tokens': 153, 'input_token_details': {'cache_read': 0}}

[joselpart]

So, I was looking into the conversion applied to the schema through the _format_to_gapic_function_declaration function. I defined the following script to check the outputs:

import json
from typing import Literal, Union
from pydantic import Field, BaseModel
from langchain_core.utils.function_calling import convert_to_openai_function
from langchain_google_genai._function_utils import _format_to_gapic_function_declaration


class CalculatorSchema(BaseModel):
    operation: Literal["add", "subtract", "multiply", "divide"] = Field(
        ...,
        description="The operation to perform.",
    )
    a: Union[int, float] = Field(
        ...,
        description="The first number.",
    )
    b: float = Field(
        ...,
        description="The second number.",
    )


if __name__ == "__main__":
    openai_func = convert_to_openai_function(CalculatorSchema)
    print(json.dumps(openai_func, indent=4))
    print("-" * 100)
    genai_func = _format_to_gapic_function_declaration(openai_func)
    print(genai_func)

The output is as follows:

{
    "name": "CalculatorSchema",
    "description": "",
    "parameters": {
        "properties": {
            "operation": {
                "description": "The operation to perform.",
                "enum": [
                    "add",
                    "subtract",
                    "multiply",
                    "divide"
                ],
                "type": "string"
            },
            "a": {
                "anyOf": [
                    {
                        "type": "integer"
                    },
                    {
                        "type": "number"
                    }
                ],
                "description": "The first number."
            },
            "b": {
                "description": "The second number.",
                "type": "number"
            }
        },
        "required": [
            "operation",
            "a",
            "b"
        ],
        "type": "object"
    }
}
----------------------------------------------------------------------------------------------------
name: "CalculatorSchema"
parameters {
  type_: OBJECT
  properties {
    key: "operation"
    value {
      type_: STRING
      description: "The operation to perform."
      enum: "add"
      enum: "subtract"
      enum: "multiply"
      enum: "divide"
    }
  }
  properties {
    key: "b"
    value {
      type_: NUMBER
      description: "The second number."
    }
  }
  properties {
    key: "a"
    value {
      type_: NUMBER
      description: "The first number."
      any_of {
        type_: INTEGER
      }
      any_of {
        type_: NUMBER
      }
    }
  }
  required: "operation"
  required: "a"
  required: "b"
}

Here, the description of parameters.a is converted to:

properties {
    key: "a"
    value {
      type_: NUMBER
      description: "The first number."
      any_of {
        type_: INTEGER
      }
      any_of {
        type_: NUMBER
      }
    }
  }

This conversion doesn't look right. However, according to the docs, anyOf should be supported. These docs are referenced in the error message, and the higher-level google-genai library seems to be the one referenced by those docs. I saw there was an effort to migrate to this library (see PR #1077), but that doesn't seem to be in the roadmap anyway.

---

By the way, as I mentioned in the issue description, for this toy example, the solution is simple. We don't need to declare the arguments as a Union, since we can just treat the arguments as float and that's just fine. But I can imagine there may be other situations when this may not be possible (e.g., see related issues). This is why I brought up this issue.