Groq LiteLLM Integration does not work when using structured outputs

[amolpsingh]

Please read this first

• Have you read the docs? Agents SDK docs
• Have you searched for related issues? Others may have faced similar issues.

Describe the bug

When using LitellmModel with Groq models and specifying an output_type parameter, the SDK attempts to use OpenAI's structured outputs feature by setting tool_choice to "json_tool_call". Groq doesn't support this tool choice, resulting in a 400 Bad Request error 1 .

Debug information

• Agents SDK version: (e.g. v0.0.3) - 0.6.1
• Python version: (e.g. Python 3.10) - 3.11.8

Repro steps

import asyncio
from pydantic import BaseModel
from agents import Agent, Runner, function_tool
from agents.extensions.models.litellm_model import LitellmModel

class AgentResponse(BaseModel):
    message: str
    status: str

@function_tool
def dummy_tool(input_text: str) -> str:
    return f"Processed: {input_text}"

agent = Agent(
    name="Demo Agent",
    instructions="Use the dummy tool to process input and respond with a structured output.",
    tools=[dummy_tool],
    output_type=AgentResponse,  # This triggers the error
    model=LitellmModel(
        model="groq/openai/gpt-oss-120b",
        api_key="your-api-key"
    )
)

async def main():
    result = await Runner.run(
        starting_agent=agent,
        input="Hello, please process this message"
    )
    print(result.final_output)

asyncio.run(main())

Expected behavior

The agent should return a structured response matching the AgentResponse schema. Instead, it fails with:

litellm.exceptions.BadRequestError: litellm.BadRequestError: GroqException - {"error":{"message":"code=400, message=Requested tool_choice `json_tool_call` was not defined in the request, type=invalid_request_error","type":"invalid_request_error"}}

Root Cause Analysis

The issue occurs because:

1. When output_type is specified, the SDK creates a JSON schema from the Pydantic model 2
2. The SDK sets tool_choice to "json_tool_call" to enforce structured output 1
3. Groq's API doesn't support this specific tool choice value
4. The LiteLLM integration passes this parameter through without validation

Flaky Workaround

1. Remove output_type and parse JSON manually:

   agent = Agent(
       name="Demo Agent",
       instructions="Respond with JSON in the format: {'message': '...', 'status': '...'}",
       tools=[dummy_tool],
       # Remove output_type
       model=LitellmModel(...)
   )
   
   # Parse manually after getting response
   parsed = AgentResponse(**json.loads(result.final_output))

[ihower]

@amolpsingh I think this is primarily a LiteLLM + Groq integration bug:

Groq does support Structured Output via response_schema, but LiteLLM is currently handling this case incorrectly:

1. LiteLLM incorrectly detects that Groq does not support response_schema, and falls back to a workaround by faking a function call and setting tool_choice="json_tool_call", instead of using Groq’s native response_schema support.
2. Even worse, it does not inject a matching json_tool_call definition into tools, which directly causes Groq to return a 400 Bad Request error.

There is a very similar LiteLLM bug report here: BerriAI/litellm#11001

FYI, Groq also provides an OpenAI-compatible Responses API endpoint. I would recommend trying that directly instead of going through LiteLLM. You will likely run into fewer compatibility issues.

from agents import Agent, Runner, function_tool, set_default_openai_client, set_tracing_disabled
from openai import AsyncOpenAI

custom_client = AsyncOpenAI(
  base_url="https://api.groq.com/openai/v1",
  api_key="your-api-key"
)

set_default_openai_client(custom_client)
set_tracing_disabled(disabled=True)

agent = Agent(
    name="Demo Agent",
    instructions="...",
    model="openai/openai/gpt-oss-120b"
)

Unfortunately, Structured Output and Function Calling still cannot be used together on Groq. You will likely hit this error:

openai.BadRequestError: Error code: 400 - {
  'error': {
    'message': 'json mode cannot be combined with tool/function calling',
    'type': 'invalid_request_error',
    'param': 'response_format'
  }
}

Not every model and provider supports using these two features at the same time. I shared a practical workaround here:
#1778 (comment)

[Chesars]

Hi. @amolpsingh & @ihower. The issue here is that groq/openai/gpt-oss-120b supports native structured outputs, but LiteLLM was incorrectly using a workaround designed for models that don't support it.

How the workaround works:

For Groq models without native response_schema support (like llama-3.3-70b-versatile), LiteLLM implements structured outputs by converting requests:

Request:

 import litellm

  litellm.completion(
      model="groq/llama-3.3-70b-versatile",  # doesn't support native json_schema
      messages=[...],
      response_format={"type": "json_schema", ...}  # would fail in a direct call to Groq
  )

Gets transformed to:

 import litellm

  litellm.completion(
      model="groq/llama-3.3-70b-versatile",
      messages=[...],
      tools=[{"type": "function", "function": {"name": "json_tool_call", "parameters": {...}}}],
      tool_choice={"type": "function", "function": {"name": "json_tool_call"}}
  )

Here, the model groq/openai/gpt-oss-120b effectively supports structured outputs as Groq documentation. So LiteLLM does not need to implement the workaround.

In the user example, Tools + Structured output was being sent in the request, per Groq docs: "Streaming and tool use are not currently supported with Structured Outputs".

For the LiteLLM issue, tool_choice gets set to json_tool_call, but tools overwrite the special tool created for structured outputs. PR #17438 solves this.

Recommendation for openai-agents-python SDK

Groq doesn't support tools + structured outputs together. Use two separate agents:

from agents import Agent, Runner, function_tool
  from agents.extensions.models.litellm_model import LitellmModel
  from pydantic import BaseModel

  # Define structured output schema
  class AgentResponse(BaseModel):
      answer: str
      confidence: float

  # Define a tool
  @function_tool
  def dummy_tool(input: str) -> str:
      """Process input"""
      return f"Processed: {input}"

  # Agent 1: Tools only (no structured output)
  tools_agent = Agent(
      name="Tools Agent",
      instructions="Use tools to gather information.",
      tools=[dummy_tool],
      model=LitellmModel(model="groq/llama-3.3-70b-versatile")
  )

  # Agent 2: Structured output only (no tools)  
  formatter_agent = Agent(
      name="Formatter Agent",
      instructions="Format the information into a structured response.",
      output_type=AgentResponse,
      model=LitellmModel(model="groq/llama-3.3-70b-versatile")
  )

  # Chain them
  result = await Runner.run(tools_agent, query)
  formatted = await Runner.run(formatter_agent, result.final_output)

Note: Not all Groq models support structured outputs. See https://console.groq.com/docs/structured-outputs#supported-models.