Python: Bug: AzureAIInferencePromptExecutionSettings does not support response_format

[ankitbko]

Describe the bug
I am transitioning to AI Inference SDK and realized response_format is currently not supported in AzureAIInferencePromptExecutionSettings. Can we get this added please?

ine 57, in __init__
    settings.response_format = SelectedAgent
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ankit.sinha/projects/agentic-sandbox/fastapi-agents/.venv/lib/python3.12/site-packages/pydantic/main.py", line 922, in __setattr__
    self.__pydantic_validator__.validate_assignment(self, name, value)
pydantic_core._pydantic_core.ValidationError: 1 validation error for AzureAIInferenceChatPromptExecutionSettings
response_format

  Object has no attribute 'response_format' [type=no_such_attribute, input_value=<class 'app.agents.princi...ed_agent.SelectedAgent'>, input_type=ModelMetaclass]
 

[mathrb]

Hello
I want to share my experience as well.
I'm using semantic kernel with python, loading my functions from yaml files.
Made a quick test by adding response_format: Literal["text", "json_object"] = None here did the trick, the output was a valid json, and not surrounded with ```json

[moonbox3]

Hello I want to share my experience as well. I'm using semantic kernel with python, loading my functions from yaml files. Made a quick test by adding response_format: Literal["text", "json_object"] = None here did the trick, the output was a valid json, and not surrounded with ```json

Thanks for sharing, @mathrb. That method of configuring the response format is valid, yes, but also considered the "old" way. If one wants structured outputs based on a response format schema, that wouldn't work as well as providing the type to the response_format parameter. If doing it via yaml, you'd need to provide the entire, well-formatted JSON schema to represent the type (just a FYI).

I will work on getting the response_format parameter added to the AzureAIInferencePromptExecutionSettings. I apologize for the delay - I didn't realize this was assigned to me.

[mathrb]

Sure, I do agree @moonbox3 !
Thanks for the upcomming work :)

[moonbox3]

@ankitbko, I am trying to hit an Azure OpenAI model using the AzureAIInferenceChatCompletion AI connector and it's throwing when I try to pass it a JSON schema for structured outputs. As @mathrb showed above, they're able to send it the "old way" of producing JSON output, but it doesn't look like they support structured outputs. I'm trying to find some docs around this, but nothing is turning up.

[moonbox3]

To follow up: I don't see the Azure AI inference SDK supporting anything aside from what @mathrb pointed out -- "json_object" mode. You'd need to open an issue on them to see if/when they will support structured outputs. Otherwise, you'll need to move back to using the AzureChatCompletion AI service we offer, and hit an Azure OpenAI model where you can indeed configure structured outputs.

Their SDK docs: https://learn.microsoft.com/en-us/azure/machine-learning/reference-model-inference-chat-completions?view=azureml-api-2#chatcompletionresponseformat

[moonbox3]

I don't see any further action here. Closing.