Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging.#12335
Merged
moonbox3 merged 7 commits intoJun 3, 2025
Conversation
Contributor
alliscode
approved these changes
Jun 2, 2025
dmytrostruk
approved these changes
Jun 3, 2025
jcruzmot-te
pushed a commit
to thousandeyes/aia-semantic-kernel
that referenced
this pull request
Sep 15, 2025
…ixes/improvements for tool call content types, and logging. (microsoft#12335) ### Motivation and Context The AzureAIAgent streaming function call invocation is broken when handling more than one required function call from the service. There is nested logic to handle passing the tool calls back to the service because it requires one to pass in a handler. In this PR, we're now controlling everything from the main loop and are passing back the handler as the event stream for further processing. This does require some logic to determine if we have an initial async context manager (based on the first call to create the run stream) or if we're now using the handler from submitting tool calls. We can now see that when we have multiple functions involved, we are able to get the results as we expect: ``` Sample Output: # User: 'What is the price of the special drink and then special food item added together?' Function Call:> MenuPlugin-get_specials with arguments: {} Function Result:> Special Soup: Clam Chowder Special Salad: Cobb Salad Special Drink: Chai Tea for function: MenuPlugin-get_specials Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"} Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"} Function Result:> $9.99 for function: MenuPlugin-get_item_price Function Result:> $9.99 for function: MenuPlugin-get_item_price Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99} Function Result:> 19.98 for function: MathPlugin-Add # AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food item, Clam Chowder, is $9.99. Added together, the total price is $19.98. ``` <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Fix handling multiple function calls for the streaming AzureAIAgent. - Fixes a missing `quote` from the url citation for streaming annotation content - Shows the correct way to retrieve Bing Grounding tool calls in the two concept samples. - Adds handling for tool content content for OpenAPI plugins. - Adds logging for streaming agent invocation tool calls. - Relaxes the handling of the `FunctionResultContent` id attribute to be `str | None` similar to that of `FunctionCallContent`. The type hints for `id` have always been `str | None`, but the actual type hint on the attribute has been `str`. We don't always get a `FunctionResultContent` id from the Azure Agent Service during streaming invocations - message addition via the overide `__add__` method still work properly. - Move the AzureAIAgent pkg dependencies to be installed by default, instead of requiring the `azure` extra. - Closes microsoft#12312 - Closes microsoft#12328 - Closes microsoft#12331 - Closes microsoft#12324 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
The AzureAIAgent streaming function call invocation is broken when handling more than one required function call from the service. There is nested logic to handle passing the tool calls back to the service because it requires one to pass in a handler.
In this PR, we're now controlling everything from the main loop and are passing back the handler as the event stream for further processing. This does require some logic to determine if we have an initial async context manager (based on the first call to create the run stream) or if we're now using the handler from submitting tool calls. We can now see that when we have multiple functions involved, we are able to get the results as we expect:
Description
Fix handling multiple function calls for the streaming AzureAIAgent.
quotefrom the url citation for streaming annotation contentFunctionResultContentid attribute to bestr | Nonesimilar to that ofFunctionCallContent. The type hints foridhave always beenstr | None, but the actual type hint on the attribute has beenstr. We don't always get aFunctionResultContentid from the Azure Agent Service during streaming invocations - message addition via the overide__add__method still work properly.azureextra.StreamingAnnotationContent#12331Contribution Checklist