Skip to content

Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging.#12335

Merged
moonbox3 merged 7 commits into
microsoft:mainfrom
moonbox3:fix-azureai-agent-streaming-func-call
Jun 3, 2025
Merged

Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging.#12335
moonbox3 merged 7 commits into
microsoft:mainfrom
moonbox3:fix-azureai-agent-streaming-func-call

Conversation

@moonbox3

@moonbox3 moonbox3 commented Jun 2, 2025

Copy link
Copy Markdown
Collaborator

Motivation and Context

The AzureAIAgent streaming function call invocation is broken when handling more than one required function call from the service. There is nested logic to handle passing the tool calls back to the service because it requires one to pass in a handler.

In this PR, we're now controlling everything from the main loop and are passing back the handler as the event stream for further processing. This does require some logic to determine if we have an initial async context manager (based on the first call to create the run stream) or if we're now using the handler from submitting tool calls. We can now see that when we have multiple functions involved, we are able to get the results as we expect:

Sample Output:

# User: 'What is the price of the special drink and then special food item added together?'
Function Call:> MenuPlugin-get_specials with arguments: {}
Function Result:> 
        Special Soup: Clam Chowder
        Special Salad: Cobb Salad
        Special Drink: Chai Tea
        for function: MenuPlugin-get_specials
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"}
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"}
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99}
Function Result:> 19.98 for function: MathPlugin-Add
# AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food 
    item, Clam Chowder, is $9.99. Added together, the total price is $19.98.

Description

Fix handling multiple function calls for the streaming AzureAIAgent.

Contribution Checklist

@moonbox3 moonbox3 self-assigned this Jun 2, 2025
@moonbox3 moonbox3 requested a review from a team as a code owner June 2, 2025 04:57
@markwallace-microsoft markwallace-microsoft added the python Pull requests for the Python Semantic Kernel label Jun 2, 2025
@moonbox3 moonbox3 changed the title Python: Fix for AzureAIAgent streaming func call invocations Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging. Jun 2, 2025
@moonbox3 moonbox3 added the agents label Jun 2, 2025
@markwallace-microsoft

markwallace-microsoft commented Jun 2, 2025

Copy link
Copy Markdown
Contributor

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
agents/azure_ai
   agent_content_generation.py27110262%74–75, 77, 95, 97, 219, 222, 262, 287–288, 296, 304, 306, 314, 321, 329, 331, 339, 346, 352–354, 356, 358–360, 369–371, 380, 420, 459–460, 462–465, 467–469, 472–473, 475, 485–486, 488, 501–502, 504, 506–510, 519, 527, 540–541, 543, 545–552, 561, 570, 584–585, 587, 589–591, 593–597, 607–609, 618–619, 621, 647–654, 660–663, 668, 673–674, 681
   agent_thread_actions.py36614460%154, 156, 193–196, 288–289, 293, 296, 299–300, 304, 307, 310–311, 315, 318, 321–322, 326–327, 427, 429, 469–470, 492, 494–496, 498–500, 502–503, 505–510, 512–520, 524–525, 528–532, 535–536, 539–540, 543, 546, 549–553, 555–556, 560–561, 568–569, 574, 578–579, 581–582, 589, 591, 595, 597–598, 602–611, 613–615, 618, 668, 671, 702, 704, 711, 713–716, 718, 720, 722–723, 783, 788–793, 821–824, 828, 833–834, 838, 855–857, 861–862, 877–878, 916, 957–960, 962–963, 967, 971–972, 977
agents/open_ai
   assistant_thread_actions.py3385683%107, 110, 116, 195, 197, 236–242, 416, 474, 500, 599–600, 602–603, 609–610, 612–613, 615–618, 620–621, 623–624, 626–627, 641–645, 648–650, 697, 699–700, 704–708, 710–711, 725–726, 797, 802, 834
   responses_agent_thread_actions.py34110768%167, 184, 195, 206–207, 374–376, 390, 395, 402, 420–421, 425, 469–471, 474, 492–493, 502–504, 508, 512–513, 517–519, 530–532, 554, 556–557, 564–565, 567–568, 570, 572–573, 575–576, 659–661, 673–674, 676–680, 682–688, 690–699, 701–702, 704–707, 714–716, 721, 723, 784–787, 828–829, 833–835, 837, 862–863, 865–869, 871, 881–882, 890, 894, 901, 907, 971
connectors/ai/azure_ai_inference/services
   utils.py42197%53
contents
   function_result_content.py920100% 
core_plugins
   math_plugin.py200100% 
TOTAL27073462582% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
3616 18 💤 0 ❌ 0 🔥 1m 52s ⏱️

@moonbox3 moonbox3 added this pull request to the merge queue Jun 3, 2025
Merged via the queue into microsoft:main with commit fb0e418 Jun 3, 2025
28 checks passed
@moonbox3 moonbox3 deleted the fix-azureai-agent-streaming-func-call branch June 3, 2025 01:19
@github-project-automation github-project-automation Bot moved this to Sprint: Done in Semantic Kernel Jun 3, 2025
jcruzmot-te pushed a commit to thousandeyes/aia-semantic-kernel that referenced this pull request Sep 15, 2025
…ixes/improvements for tool call content types, and logging. (microsoft#12335)

### Motivation and Context

The AzureAIAgent streaming function call invocation is broken when
handling more than one required function call from the service. There is
nested logic to handle passing the tool calls back to the service
because it requires one to pass in a handler.

In this PR, we're now controlling everything from the main loop and are
passing back the handler as the event stream for further processing.
This does require some logic to determine if we have an initial async
context manager (based on the first call to create the run stream) or if
we're now using the handler from submitting tool calls. We can now see
that when we have multiple functions involved, we are able to get the
results as we expect:

```
Sample Output:

# User: 'What is the price of the special drink and then special food item added together?'
Function Call:> MenuPlugin-get_specials with arguments: {}
Function Result:> 
        Special Soup: Clam Chowder
        Special Salad: Cobb Salad
        Special Drink: Chai Tea
        for function: MenuPlugin-get_specials
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"}
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"}
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99}
Function Result:> 19.98 for function: MathPlugin-Add
# AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food 
    item, Clam Chowder, is $9.99. Added together, the total price is $19.98.
```

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Fix handling multiple function calls for the streaming AzureAIAgent.
- Fixes a missing `quote` from the url citation for streaming annotation
content
- Shows the correct way to retrieve Bing Grounding tool calls in the two
concept samples.
- Adds handling for tool content content for OpenAPI plugins.
- Adds logging for streaming agent invocation tool calls.
- Relaxes the handling of the `FunctionResultContent` id attribute to be
`str | None` similar to that of `FunctionCallContent`. The type hints
for `id` have always been `str | None`, but the actual type hint on the
attribute has been `str`. We don't always get a `FunctionResultContent`
id from the Azure Agent Service during streaming invocations - message
addition via the overide `__add__` method still work properly.
- Move the AzureAIAgent pkg dependencies to be installed by default,
instead of requiring the `azure` extra.
- Closes microsoft#12312
- Closes microsoft#12328
- Closes microsoft#12331
- Closes microsoft#12324

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents documentation python Pull requests for the Python Semantic Kernel

Projects

Archived in project

4 participants