[FEATURE REQ] Please improve the Realtime sample

[JohnGalt1717]

Describe the feature or improvement you are requesting

https://github.com/openai/openai-dotnet/blob/main/examples/Realtime/Example01_AudioFromFileWithToolsAsync.cs

Doesn't really show you how to use tools. It uses a hard coded response instead of the response that the AI actually returned for the tool. It should provide a sample for the real world that would use the AI's tool response output to create the RealtimeItem.

Additional context

No response

[jsquire]

Hi @JohnGalt1717. Thanks for reaching out and for your feedback. Looking over the sample, I agree that there are several areas where additional context would be helpful to better understand the end-to-end. In my opinion, that sample attempts to demonstrate too many concepts in a single sample, so we'll also evaluate potentially breaking that up to help with clarity.

AI's tool response output to create the RealtimeItem.

Forgive me if I misunderstand, but I believe there may be some confusion. In this scenario, the demonstration is the model invoking a local tool that the application registered for it.

• The sample registers a local function ("get_weather_for_location") with the model, telling it what parameters are needed to call it.

• The model makes a request to call the tool as part of the update and the local function is expected to provide the tool (function) output. The hardcoded text is meant to simulate the result of a local function call. The application owns the responsibility to produce that result (and send it to the model.) The model does not produce the result.

If you're interested in more context, this is discussed in the OpenAI docs in Using tools and Function calling

With respect to the sample, we'll take a look at cleaning things up a bit:

• The sample is busy because it tries to demonstrate too much in a single sample. As a result, it lacks focus.
• Commenting does not make the context and responsibilities clear. You have to already understand function calling and the flow.
• The local function call is difficult to follow because the text is hard-coded and totally ignores the parameters that the model sends. Extracting to a method to demonstrate would be clearer.

[JohnGalt1717]

This line:

 RealtimeItem functionOutputItem = RealtimeItem.CreateFunctionCallOutput(
                        callId: itemStreamingFinishedUpdate.FunctionCallId,
                        output: "70 degrees Fahrenheit and sunny");

It should be using the actual returned value from the tool call, instead it's just making it up and filling it in which doesn't show anyone how to actually get the result of the tool call and have the AI be able to use it.

To make this a better sample I'd suggest a tool that actually returns something (even if that itself is hard coded) and then the code processes that result and pushes it into the realtime system. (and explains what it's actually doing when it does that.)

[JohnGalt1717]

FYI with 2.2.0 since I don't have a sample to copy I did this:

        switch (update)
        {
            case OutputStreamingFinishedUpdate itemFinished:
                if (string.IsNullOrEmpty(itemFinished.FunctionCallId))
                    return false;

                Logger.LogInformation("Calling Tool: {Function}", itemFinished.FunctionName);

                try
                {
                    if (!string.IsNullOrEmpty(itemFinished.FunctionCallOutput))
                    {
                        var output = RealtimeItem.CreateFunctionCallOutput(
                            itemFinished.FunctionCallId,
                            itemFinished.FunctionCallOutput
                        );

                        Logger.LogInformation(
                            "Function Call ({FunctionName}) Result: {Result}",
                            itemFinished.FunctionName,
                            JsonSerializer.Serialize(output)
                        );
                        await session.AddItemAsync(output, cancellationToken);

                        // Ensure response continues after tool execution
                    }
                    else
                    {
                        Logger.LogWarning(
                            "Function Call ({FunctionName}) returned no output",
                            itemFinished.FunctionName
                        );
                        // Try to continue conversation even if tool returned no output
                    }

                    await StartNextMessageAsync(session, cancellationToken);
                }
                catch (Exception ex)
                {
                    Logger.LogError(
                        ex,
                        "Error executing function {FunctionName}: {ErrorMessage}",
                        itemFinished.FunctionName,
                        ex.Message
                    );
                    // Add error message to conversation and continue
                    await session.AddItemAsync(
                        RealtimeItem.CreateSystemMessage(
                            [
                                ConversationContentPart.CreateInputTextPart(
                                    $"Error executing function {itemFinished.FunctionName}: {ex.Message}. Please continue without this tool."
                                ),
                            ]
                        ),
                        cancellationToken
                    );

                    await StartNextMessageAsync(session, cancellationToken);
                }

itemFinished.FunctionCallOutput is null.

And the way this was done prior to 2.2.0 doesn't work because the method doesn't exist to create the function call output and there's no sample.

So code that was working doesn't work, and there is no indication because of this issue what I have to change to make it work (or if this is a bug as I would expect that FunctionCallOutput would be not null.

[jsquire]

It should be using the actual returned value from the tool call, instead it's just making it up and filling it in which doesn't show anyone how to actually get the result of the tool call and have the AI be able to use it.

@JohnGalt1717: There is no return from the tool call until the application provides it. In the sample, the "tool" registered is a local function, not one built into the model. The block that you highlighted is part of the request that the model makes of the application to execute the local function get_weather_for_location which we told it about. (this is discussed in the docs that I linked above for Function calling)

if (itemStreamingFinishedUpdate.FunctionCallId is not null)

This line is checking to see if the model responded asking to call a local function. It specifies which function in itemStreamingFinishedUpdate.FunctionName which should have been called out in the sample. Because that is the only local function registered, the sample just assumed it.

RealtimeItem functionOutputItem = RealtimeItem.CreateFunctionCallOutput(
    callId: itemStreamingFinishedUpdate.FunctionCallId,
    output: "70 degrees Fahrenheit and sunny");

 await session.AddItemAsync(functionOutputItem);

These lines simulate how you would inform the model of the results of a local function. In this case, the simulated get_weather_for_location always returns a hardcoded "70 degrees Fahrenheit and sunny" no matter the parameters the model tries to call it with. As mentioned, this is some important context that the sample does not demonstrate, which I believe is part of the confusion.

[JohnGalt1717]

Ok, but previous samples all over the internet in C# including Steve Sanderson's code had an extension method like this:

    public static async Task<ConversationItem?> GetFunctionCallOutputAsync(this ConversationItemStreamingFinishedUpdate update, IReadOnlyList<AIFunction> tools)
    {
        if (!string.IsNullOrEmpty(update.FunctionName) && tools.FirstOrDefault(t => t.Metadata.Name == update.FunctionName) is AIFunction aiFunction)
        {
            Dictionary<string, object?>? jsonArgs = null;
            try
            {
                jsonArgs = JsonSerializer.Deserialize<Dictionary<string, object?>>(update.FunctionCallArguments)!;
                var output = await aiFunction.InvokeAsync(jsonArgs);
                return ConversationItem.CreateFunctionCallOutput(update.FunctionCallId, output?.ToString() ?? "");
            }
            catch (JsonException)
            {
                return ConversationItem.CreateFunctionCallOutput(update.FunctionCallId, "Invalid JSON");
            }
            catch
            {
                return ConversationItem.CreateFunctionCallOutput(update.FunctionCallId, "Error calling tool");
            }
        }

        return null;
    }

I think I just figured out how to convert it to the new stuff:

    public static async Task<RealtimeItem?> GetFunctionCallOutputAsync(
        this OutputStreamingFinishedUpdate update,
        IReadOnlyList<AIFunction> tools,
        CancellationToken cancellationToken = default
    )
    {
        if (
            !string.IsNullOrEmpty(update.FunctionName)
            && tools.FirstOrDefault(t => t.Name == update.FunctionName) is AIFunction aiFunction
        )
        {
            Dictionary<string, object?>? jsonArgs = null;
            try
            {
                jsonArgs = JsonSerializer.Deserialize<Dictionary<string, object?>>(
                    update.FunctionCallArguments
                )!;
                var output = await aiFunction.InvokeAsync(
                    new AIFunctionArguments(jsonArgs),
                    cancellationToken
                );
                return RealtimeItem.CreateFunctionCallOutput(
                    update.FunctionCallId,
                    output?.ToString() ?? ""
                );
            }
            catch (JsonException)
            {
                return RealtimeItem.CreateFunctionCallOutput(update.FunctionCallId, "Invalid JSON");
            }
            catch
            {
                return RealtimeItem.CreateFunctionCallOutput(
                    update.FunctionCallId,
                    "Error calling tool"
                );
            }
        }

        return null;
    }

I'd argue that this very thing needs to be there in the library, and shouldn't require the tools to be passed, it should take the session and get the tools from the session.

This is why I said that you were doing magic. You're not actually calling the tool (AIFunction) and getting a result back based on the functions that the AI passed in, nor passing back the function's output back to the AI. You're just injecting a magic value back to the AI and hand waving the above away.

So:

This should be embedded in the library to enable this and the sample should use it with a note that if you really want you can do this by hand but here's this helper function that does it for you. Not having it means that everyone is going to have to write the same thing badly, to use tool calling against AIFunction.

I mean in the perfect world, this actually breaks out and is automatically handled so that tool calls are just done for you just as they are done with IChatClient with a Update raised for the tool call if you care to capture it and do something about it (before and after I guess)

[stephentoub]

We'd had a helper for this in the Microsoft.Extensions.AI.OpenAI library. We'd removed it as part of trimming back the surface area to just be for non-[Experimental] OpenAI things, but at this point we should probably bring it back.