Create Response with file as a base64 data url

[jomhyre]

Confirm this is a feature request for the .NET library and not the underlying OpenAI API

• This is a feature request for the .NET library

Describe the feature or improvement you are requesting

Sorry if this is just missing because I'm using the 2.2.0-beta.4 SDK version. I assume it's the implementation isn't fully complete

I want to be able to provide a file to the responses API as described in the API documentation here

import fs from "fs";
import OpenAI from "openai";
const client = new OpenAI();

const data = fs.readFileSync("draconomicon.pdf");
const base64String = data.toString("base64");

const response = await client.responses.create({
    model: "gpt-4o",
    input: [
        {
            role: "user",
            content: [
                {
                    type: "input_file",
                    filename: "draconomicon.pdf",
                    file_data: `data:application/pdf;base64,${base64String}`,
                },
                {
                    type: "input_text",
                    text: "What is the first dragon in the book?",
                },
            ],
        },
    ],
});

console.log(response.output_text);

There doesn't seem to be an example of this feature for the .NET library, and I also can't find a matching interface.

This is the minimal version of what I'm trying to do:

    private static async Task<ClientResult<OpenAIResponse?>> PassFileToResponseApi(
        OpenAIClient openAiClient,
        Uri url
    )
    {
        var responsesClient = openAiClient.GetOpenAIResponseClient("gpt-4o");

        using var httpClient = new HttpClient();

        var fileResponse = await httpClient.GetAsync(url);

        var fileData = await BinaryData.FromStreamAsync(
            await fileResponse.Content.ReadAsStreamAsync()
        );

        ResponseItem[] responseInputs =
        [
            ResponseItem.CreateUserMessageItem(
                [ResponseContentPart.CreateInputFilePart("fileId", "fileName.pdf", fileData)]
            ),
        ];

        ResponseCreationOptions options = new()
        {
            Temperature = 2.8f,
            Instructions = "Tell me something about the file",
        };

        var clientResult = await responsesClient.CreateResponseAsync(responseInputs, options);

        return clientResult;
    }

but the call to CreateResponseAsync fails:

System.Text.Json.JsonReaderException: '%' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
   at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.Utf8JsonWriter.WriteRawValueCore(ReadOnlySpan`1 utf8Json, Boolean skipInputValidation)
   at System.Text.Json.Utf8JsonWriter.WriteRawValue(ReadOnlySpan`1 utf8Json, Boolean skipInputValidation)
   at OpenAI.Responses.InternalResponsesInputFileContentPart.JsonModelWriteCore(Utf8JsonWriter writer, ModelReaderWriterOptions options)
   at OpenAI.Responses.InternalResponsesInputFileContentPart.System.ClientModel.Primitives.IJsonModel<OpenAI.Responses.InternalResponsesInputFileContentPart>.Write(Utf8JsonWriter writer, ModelReaderWriterOptions options)
   at OpenAI.ModelSerializationExtensions.WriteObjectValue[T](Utf8JsonWriter writer, T value, ModelReaderWriterOptions options)
   at OpenAI.Responses.InternalResponsesSystemMessage.JsonModelWriteCore(Utf8JsonWriter writer, ModelReaderWriterOptions options)
   at OpenAI.Responses.InternalResponsesSystemMessage.System.ClientModel.Primitives.IJsonModel<OpenAI.Responses.InternalResponsesSystemMessage>.Write(Utf8JsonWriter writer, ModelReaderWriterOptions options)
   at OpenAI.ModelSerializationExtensions.WriteObjectValue[T](Utf8JsonWriter writer, T value, ModelReaderWriterOptions options)
   at OpenAI.Responses.ResponseCreationOptions.JsonModelWriteCore(Utf8JsonWriter writer, ModelReaderWriterOptions options)
   at OpenAI.Responses.ResponseCreationOptions.System.ClientModel.Primitives.IJsonModel<OpenAI.Responses.ResponseCreationOptions>.Write(Utf8JsonWriter writer, ModelReaderWriterOptions options)
   at System.ClientModel.Internal.ModelWriter`1.ExtractReader()
   at System.ClientModel.BinaryContent.ModelBinaryContent`1.get_SequenceReader()
   at System.ClientModel.BinaryContent.ModelBinaryContent`1.TryComputeLength(Int64& length)
   at System.ClientModel.Primitives.HttpClientPipelineTransport.HttpPipelineRequest.MessageBodyAdapter.TryComputeLength(Int64& length)
   at System.Net.Http.HttpContent.GetComputedOrBufferLength()
   at System.Net.Http.Headers.HttpContentHeaders.get_ContentLength()
   at System.Net.Http.SocketsHttpHandler.ValidateAndNormalizeRequest(HttpRequestMessage request)
   at System.Net.Http.SocketsHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.ClientModel.Primitives.HttpClientPipelineTransport.ProcessSyncOrAsync(PipelineMessage message, Boolean async)
   at System.ClientModel.Primitives.HttpClientPipelineTransport.ProcessCoreAsync(PipelineMessage message)
   at System.ClientModel.Primitives.PipelineTransport.ProcessSyncOrAsync(PipelineMessage message, Boolean async)
   at System.ClientModel.Primitives.PipelineTransport.ProcessAsync(PipelineMessage message)
   at System.ClientModel.Primitives.PipelineTransport.ProcessAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex)
   at System.ClientModel.Primitives.PipelinePolicy.ProcessNextAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex)
   at System.ClientModel.Primitives.ApiKeyAuthenticationPolicy.ProcessAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex)
   at System.ClientModel.Primitives.PipelinePolicy.ProcessNextAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex)
   at System.ClientModel.Primitives.ClientRetryPolicy.ProcessSyncOrAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex, Boolean async)
   at System.ClientModel.Primitives.ClientRetryPolicy.ProcessSyncOrAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex, Boolean async)
   at System.ClientModel.Primitives.ClientRetryPolicy.ProcessAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex)
   at OpenAI.GenericActionPipelinePolicy.ProcessAsync(PipelineMessage message, IReadOnlyList`1 pipeline, Int32 currentIndex)
   at System.ClientModel.Primitives.ClientPipeline.SendAsync(PipelineMessage message)
   at OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at OpenAI.Responses.OpenAIResponseClient.CreateResponseAsync(BinaryContent content, RequestOptions options)
   at OpenAI.Responses.OpenAIResponseClient.CreateResponseAsync(IEnumerable`1 inputItems, ResponseCreationOptions options, CancellationToken cancellationToken)

so it seems it's trying to parse the fileBytes as JSON when generating the API request (the file is a PDF with the first byte being '%'). I'm unsure what JSON it is expecting. I've also tried encoding the data url myself and passing that in but that obviously doesn't work. Is it just a bug?

Additional context

No response

[trrwilson]

Thanks for the report, @jomhyre! You're correct in that the logic isn't properly emplacing the base64-encoded data in a data URI -- that will block the in-operation upload and we'll get that addressed ASAP.

In the interim, you can work around this by uploading the file separately and referring to it by ID, e.g.:

OpenAIFile newFileToUse = await fileClient.UploadFileAsync(
    BinaryData.FromBytes(File.ReadAllBytes(filePath)),
    "test_favorite_foods.pdf",
    FileUploadPurpose.UserData);

ResponseItem messageItem = ResponseItem.CreateUserMessageItem(
    [
        ResponseContentPart.CreateInputTextPart("Based on this file, what food should I order for whom?"),
        ResponseContentPart.CreateInputFilePart(fileId: newFileToUse.Id, filename: null, null),
    ]);

OpenAIResponse response = await client.CreateResponseAsync([messageItem]);

For the appropriate fix, we'll likely create dedicated overloads of CreateInputFilePart() for file ID vs. binary+filename usage, much like ChatMessageContentPart.CreateFilePart() now does.

[javeke]

Relieved to know I'm not alone on this one😆

Nevertheless, I truly appreciate this package. Amazing work @trrwilson !

[jamesgurung]

The workaround is useful. However, in some situations it's much easier to include a file as base64 to avoid needing to manage/delete it afterwards. It would be great if it's possible to implement a fix.