How should I handle when the response from MCP or some tool is JSON, which contains the base64 encoding of the file.

[cthulhu-tww]

Question

For example, my MCP is a StreamableHttp, and the tool is text2image, which returns an image containing base64 in JSON. When I called the tool, I got an error: An error occurred while running the tool. Please try again. Error: Error code: 400 - {'error': {'code': 'invalid_parameter_error', 'param': None, 'message': '<400> InternalError.Algo.InvalidParameter: Range of input length should be [1, 129024]', 'type': 'invalid_request_error'}, 'id': 'chatcmpl-049fe5e6-3977-929d-b1ad-1b069f5980b5', 'request_id': '049fe5e6-3977-929d-b1ad-1b069f5980b5'}， After my investigation, I found that the error occurred in the _get_new_response() function. How should I preprocess my JSON before passing it as a parameter to this function, such as saving the file locally and modifying the JSON to a regular short string.

[Gen-AI-Developer]

To resolve the invalid_parameter_error in the OpenAI Agents SDK when using a text2image tool returning base64-encoded image data in JSON, follow these steps to preprocess the JSON before passing it to _get_new_response():

1. Extract Base64 Data: Parse the JSON output from the text2image tool to extract the base64-encoded image string.

2. Save Image Locally: Decode the base64 string and save it as an image file (e.g., PNG or JPEG) to a local directory.

3. Replace Base64 with File Path: Modify the JSON to replace the base64 string with the file path or a short string reference (e.g., a URL or file identifier).

4. Pass Modified JSON: Use the updated JSON with the short string reference in _get_new_response() to avoid exceeding the input length limit (1 to 129,024 characters).

Example Code

import base64
import json
import os

def preprocess_text2image_response(json_response, output_dir="images"):
    # Ensure output directory exists
    os.makedirs(output_dir, exist_ok=True)
    
    # Parse JSON response
    data = json.loads(json_response) if isinstance(json_response, str) else json_response
    
    # Extract base64 image data (adjust key based on your JSON structure)
    base64_string = data.get("image_base64")  # Replace with actual key name
    
    if base64_string:
        # Decode base64 to bytes
        image_data = base64.b64decode(base64_string)
        
        # Save image to file
        image_path = os.path.join(output_dir, "output_image.png")
        with open(image_path, "wb") as f:
            f.write(image_data)
        
        # Replace base64 with file path in JSON
        data["image_base64"] = image_path  # Or use a short string identifier
    
    # Return modified JSON as string
    return json.dumps(data)

# Example usage
json_response = '{"image_base64": "BASE64_STRING_HERE"}'  # Replace with actual response
processed_json = preprocess_text2image_response(json_response)

# Pass processed_json to _get_new_response()
# agent._get_new_response(processed_json)

Key Points

• Error Cause: The base64 string likely exceeds the input length limit (129,024 characters) for _get_new_response().
• Solution: Saving the image locally reduces the JSON size, ensuring compliance with the limit.
• File Path: Use a local file path or a short identifier in the JSON instead of the base64 string.
• SDK Reference: Per the OpenAI Agents SDK (https://openai.github.io/openai-agents-python/), ensure the modified JSON adheres to expected formats for tool outputs.

If the issue persists, verify the JSON structure matches the tool’s expected schema and check for additional parameter constraints in the SDK documentation.

[cthulhu-tww]

@Gen-AI-Developer
Firstly, thank you for your reply. My current issue is not how to save my base64 file locally. But how to intercept this JSON when the MCP tool returns an excessively long base64 in Runner.run? I have tried the on_tool_end hook, but the hook cannot adjust the result. I need a function similar to this hook, but it can modify the returned result as soon as the tool is called, and then proceed to the next round of _run_single_turn

[seratch]

This SDK does not provide a way to handle image data from an MCP server, so I’d suggest either modifying the MCP server to return a URL where the image is saved instead of returning the image data (if you own the MCP server), or placing a proxy between the MCP and your agents to adjust the tool responses.

Even if this SDK had the ability to handle image data, I don't recommend passing the large base64 encoded data to Responses API because it is cost-inefficient plus extremely slow.

Please refer to openai/openai-agents-js#91 for more details.

Hope this helps.

[github-actions]

This issue is stale because it has been open for 7 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 3 days since being marked as stale.