[BUG]: LLMObs annotate fails for openai vision payload

[harshilprajapati96]

Tracer Version(s)

2.20.0

Python Version(s)

Python 3.11.7

Pip Version(s)

pip 24.2

Bug Report

When attempting to initialize the Messages class from the ddtrace.llmobs.utils module with a message containing non-string content types (e.g., images) intended for vision models, a TypeError is raised. The error message indicates that the message content must be a string, suggesting that the Messages class does not support complex content structures within the content field, particularly those required for vision model integrations as outlined in OpenAI's Vision Models Guide .

Reproduction Code

from ddtrace.llmobs.utils import Messages

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
                },
            },
        ],
    }
]

# Attempt to initialize Messages with the above data
Messages(messages)

Additional Code Context:
The error originally occurred when executing the following code block:

with LLMObs.llm(model_name=model, model_provider="openai") as llm_span:
    response = self.client.beta.chat.completions.parse(
        model=model,
        messages=messages,
        **parameters,
    )
    LLMObs.annotate(
        span=llm_span,
        input_data=messages,
        output_data=[
            {"role": choice.message.role, "content": choice.message.content} for choice in response.choices
        ],
    )

Error Logs

2025-02-07T15:16:39.188Z Failed to parse input messages.
2025-02-07T15:16:39.188Z Traceback (most recent call last):
  File "/var/task/ddtrace/llmobs/_llmobs.py", line 823, in _tag_llm_io
    input_messages = Messages(input_messages)
  File "/var/task/ddtrace/llmobs/utils.py", line 52, in __init__
    raise TypeError("Message content must be a string.")
TypeError: Message content must be a string.

Libraries in Use

aiohappyeyeballs==2.4.4
aiohttp==3.11.11
aiosignal==1.3.2
annotated-types==0.7.0
anyio==4.8.0
asttokens==3.0.0
attrs==25.1.0
aws-encryption-sdk==4.0.0
aws-xray-sdk==2.14.0
aws_lambda_powertools==3.5.0
black==25.1.0
boto3==1.36.9
botocore==1.36.9
bytecode==0.16.1
cachetools==5.5.1
certifi==2024.12.14
cffi==1.17.1
charset-normalizer==3.4.1
click==8.1.8
cloudpickle==3.1.1
coverage==7.6.10
cryptography==44.0.0
databricks-sdk==0.42.0
ddtrace==2.20.0
decorator==5.1.1
Deprecated==1.2.18
distro==1.9.0
envier==0.6.1
executing==2.2.0
fastjsonschema==2.21.1
frozenlist==1.5.0
gitdb==4.0.12
GitPython==3.1.44
google-auth==2.38.0
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
httpx_auth==0.23.1
idna==3.10
importlib_metadata==8.6.1
iniconfig==2.0.0
ipdb==0.13.13
ipython==8.31.0
isort==6.0.0
jedi==0.19.2
jiter==0.8.2
jmespath==1.0.1
jsonpath-ng==1.7.0
matplotlib-inline==0.1.7
mlflow-skinny==2.18.0
multidict==6.1.0
mypy-extensions==1.0.0
numpy==2.2.2
openai==1.60.2
opentelemetry-api==1.16.0
opentelemetry-sdk==1.16.0
opentelemetry-semantic-conventions==0.37b0
packaging==24.2
pandas==2.2.3
parso==0.8.4
pathspec==0.12.1
pexpect==4.9.0
platformdirs==4.3.6
pluggy==1.5.0
ply==3.11
prompt_toolkit==3.0.50
propcache==0.2.1
protobuf==5.29.3
ptyprocess==0.7.0
pure_eval==0.2.3
pyasn1==0.6.1
pyasn1_modules==0.4.1
pycparser==2.22
pydantic==2.10.6
pydantic-settings==2.7.1
pydantic_core==2.27.2
Pygments==2.19.1
pytest==8.3.4
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2025.1
PyYAML==6.0.2
requests==2.32.3
rsa==4.9
ruff==0.1.15
s3transfer==0.11.2
sentry-sdk==2.20.0
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
sqlparse==0.5.3
stack-data==0.6.3
tqdm==4.67.1
traitlets==5.14.3
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
wcwidth==0.2.13
webdriver-manager==4.0.2
wrapt==1.17.2
xmltodict==0.14.2
yarl==1.18.3
zipp==3.21.0

Operating System

AWS Lambda x86_64

[Kyle-Verhoog]

Hi @harshilprajapati96, yes we don't currently support reporting vision payloads. Are you seeing these errors with our OpenAI auto instrumentation or just when instrumenting manually?