Skip to content

Conversation

@iamemilio
Copy link
Contributor

@iamemilio iamemilio commented Aug 18, 2025

Fix gaps in Responses Response Object

Fixes #3040

Note: this is hand written and will need to be maintained, a long term strategy to stay in sync with OpenAI is needed.

@meta-cla
Copy link

meta-cla bot commented Aug 18, 2025

Hi @iamemilio!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 18, 2025

created_at: int
error: OpenAIResponseError | None = None
error: Optional[OpenAIResponseError] = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think mypy will yell at you for this and ask you to do | None

"""


class ToolChoiceAllowed(BaseModel):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if these are nested you might need to add json_schema

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you run pre-commit run --all-files the openapi spec will try to regen. Might need to pip install pre-commit if you haven't

format: OpenAIResponseTextFormat | None = None
# Default to text format to avoid breaking the loading of old responses
# before the field was added. New responses will have this set always.
format: OpenAIResponseTextFormat | None = Field(default_factory=lambda: OpenAIResponseTextFormat(type="text"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes sense to me I think

@iamemilio
Copy link
Contributor Author

Thanks, I'll patch this!

@ashwinb
Copy link
Contributor

ashwinb commented Aug 20, 2025

Pre-commit seems to be still failing?

@iamemilio
Copy link
Contributor Author

iamemilio commented Aug 20, 2025

I think I found a good resolution for this. I enhanced the openapi generation to allow Literals with multiple fields. I also realize that these fields are not available in the create_openai_response() function either, but I am not sure what other obligation to implement them that would come with. I would prefer to address that as a series of follow up pull requests to be more cautious.

@iamemilio iamemilio force-pushed the responses_object branch 2 times, most recently from e871e57 to 82a9099 Compare August 20, 2025 22:10
@iamemilio
Copy link
Contributor Author

updated with merge commit, hopefully the protobuff bug in the vec-io test has been patched

Copy link
Contributor

@ashwinb ashwinb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the test plan for this?

@iamemilio
Copy link
Contributor Author

iamemilio commented Aug 26, 2025

Thanks for the review, I am still very new to this code base so its helpful to get some expert guidance! I added some unit tests just to make sure this change doesn't come with any unintended consequences. I want to clarify that these changes only impact the responses.Response object that gets returned, it makes no attempt to actually utilize the new fields that get added and connect them to the providers they need to be connected to. That is something that will come in follow up issues. My goal for this PR is to just define the types we need to support, and make sure that the clients get updated to support them to handle compatibility errors.

Do you think these tests are appropriate enough, or would you recommend additional testing for this PR? Also, is there additional testing that needs to be done to support the stainless clients?

@cdoern
Copy link
Contributor

cdoern commented Sep 16, 2025

@iamemilio , can you please rebase and re-push so the API Conformance test triggers?

@iamemilio iamemilio closed this Sep 17, 2025
@iamemilio iamemilio reopened this Sep 17, 2025
@iamemilio
Copy link
Contributor Author

@ashwinb could you take one more look at this

@iamemilio iamemilio requested a review from ashwinb September 22, 2025 14:00
@cdoern
Copy link
Contributor

cdoern commented Sep 23, 2025

running an oasdiff against the openAI API:

╰─ oasdiff breaking --fail-on ERR \
  docs/_static/llama-stack-spec.yaml \
  /Users/charliedoern/Downloads/openapi.documented.yml \
  --strip-prefix-base "/v1/openai/v1" \
  --match-path '(^/v1/openai/v1/responses.*|^/responses.*)'
74 changes: 22 error, 52 warning, 0 info
error	[api-removed-without-deprecation] at docs/_static/llama-stack-spec.yaml
	in API GET /responses
		api removed without deprecation

error	[request-body-all-of-added] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		added '#/components/schemas/CreateModelResponseProperties, #/components/schemas/ResponseProperties, subschema #3' to the request body 'allOf' list

error	[request-body-type-changed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		the request's body type/format changed from 'object'/'' to ''/''

error	[response-body-type-changed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		the response's body type/format changed from 'object'/'' to ''/'' for status '200'

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'created_at' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'id' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'model' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'object' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'output' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'parallel_tool_calls' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the required property 'status' from the response with the '200' status

error	[response-media-type-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API DELETE /responses/{response_id}
		removed the media type 'application/json' for the response with the status '200'

error	[response-body-type-changed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		the response's body type/format changed from 'object'/'' to ''/'' for status '200'

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'created_at' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'id' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'model' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'object' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'output' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'parallel_tool_calls' from the response with the '200' status

error	[response-required-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the required property 'status' from the response with the '200' status

error	[request-parameter-default-value-added] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}/input_items
		for the 'query' request parameter 'limit', default value '20.00' was added

error	[response-property-type-changed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}/input_items
		the 'object' response's property type/format changed from 'string'/'' to ''/'' for status '200'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'include'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'input'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'instructions'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'max_infer_iters'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'model'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'previous_response_id'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'store'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'stream'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'temperature'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'text'

warning	[request-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the request property 'tools'

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'error' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'incomplete_details' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'instructions' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'max_output_tokens' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'max_tool_calls' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'metadata' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'previous_response_id' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'prompt' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'prompt_cache_key' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'reasoning' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'safety_identifier' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'service_tier' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'temperature' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'text' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'tool_choice' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'tools' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'top_logprobs' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'top_p' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'truncation' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API POST /responses
		removed the optional property 'user' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'error' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'incomplete_details' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'instructions' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'max_output_tokens' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'max_tool_calls' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'metadata' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'previous_response_id' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'prompt' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'prompt_cache_key' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'reasoning' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'safety_identifier' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'service_tier' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'temperature' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'text' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'tool_choice' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'tools' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'top_logprobs' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'top_p' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'truncation' from the response with the '200' status

warning	[response-optional-property-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}
		removed the optional property 'user' from the response with the '200' status

warning	[request-parameter-removed] at /Users/charliedoern/Downloads/openapi.documented.yml
	in API GET /responses/{response_id}/input_items
		deleted the 'query' request parameter 'before'
		This is a warning because some apps may return an error when receiving a parameter that they do not expect. It is recommended to deprecate the parameter first.

there are a few "breakages" but they might be misnomers, please take a look ! @iamemilio

@cdoern
Copy link
Contributor

cdoern commented Sep 23, 2025

I could be using a different version of the spec, or the tool could be too picky on what is "broken" but I figured I'd share just in case ^

@iamemilio
Copy link
Contributor Author

Seems like at least some of these are a me problem. Let me see if there is a better way to address this. Where are you getting the latest openapi spec for openai?

@cdoern
Copy link
Contributor

cdoern commented Sep 23, 2025

@bbrowning
Copy link
Collaborator

Was Ashwin's comment and request about a test plan addressed yet? Were you able to run Llama Stack with these changes against a common provider, such as vLLM, and ensure the integration tests all pass (or at least no new ones fail...) from before/after this change?

@iamemilio
Copy link
Contributor Author

iamemilio commented Sep 25, 2025

It seems a major refactor is in order in the openapi spec, this PR will need to be re-written. Then I will address the review concerns.

@iamemilio
Copy link
Contributor Author

Hi all, I am going to close this and re-address this using the smaller issues filed.

@iamemilio iamemilio closed this Sep 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Responses endpoint - return JSON does not match spec

4 participants