support usage tracking with response api to _merge_usage_entries #8728
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@okhat @TomeHirata please have a look at this too 🙇🏼 |
TomeHirata
reviewed
Sep 3, 2025
| @@ -40,6 +42,8 @@ def _merge_usage_entries(self, usage_entry1: dict[str, Any] | None, usage_entry2 | |||
| current_v = result.get(k) | |||
| if isinstance(v, dict) or isinstance(current_v, dict): | |||
| result[k] = self._merge_usage_entries(current_v, v) | |||
| elif hasattr(v, "__dict__") or hasattr(current_v, "__dict__"): | |||
There was a problem hiding this comment.
Do we need to expose input_tokens_details or output_tokens_details in dspy? It might be confusing if we use both chat completion and responses together. Can we just filter out these new fields for now?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With response API you get additional usage token information in the form of litellm's InputTokenDetails and OutputTokenDetails.
Because these are objects the current _merge_usage_entries cannot handle them and thus cause the entire request to fail.
This PR fixes that.
This pull request improves the aggregation logic for usage tracking and adds comprehensive tests to ensure correct merging of token usage details, including handling of custom objects and
Nonevalues.Improvements to usage aggregation logic
_merge_usage_entriesmethod indspy/utils/usage_tracker.pyto correctly merge usage entries when values are custom objects (with a__dict__), ensuring that details from types likeInputTokensDetailsandOutputTokensDetailsare properly aggregated._merge_usage_entriesfor better readability and consistency.Expanded test coverage
tests/utils/test_usage_tracker.pyto verify that input and output token details are accurately aggregated across multiple usage entries, including cases with missing orNonevalues.InputTokensDetailsandOutputTokensDetailsfromlitellm.types.llms.openaito support new test scenarios.