[FT] Save reasoning_content of responses as details

[muupan]

Issue encountered

When evaluating a reasoning model via litellm, it seems like reasoning_content is ignored by lighteval as of now:

lighteval/src/lighteval/models/endpoints/litellm_model.py

Line 296 in 458a3e9

result: list[str] = [choice.message.content for choice in response.choices]

It would be nice to save reasoning_content of responses as details even if they are not used for computing metrics. Inspecting the reasonings can be helpful for development.

Solution/Feature

litellm can return reasoning_content separately from content. https://github.com/BerriAI/litellm/blob/aac7cdf2c8b58f71042c661bc732d936fd4b09ce/litellm/types/utils.py#L575.

However, lighteval's ModelResponse does not have an attributes for reasoning texts.

lighteval/src/lighteval/models/model_output.py

Line 29 in 4d2bf42

class ModelResponse:

I guess adding an attribute to it and setting its value in LiteLLMClient would be sufficient for the litellm entry point. I can make a pull request for the litellm entry point if this sounds ok. Other entry points would require similar modifications, but I haven't checked.

[NathanHB]

Hi ! Feel free to open a PR for this. For the other entry point, the reasoning is bundled in the output text

[muupan]

Thanks, I opened a PR: #929