Skip to content

Mistral-Small-24B-Instruct-2501 tasks.yml files #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 15 commits into from

Conversation

derekk-nm
Copy link
Contributor

@derekk-nm derekk-nm commented Apr 21, 2025

SUMMARY:
adds task.yml config files for the models derived from mistralai/Mistral-Small-24B-Instruct-2501.
The task metric values are from the model card for the specific model (though the values for the "parent" come from one of the RedHatAI models, where the evaluation accuracy table shows the measured parent metrics).

[RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w8a8](https://huggingface.co/RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w8a8) is currently (202504210830) missing model card content.

TEST PLAN:
All runs are showing failures. these are likely due to the fact that either the server or client do not have configuration settings set exactly as described in the model card's evaluation section.

@derekk-nm
Copy link
Contributor Author

a different source has been chose for the ground truth. This PR is now deprecated.

@derekk-nm derekk-nm closed this Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant