Skip to content

add task.yml files for granite-3.1-8B-Instruct models #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

derekk-nm
Copy link
Contributor

@derekk-nm derekk-nm commented Apr 17, 2025

SUMMARY:
adds task.yml config files for the models derived from ibm-granite/granite-3.1-8B-Instruct.
The task metric values are from the model card for the specific model (though the values for the "parent" come from the one of the RedHatAI models, where the evaluation accuracy table shows the measured parent metrics).

TEST PLAN:
All runs are initially showing failures. these are likely due to the fact that either the server or client do not have configuration settings set exactly as described in the model card's evaluation section.
Follow up runs were done w/ the relative tolerance (rtol) set for tasks that were too far off the default 0.04 setting. these need to be reviewed by research.

  • ibm-granite/granite-3.1-8b-instruct accuracy ... follow up with new rtol and use 0.8.4 accuracy
  • RedHatAI/granite-3.1-8b-instruct-quantized.w4a16 accuracy ... follow up with new rtol and use 0.8.4 accuracy
  • RedHatAI/granite-3.1-8b-instruct-quantized.w8a8 accuracy ... follow up with new rtol and use 0.8.4 accuracy
  • RedHatAI/granite-3.1-8b-instruct-FP8-dynamic accuracy ... follow up with new rtol for problem tasks accuracy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants