Skip to content

Benchmark HF optimum-executorch #11450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Benchmark HF optimum-executorch #11450

wants to merge 2 commits into from
+210 −53

Conversation

guangy10
Copy link
Contributor

@guangy10 guangy10 commented Jun 6, 2025 •
edited
Loading

Benchmark LLMs from optimum-executorch. With all the work recently happening in optimum-executorch, we are able to boost the out-of-the-box performance. Putting these models on benchmark infra to gather perf numbers and understand the remaining perf gaps between the in-house generated model via export_llama.

We are able to do apple-to-apple comparison for CPU backend by introducing quant, custom SPDA, custom KV cache to native Hugging Face models in optimum-executorch: hf_xnnpack_custom_spda_kv_cache_8da4w represents the recipe used by optimum-et, et_xnnpack_custom_spda_kv_cache_8da4w is the counterpart for etLLM.

Here are the benchmark jobs in our infra:

Note there may be failures when running optimum-et models on-device due to lack of support HF tokenizers in llama runner. I will remove packing tokenizer.json from the .zip shortly so that the benchmark apps will take optimum-et LLMs as non-GenAI models.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 6, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11450

Note: Links to docs will display an error until the docs builds have been completed.

❌ 31 New Failures, 13 Cancelled Jobs, 6 Pending

As of commit 6c80e04 with merge base b632906 (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 6, 2025
@guangy10 guangy10 had a problem deploying to upload-benchmark-results June 6, 2025 19:38 — with GitHub Actions Failure
@guangy10 guangy10 had a problem deploying to upload-benchmark-results June 6, 2025 19:38 — with GitHub Actions Failure
@guangy10 guangy10 had a problem deploying to upload-benchmark-results June 6, 2025 19:40 — with GitHub Actions Failure
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 19:43 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from e4718b0 to fff15c6 Compare June 6, 2025 19:46
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 19:51 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 20:37 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 20:45 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from fff15c6 to 00149f2 Compare June 6, 2025 20:46
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 20:51 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from 00149f2 to 112eb2b Compare June 6, 2025 21:14
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 21:19 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 21:38 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from 112eb2b to a38a694 Compare June 6, 2025 22:02
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from a38a694 to a0f636f Compare June 6, 2025 22:50
@guangy10 guangy10 marked this pull request as ready for review June 6, 2025 22:50
@guangy10 guangy10 changed the title Benchmark optimum-executorch Benchmark HF optimum-executorch Jun 6, 2025
@guangy10 guangy10 added the release notes: none Do not include this in the release notes label Jun 6, 2025
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 23:43 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 6, 2025 23:50 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from a0f636f to 5d6dd04 Compare June 7, 2025 01:23
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 01:25 — with GitHub Actions Inactive
@guangy10
Copy link
Contributor Author

guangy10 commented Jun 7, 2025 •
edited
Loading

@huydhn In the apple's workflow, though I have specified the python version to be "3.11", it still install the python "3.13". Then when trying to pip install executorch, it ended with no package found. That's because we only publish with python 3.10, 3.11, and 3.12. https://github.com/pytorch/executorch/actions/runs/15500604843/job/43647388676#step:9:13372

Okay, it turns out that I need to run install with ${CONDA_RUN}

@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 02:15 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 02:16 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from 5d6dd04 to 01ce07b Compare June 7, 2025 03:19
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 03:33 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch 2 times, most recently from df785ca to 8aa9c02 Compare June 7, 2025 03:48
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 03:51 — with GitHub Actions Inactive
@guangy10
Copy link
Contributor Author

guangy10 commented Jun 7, 2025 •
edited
Loading

Another issue. @huydhn Why isn't the artifact uploaded though it's zipped and moved to the dest dir successfully?
https://github.com/pytorch/executorch/actions/runs/15502766906/job/43653221066#step:20:1

This is fixed. It turns out the artifacts must be moved from the ET root dir to the artifacts-to-be-uploaded, mv from other dir under ET root will end up with the above issue.

@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 04:27 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 04:35 — with GitHub Actions Inactive
@guangy10 guangy10 force-pushed the optimum_et_benchmark branch from 8aa9c02 to b0d829a Compare June 7, 2025 04:35
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 05:40 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 05:55 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 06:00 — with GitHub Actions Inactive
@guangy10 guangy10 temporarily deployed to upload-benchmark-results June 7, 2025 06:19 — with GitHub Actions Inactive
@guangy10 guangy10 had a problem deploying to upload-benchmark-results June 7, 2025 08:08 — with GitHub Actions Failure
@guangy10 guangy10 had a problem deploying to upload-benchmark-results June 7, 2025 09:30 — with GitHub Actions Failure
kimishpatel
kimishpatel reviewed Jun 9, 2025
View reviewed changes
.ci/scripts/gather_benchmark_configs.py
Comment on lines +35 to +36
"hf_xnnpack_custom_spda_kv_cache_8da4w",
"et_xnnpack_custom_spda_kv_cache_8da4w",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whats the difference between these two?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the explaination in the PR summary: hf_xnnpack_custom_spda_kv_cache_8da4w represents the recipe used by optimum-et, et_xnnpack_custom_spda_kv_cache_8da4w is the counterpart for etLLM.

kimishpatel
kimishpatel reviewed Jun 9, 2025
View reviewed changes
.github/workflows/android-perf.yml
-X \
--xnnpack-extended-ops \
-qmode 8da4w -G 32 -E 8,0 \
--metadata '{"get_bos_id":128000, "get_eos_ids":[128009, 128001]}' \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these for llama_3_2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kimishpatel Yeah, for llama_3_2.

@guangy10
Copy link
Contributor Author

guangy10 commented Jun 9, 2025 •
edited
Loading

I'm seeing jobs hitting API Limits in AWS Device Farm. We lifted it for public AWS devices, @huydhn do we need to do same and separately for new devices in private pools? https://github.com/pytorch/executorch/actions/runs/15504512047

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kimishpatel kimishpatel kimishpatel left review comments

@huydhn huydhn Awaiting requested review from huydhn

@kirklandsign kirklandsign Awaiting requested review from kirklandsign

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: none Do not include this in the release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@guangy10 @kimishpatel @facebook-github-bot