Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Subset of E2E models for accuracy tests #2246

Closed
pbchekin opened this issue Sep 13, 2024 · 2 comments · Fixed by #2272, #2429 or #2428
Closed

Subset of E2E models for accuracy tests #2246

pbchekin opened this issue Sep 13, 2024 · 2 comments · Fixed by #2272, #2429 or #2428
Assignees
@kwasd
Labels
accuracy ci enhancement New feature or request tests: e2e validation
Milestone
3.0 [PT 2.5 E2E] ...

Comments

@pbchekin
Copy link
Contributor

pbchekin commented Sep 13, 2024 •
edited
Loading

We currently have a subset of models to test E2E performance. We need similar subset for E2E accuracy tests. If the E2E accuracy workflow started with the subset, it should check that all models in the subset passed the accuracy check.

Periodic E2E jobs (E2E accuracy, Try latest PyTorch) need to use the subset mode to catch regressions early.
@vlad-penkin
Copy link
Contributor

@pbchekin this ticket partially duplicates

Let's align scope between two tickets.

@vlad-penkin vlad-penkin assigned pbchekin and unassigned kwasd and pbchekin Sep 13, 2024
@vlad-penkin vlad-penkin added validation enhancement New feature or request labels Sep 13, 2024
@kwasd kwasd mentioned this issue Sep 18, 2024
Merged
kwasd added a commit that referenced this issue Sep 18, 2024
@kwasd
Switch e2e for try-latest-pytorch.yml to subset (#2272)
f54d008 
For #2246
@kwasd kwasd linked a pull request Sep 18, 2024 that will close this issue
Switch e2e for try-latest-pytorch.yml to subset #2272
Merged
This was referenced Oct 7, 2024
Merged
Merged
This was linked to pull requests Oct 7, 2024
Use separate models subset folder for performance/accuracy e2e tests #2429
Merged
Add test_all_subset_models mode #2428
Merged
@kwasd kwasd reopened this Oct 15, 2024
@kwasd
Copy link
Contributor

kwasd commented Oct 21, 2024

Accuracy run: https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/11445587582
Performance run: https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/11445592099

kwasd added a commit that referenced this issue Oct 21, 2024
@kwasd @pbchekin
Add test_all_subset_models mode (#2428)
b6cdccd 
For #2246

---------

Co-authored-by: Pavel Chekin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kwasd kwasd

Labels
accuracy ci enhancement New feature or request tests: e2e validation
Projects
None yet
Milestone
3.0 [PT 2.5 E2E] Pass rate
3 participants
@kwasd @pbchekin @vlad-penkin