-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[e2e] Add option for continue running e2e testcases after failure #16194
Conversation
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
Builds ready [6038849]
Page Load Metrics (2323 ± 94 ms)
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: This looks good. I'm wondering if we could have used mocha's bail
option, and further extend it as an option in run-all.js
replacing resume
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deleted as I seem to have misunderstood
This PR has been automatically marked as stale because it has not had recent activity in the last 60 days. It will be closed in 14 days. Thank you for your contributions. |
This PR was closed because there has been no follow up activity in the last 14 days. Thank you for your contributions. |
Explanation
Why: currently when we run the script for executing all e2e tests, they are stopped and script is terminated if a failure is encountered. There are certain occasions that we want to continue them running.
For example, on the MV3 job, we want to run all the tests to see all failing tests. Or a developer might want to run them all to see all the failures at once.
There are also cases when we want the tests to fail fast, i.e. capture errors fast on a circle ci run, or avoid spending too much resources on circle ci.
For this reason we'll include this change as an option.
What: we'll include an option for running all e2e tests no matter if they fail.
How:
--resume
flag.More Information
Resolves #16192
Screenshots/Screencaps
Before
Run-all
script terminated on failureAfter
Run-all
script continues after failure if--resume
option is passedManual Testing Steps
Locally
account-details.spec.js
file in order to make it fail:assert.equal(await qrCode.isDisplayed(), "sdkfha");
resume
option is passedyarn test:e2e:chrome --resume
On Circle CI
Check that MV3 e2e testcases continue after failure, on circle ci job.
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/29114/workflows/dc9db3da-099a-44a5-8896-dad6aa71b091/jobs/756717
Pre-Merge Checklist
+ If there are functional changes: