Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[e2e] Add option for continue running e2e testcases after failure #16194

Closed
wants to merge 1 commit into from

Conversation

seaona
Copy link
Contributor

@seaona seaona commented Oct 13, 2022

Explanation

Why: currently when we run the script for executing all e2e tests, they are stopped and script is terminated if a failure is encountered. There are certain occasions that we want to continue them running.
For example, on the MV3 job, we want to run all the tests to see all failing tests. Or a developer might want to run them all to see all the failures at once.
There are also cases when we want the tests to fail fast, i.e. capture errors fast on a circle ci run, or avoid spending too much resources on circle ci.
For this reason we'll include this change as an option.

What: we'll include an option for running all e2e tests no matter if they fail.

How:

  • We'll make it the default option for the MV3 run, for now, as it is convenient to see ALL the failing tests.
  • Anyone can add this option flag when running their tests locally by adding the --resume flag.

More Information

Resolves #16192

Screenshots/Screencaps

Before

  • Run-all script terminated on failure

After

  • Run-all script continues after failure if --resume option is passed

Manual Testing Steps

Locally

  1. Change last assertion on the account-details.spec.js file in order to make it fail:
    assert.equal(await qrCode.isDisplayed(), "sdkfha");
  2. Run all tests and see how they continue running after failure if resume option is passed
    yarn test:e2e:chrome --resume

On Circle CI
Check that MV3 e2e testcases continue after failure, on circle ci job.
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/29114/workflows/dc9db3da-099a-44a5-8896-dad6aa71b091/jobs/756717

Pre-Merge Checklist

  • PR template is filled out
  • IF this PR fixes a bug, a test that would have caught the bug has been added
  • PR is linked to the appropriate GitHub issue
  • PR has been added to the appropriate release Milestone

+ If there are functional changes:

  • Manual testing complete & passed
  • "Extension QA Board" label has been applied

@github-actions
Copy link
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@seaona seaona marked this pull request as ready for review October 13, 2022 09:50
@seaona seaona requested review from a team and kumavis as code owners October 13, 2022 09:50
@seaona seaona requested a review from darkwing October 13, 2022 09:50
@metamaskbot
Copy link
Collaborator

Builds ready [6038849]
Page Load Metrics (2323 ± 94 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint852911164220
domContentLoaded18422712230520398
load19022712232319594
domInteractive18422712230520498

Copy link
Contributor

@PeterYinusa PeterYinusa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This looks good. I'm wondering if we could have used mocha's bail option, and further extend it as an option in run-all.js replacing resume.

Copy link
Contributor

@brad-decker brad-decker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted as I seem to have misunderstood

@github-actions
Copy link
Contributor

This PR has been automatically marked as stale because it has not had recent activity in the last 60 days. It will be closed in 14 days. Thank you for your contributions.

@github-actions github-actions bot added the stale issues and PRs marked as stale label Jul 20, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Aug 3, 2023

This PR was closed because there has been no follow up activity in the last 14 days. Thank you for your contributions.

@github-actions github-actions bot closed this Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale issues and PRs marked as stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[e2e] Run all tests without terminating the script on a failure
4 participants