Implement automatic test suite skipping facility into the browser test harness #25533
Conversation
…t harness, so that users do not need to manually maintain skip lists of features. Paves the way toward test harness working for users out of the box.
There was a problem hiding this comment.
Very nice!
I do worry a little bit about any mechanism that automatically skips test though.
Having to spell out EMTEST_LACKS_GRAPHICS_HARDWARE explicitly in the CI configuration means that the we don't need to trust the auto-disable feature.
This is why we have EMTEST_SKIP_NODE_CANARY explicitly rather than trying to automatically figure out at runtime if we should skip them.
I think I've seen cases where we were mistakenly always skipping certain tests due to a bug in the auto-skip magic, and its went unnoticed.
Imagine this scenario where we have an auto-skip mechanism:
1.I setup a CI bot with idea that it will run all the tests under node.
2. I install node on the bot
3. I run the tests and see that they all pass
4. Someone adds some new tests that depend on a more recent version of node.
5. An auto-skip mechanism means I never actually run them and bot keeps passing
I would much rather have my the bot start failing at (4) so can explictly choose to either:
- Update node.
- Set some environment variable such as
SKIP_NODE24orSKIP_MEMORY64to opt out explicitly.
Another example is that we don't automatically skip d8 tests when d8 is not found. We expect folks to explicitly opt out with EMTEST_SKIP_V8.
Maybe your are not proposing that we go that far, but JIC I thought I would mention the rationale for the status quo.
| # On Nightly and BEta, e.g. 145.0a1, pretend it to still mean version 144, | ||
| # since it is a pre-release version | ||
| if any(c in milestone for c in ("a", "b")): | ||
| version -= 1 |
There was a problem hiding this comment.
Can we not use firefox --version for this?
|
|
||
| def current_browser_lacks_feature(feature): | ||
| min_required = min_browser_versions[feature] | ||
| return get_firefox_version() < min_required['firefox'] or get_safari_version() < min_required['safari'] |
There was a problem hiding this comment.
Move this up to before its first usage.
Implement automatic test suite skipping facility into the browser test harness, so that users do not need to manually maintain skip lists of features. (or in present state, have to maintain fewer of them).
Paves the way toward test harness working for users out of the box, improving First-Time User Experience, rather than have a large bulk of tests fail for users by default.