Skip to content

Commit

Permalink
Document and generalize DISABLE_DEMO as RUN_XFAIL.
Browse files Browse the repository at this point in the history
I want to use this for a few other longstanding known failures.

This patch:

- Changes the variable names from disable demo to run xfail, for the
  sake of being general and avoiding double negatives everywhere.
- Changes the way the check is made to be consistent with SHOW_BROWSER
- Documents the variable in tests/README.md
  • Loading branch information
zenhack committed Apr 10, 2022
1 parent 8cfd01c commit 87650fc
Show file tree
Hide file tree
Showing 4 changed files with 27 additions and 6 deletions.
21 changes: 21 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,27 @@ windows when debugging. You can do this by setting `SHOW_BROWSER=true`:

SHOW_BROWSER=true make test

## Dealing with tests which are expected to fail

Some tests are known to fail, either always or intermittently. Obviously
we should fix these, but so that the full test suite can remain useful
in the interim, we disable these tests by default; if you want to run
them you can set `RUN_XFAIL=true`:

RUN_XFAIL=true make test

When writing tests, this variable is exposed as `run_xfail` in
`tests/util.js`; you can disable a test by simply wrapping it in an
if statement:

```js
if (utils.run_xfail) {
module.exports["Test something broken"] = function (browser) {
// ...
};
}
```

## How to dump the DOM for debugging

Stick this in the test:
Expand Down
2 changes: 1 addition & 1 deletion tests/run-local.sh
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ echo

set +e

export DISABLE_DEMO=true
export RUN_XFAIL="${RUN_XFAIL:-false}"
"$NPM" test

cleanExit $?
4 changes: 2 additions & 2 deletions tests/tests/basic.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

var utils = require('../utils'),
short_wait = utils.short_wait,
disable_demo = utils.disable_demo;
run_xfail = utils.run_xfail;

module.exports = {
"Test title" : function (browser) {
Expand All @@ -36,7 +36,7 @@ module.exports = {
.end();
},
};
if (!disable_demo) {
if (run_xfail) {
module.exports["Test demo login command"] = function (browser) {
browser
.loginDemo()
Expand Down
6 changes: 3 additions & 3 deletions tests/utils.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
'use strict';

var disable_demo = !!process.env.DISABLE_DEMO;
var run_xfail = process.env.RUN_XFAIL !== "false";

var wrapLoginDemo = function(test) {
return function (browser) {
Expand All @@ -22,13 +22,13 @@ module.exports = {
very_long_wait: 180000,
default_width: 1280,
default_height: 1024,
disable_demo: disable_demo,
run_xfail: run_xfail,
testAllLogins: function (tests) {
var newTests = {};

var count = 0;
var name, test;
if (!disable_demo) {
if (run_xfail) {
for(name in tests) {
test = tests[name];
if (count === 0) {
Expand Down

0 comments on commit 87650fc

Please sign in to comment.