|
| 1 | +# Web Platform Test (WPT) |
| 2 | + |
| 3 | +Deno uses a custom test runner for Web Platform Tests. It can be found at |
| 4 | +`./tests/wpt/wpt.ts`, relative to the root of this codebase. |
| 5 | + |
| 6 | +## Running tests |
| 7 | + |
| 8 | +> If you are on Windows, or your system does not support hashbangs, prefix all |
| 9 | +> `./tests/wpt/wpt.ts` commands with |
| 10 | +> `deno run --unstable --allow-write --allow-read --allow-net --allow-env --allow-run`. |
| 11 | +
|
| 12 | +Before attempting to run WPT tests for the first time, please run the WPT setup. |
| 13 | +You must also run this command every time the `./test_util/wpt` submodule is |
| 14 | +updated: |
| 15 | + |
| 16 | +```shell |
| 17 | +./tests/wpt/wpt.ts setup |
| 18 | +``` |
| 19 | + |
| 20 | +To run all available web platform tests, run the following command: |
| 21 | + |
| 22 | +```shell |
| 23 | +./tests/wpt/wpt.ts run |
| 24 | + |
| 25 | +# You can also filter which test files to run by specifying filters: |
| 26 | +./tests/wpt/wpt.ts run -- streams/piping/general hr-time |
| 27 | +``` |
| 28 | + |
| 29 | +The test runner will run each web platform test and record its status (failed or |
| 30 | +ok). It will then compare this output to the expected output of each test as |
| 31 | +specified in the `./tests/wpt/runner/expectation.json` file. This file is a |
| 32 | +nested JSON structure that mirrors the `./tests/wpt/suite` directory. It |
| 33 | +describes for each test file, if it should pass as a whole (all tests pass, |
| 34 | +`true`), if it should fail as a whole (test runner encounters an exception |
| 35 | +outside of a test or all tests fail, `false`), or which tests it expects to fail |
| 36 | +(a string array of test case names). |
| 37 | + |
| 38 | +## Updating enabled tests or expectations |
| 39 | + |
| 40 | +You can update the `./tests/wpt/runner/expectation.json` file manually by |
| 41 | +changing the value of each of the test file entries in the JSON structure. The |
| 42 | +alternative and preferred option is to have the WPT runner run all, or a |
| 43 | +filtered subset of tests, and then automatically update the `expectation.json` |
| 44 | +file to match the current reality. You can do this with the `./wpt.ts update` |
| 45 | +command. Example: |
| 46 | + |
| 47 | +```shell |
| 48 | +./tests/wpt/wpt.ts update -- hr-time |
| 49 | +``` |
| 50 | + |
| 51 | +After running this command the `expectation.json` file will match the current |
| 52 | +output of all the tests that were run. This means that running `wpt.ts run` |
| 53 | +right after a `wpt.ts update` should always pass. |
| 54 | + |
| 55 | +## Subcommands |
| 56 | + |
| 57 | +### `setup` |
| 58 | + |
| 59 | +Validate that your environment is configured correctly, or help you configure |
| 60 | +it. |
| 61 | + |
| 62 | +This will check that the python3 (or `python.exe` on Windows) is actually |
| 63 | +Python 3. |
| 64 | + |
| 65 | +You can specify the following flags to customize behaviour: |
| 66 | + |
| 67 | +```console |
| 68 | +--rebuild |
| 69 | + Rebuild the manifest instead of downloading. This can take up to 3 minutes. |
| 70 | + |
| 71 | +--auto-config |
| 72 | + Automatically configure /etc/hosts if it is not configured (no prompt will be shown). |
| 73 | +``` |
| 74 | + |
| 75 | +### `run` |
| 76 | + |
| 77 | +Run all tests like specified in `expectation.json`. |
| 78 | + |
| 79 | +You can specify the following flags to customize behaviour: |
| 80 | + |
| 81 | +```console |
| 82 | +--release |
| 83 | + Use the ./target/release/deno binary instead of ./target/debug/deno |
| 84 | + |
| 85 | +--quiet |
| 86 | + Disable printing of `ok` test cases. |
| 87 | + |
| 88 | +--json=<file> |
| 89 | + Output the test results as JSON to the file specified. |
| 90 | +``` |
| 91 | + |
| 92 | +You can also specify exactly which tests to run by specifying one of more |
| 93 | +filters after a `--`: |
| 94 | + |
| 95 | +```console |
| 96 | +./tests/wpt/wpt.ts run -- hr-time streams/piping/general |
| 97 | +``` |
| 98 | + |
| 99 | +### `update` |
| 100 | + |
| 101 | +Update the `expectation.json` to match the current reality. |
| 102 | + |
| 103 | +You can specify the following flags to customize behaviour: |
| 104 | + |
| 105 | +```console |
| 106 | +--release |
| 107 | + Use the ./target/release/deno binary instead of ./target/debug/deno |
| 108 | + |
| 109 | +--quiet |
| 110 | + Disable printing of `ok` test cases. |
| 111 | + |
| 112 | +--json=<file> |
| 113 | + Output the test results as JSON to the file specified. |
| 114 | +``` |
| 115 | + |
| 116 | +You can also specify exactly which tests to run by specifying one of more |
| 117 | +filters after a `--`: |
| 118 | + |
| 119 | +```console |
| 120 | +./tests/wpt/wpt.ts update -- hr-time streams/piping/general |
| 121 | +``` |
| 122 | + |
| 123 | +## FAQ |
| 124 | + |
| 125 | +### Upgrading the wpt submodule: |
| 126 | + |
| 127 | +```shell |
| 128 | +cd tests/wpt/suite |
| 129 | +git fetch origin |
| 130 | +git checkout origin/epochs/daily |
| 131 | +cd ../../../ |
| 132 | +git add ./tests/wpt/suite |
| 133 | +``` |
| 134 | + |
| 135 | +All contributors will need to rerun `./tests/wpt/wpt.ts setup` after this. |
| 136 | + |
| 137 | +Since upgrading WPT usually requires updating the expectations to cover all |
| 138 | +sorts of upstream changes, it's best to do that as a separate PR, rather than as |
| 139 | +part of a PR that implements a fix or feature. |
0 commit comments