Add One Double Zero as coverage provider

[ericmorand]

Summary

As explained in the issue, v8 coverage provider comes with some important tradeoffs compared to the babel/istanbul one. One Double Zero is a code coverage tool and API that consumes V8 coverage data and targets the accuracy and correctness of istanbul. It does this by operating at the AST level.

This PR adds One Double Zero as a coverage provider.

It also updates the documentation, and explains the tradeoffs of the v8 coverage provider.

Test plan

The plan is to execute odz on each e2e test executed by the v8 coverage provider test suite, and use the output result as the snapshot for e2e/__tests__/coverageProviderODZ.test.ts. The test script of each e2e has to be changed to be executable wth either node or ts-node, which does not impact the coverage result.

• e2e/coverage-provider-v8/cjs-native-without-sourcemap

e2e/coverage-provider-v8/cjs-native-without-sourcemap$ odz --sources=module.js --sources=uncovered.js node __tests__/test.js

this will print
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |      60 |       50 |      50 |      60 |                   
 module.js    |   66.66 |       50 |      50 |   66.66 | 14-15,19          
 uncovered.js |       0 |      100 |     100 |       0 | 8                 
--------------|---------|----------|---------|---------|-------------------

• e2e/coverage-provider-v8/cjs-with-babel-transformer

e2e/coverage-provider-v8/cjs-with-babel-transformer$ odz --sources=module.ts --sources=uncovered.ts --sources=types.ts ts-node -T __tests__/test.ts

this will print
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |    62.5 |       50 |      50 |    62.5 |                   
 module.ts    |    62.5 |       50 |      50 |    62.5 | 16-17,21          
 types.ts     |       0 |        0 |       0 |       0 |                   
 uncovered.ts |       0 |        0 |       0 |       0 |                   
--------------|---------|----------|---------|---------|-------------------

• e2e/coverage-provider-v8/empty-sourcemap

e2e/coverage-provider-v8/empty-sourcemap$ odz  --sources=types.ts ts-node -T __tests__/test.ts

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |                   
 types.ts |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------

• e2e/coverage-provider-v8/esm-native-without-sourcemap

e2e/coverage-provider-v8/esm-native-without-sourcemap$ odz --sources=module.js node __tests__/test.js 

--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |    62.5 |       50 |      50 |    62.5 |                   
 module.js    |    62.5 |       50 |      50 |    62.5 | 14-15,19          
 uncovered.js |       0 |        0 |       0 |       0 |                   
--------------|---------|----------|---------|---------|-------------------

• e2e/coverage-provider-v8/esm-with-custom-transformer

e2e/coverage-provider-v8/esm-with-custom-transformer$ odz --sources=module.ts --sources=uncovered.ts --sources=types.ts ts-node -T __tests__/test.ts

--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |    62.5 |       50 |      50 |    62.5 |                   
 module.ts    |    62.5 |       50 |      50 |    62.5 | 16-17,21          
 types.ts     |       0 |        0 |       0 |       0 |                   
 uncovered.ts |       0 |        0 |       0 |       0 |                   
--------------|---------|----------|---------|---------|-------------------

• e2e/coverage-provider-v8/no-sourcemap

e2e/coverage-provider-v8/no-sourcemap$ odz --sources=Thing.js --sources=x.css node __tests__/Thing.test.js

42
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |     100 |      100 |     100 |     100 |                   
 Thing.js |     100 |      100 |     100 |     100 |                   
 x.css    |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------

• e2e/coverage-provider-v8/with-resetModules

e2e/coverage-provider-v8/with-resetModules$ odz --sources=module.js --sources=uncovered.js node __tests__/test.js

this will print
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |      60 |       50 |      50 |      60 |                   
 module.js    |   66.66 |       50 |      50 |   66.66 | 14-15,19          
 uncovered.js |       0 |      100 |     100 |       0 | 8                 
--------------|---------|----------|---------|---------|-------------------

• e2e/vmscript-coverage

e2e/vmscript-coverage$ odz --sources=module.js --sources=package/vmscript.js node __tests__/extract-coverage.test.js

-------------|---------|----------|---------|---------|-------------------
File         | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-------------|---------|----------|---------|---------|-------------------
All files    |      80 |       75 |   66.66 |      80 |                   
 vmscript.js |      80 |       75 |   66.66 |      80 | 20-21             
-------------|---------|----------|---------|---------|-------------------

A few changes had to be made, like passing the list of files to cover to the _getCoverageResult method, but globally the changes required to add odz were minimal.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ericmorand / name: Eric MORAND (cf49e94, 278a534)

[netlify]

✅ Deploy Preview for jestjs ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	278a534
🔍 Latest deploy log	https://app.netlify.com/sites/jestjs/deploys/67915455e2ca6f0008b22eea
😎 Deploy Preview	https://deploy-preview-15356--jestjs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[ericmorand-sonarsource]

Another interesting test is using jest to check the coverage of the jest project itself, with every istanbul ignore pragma removed for consistency.

Using babel as provider

$ yarn jest --coverage --coverageProvider=babel

------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
File                                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                     
------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
All files                                 |   68.51 |    65.55 |   65.33 |   68.54 |                                                                                                                                       

Test Suites: 1 failed, 470 passed, 471 total
Tests:       1 failed, 51 skipped, 5132 passed, 5184 total
Snapshots:   1766 passed, 1766 total
Time:        96.649 s
Ran all test suites in 15 projects.

Using v8 as provider

$ yarn jest --coverage --coverageProvider=v8

------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
File                                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                     
------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
All files                                 |   68.33 |    86.69 |   75.97 |   68.33 |                                                                                                                                       

Test Suites: 1 failed, 470 passed, 471 total
Tests:       1 failed, 51 skipped, 5132 passed, 5184 total
Snapshots:   1766 passed, 1766 total
Time:        81.74 s, estimated 90 s
Ran all test suites in 15 projects.

Using odz as provider

$ yarn jest --coverage --coverageProvider=odz

------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
File                                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                     
------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
All files                                 |   65.68 |    61.61 |   64.67 |   65.76 |                                                                                                                                    

Test Suites: 1 failed, 470 passed, 471 total
Tests:       1 failed, 51 skipped, 5132 passed, 5184 total
Snapshots:   1766 passed, 1766 total
Time:        76.922 s, estimated 79 s
Ran all test suites in 15 projects.

The most striking difference is the function and branch coverage reported by v8 that is way too high compared to the two others. This is expected and is a well-known issue:

• the v8 provider doesn't consider the relevancy of lines: empty lines and comments are considered as covered.
• the v8 provider considers that every single file contains at least one covered function.

Using v8 coverage data without knowledge of the meaning of each line of code makes for a very inaccurate coverage computation. Both babel and odz know about the meaning of each line of code: babel parses the code at instrumentation phase; odz parses the source files after the execution of the script.

The differences between babel and odz, minor, may come from how the tools work: babel parses and instruments the executed script; odz parses the source files. It is likely that both approaches lead to slightly different results. In any case, the ultimate goal of One Double Zero is to match istanbul accuracy, so the differences here are likely to become less and less important as One Double Zero continues to improve.

About performance differences: without a proper benchmark under some controlled environment and context, and multiple executions, they don't mean much. But it is likely that babel is slower in the end: instrumenting and running an instrumented code is more taxing than just executing the code and parsing the source files.

[SimenB]

thanks for this, looks really exciting!

Left a comment inline, but I think we should avoid the dependency and make it pluggable, but the code itself looks good!

[SimenB]

seems v1 stable is out?

[ericmorand]

Yes. Let me update the version. It should also make the coverage even closer to nyc.

[SimenB]

this shouldn't be here

[ericmorand]

I agree. It is likely an oversight, there is no dependency on these two packages. I will fix that.

[SimenB]

I don't think we should add a dependency that brings in typescript 🤔 I guess we could make the dep pluggable (i.e. optional peer dependency).

That would also handle the fact that it is minimum node v18 and Jest v30 supports Node 16

[ericmorand-sonarsource]

@SimenB,

Thanks a lot for the review.

typescript dependency was removed entirely.

I replaced one-double-zero dependency with one-double-zero-core, the API that fuels one-double-zero. It embeds its own parser, removing the need to depend on the typescript package. The API is stable and supports Node.js 16 (as you can see there), honoring Jest V30 commitments.

I rebased my PR on the head of main, and added a unit test to pass the coverage check.

Let me know what you think.

[AriPerkkio]

I've been checking this PR from time to time and thinking whether odz would fit into Vitest as well. We ended up taking v8-to-istanbul's API and adding AST-awareness to that.

It doesn't have the tradeoffs that odz has, and outputs ~100% identical coverage maps as Istanbul does. Also package size is much smaller (1.9MB vs 29kB).

[ericmorand]

That makes sense. I was about to investigate having odz fueled by v8-to-istanbul.

What parser did you end up using? The vast majority of odz size comes from the typescript parser, so if you were able to keep the size under control, i'm'very interested.

[AriPerkkio]

It's intentionally not tied to any parser. Testing tools like Jest and Vitest have their own transform pipelines, and they already have a parser that they can utilize. Similar to source maps, this tool accepts AST as argument. See https://www.npmjs.com/package/ast-v8-to-istanbul for the API.

I'm not sure how odz works, but you probably should not be using Typescript parser there. Instead it should be parsing the executed, possibly transpiled, Javascript code. Otherwise you'll need to bring in compilers for all transpiled languages, like Vue, Svelte, etc.

[ericmorand]

I like the idea, a lot. That would also solve the current PR issue: odz core would not include a parser at all.

I assume that the parse function excepted by v8-to-istanbul needs to return an estree-compatible ast, does it?

[AriPerkkio]

It's tested with 4 different parsers; 3 that output their own kind of estree, and Babel parser for Babel AST:

Passes all 195 tests^* of istanbul-lib-instrument. ✅

Test cases run against:

• vite/parseAst ✅
• acorn ✅
• oxc-parser ✅
• @babel/parser ✅

[SimenB]

👀