.nyc_output/out.json is an empty object when using with @cypress/react

[chrisrzhou]

Logs and screenshots
Please provide debug logs by running Cypress from the terminal with DEBUG=code-coverage environment variable set, see the Debugging section of the README file.

Debug logs

  code-coverage reset code coverage in interactive mode +0ms
  code-coverage parsed sent coverage +32ms
  code-coverage wrote coverage file /Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output/out.json +1ms
  code-coverage parsed sent coverage +25ms
  code-coverage wrote coverage file /Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output/out.json +0ms
⚠️ file /Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output/out.json has no coverage information
Did you forget to instrument your web application? Read https://github.com/cypress-io/code-coverage#instrument-your-application
  code-coverage ⚠️ file /Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output/out.json has no coverage information +8s
  code-coverage ⚠️ file /Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output/out.json has no coverage information +1ms
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/chrisrzhou/Github/cypress-coverage-repo/coverage', reporter: [ 'lcov', 'clover', 'json', 'json-summary' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: false, 'temp-dir': '/Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output', tempDir: '/Users/chrisrzhou/Github/cypress-coverage-repo/.nyc_output', reportDir: '/Users/chrisrzhou/Github/cypress-coverage-repo/coverage' } +7ms
  code-coverage current working directory is /Users/chrisrzhou/Github/cypress-coverage-repo +0ms
  code-coverage after reporting, returning the report folder name /Users/chrisrzhou/Github/cypress-coverage-repo/coverage +40ms
  code-coverage Final coverage in /Users/chrisrzhou/Github/cypress-coverage-repo/coverage/coverage-final.json +0ms
  code-coverage There are 0 key(s) in /Users/chrisrzhou/Github/cypress-coverage-repo/coverage/coverage-final.json +0ms

Versions

• What is this plugin's version? @cypress/[email protected] (latest)
• What is Cypress version? [email protected], @cypress/[email protected], @cypress/[email protected] (see package.json in repo for more details).
• What is your operating system? Mac
• What is the shell? zsh
• What is the Node version? v12.22.1
• What is the NPM version? 6.14.12
• How do you instrument your application? Through the official docs' recommendation. See repo below for details.
• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Following the official docs, I can see window.__coverage__ object for the files that the test is touching.
  
• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Both folder and file exists, the contents is just the empty object {}.
• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files No
• Do you run Cypress tests in a Docker container? No, just locally

Describe the bug

Bug Report

Repro Link

https://github.com/chrisrzhou/cypress-code-coverage-issue-472

Context

It is a minimal example with the following requirements:

• Is an ESM module
  • i.e. "type": "module" is explicitly marked in the package.json
  • appropriate Cypress files need to be marked as .cjs in order for the tests to run.
    In cypress/plugins/index.cjs and fields in cypress.json.
• Uses component testing (i.e. @cypress/react and using open-ct and run-ct test runners).
• Uses code coverage (i.e. @cypress/code-coverage and babel-plugin-istanbul based on the official docs for instrumentation).

The requirements above are needed in the actual bug faced in the project, so the repo is capturing the minimal configuration of reproducing the same bug.

Reproduce

npm i
npm test

The test runner runs fine, and tests the code correctly.

Expected Behavior

Code coverage to output something meaningful in .nyc_output/out.json

Actual Behavior

.nyc_output/out.json is an empty object. The test runner complains that the code is not instrumented.

[chrisrzhou]

Related issues that I've tried out for the last few days but without success:

• Component tests with create-react-app (react-scripts-v4) don't instrument #461

To give context on the actual project where I first encountered the bug:

• I'm building a component library (does not actually use React).
• The code is authored as an ESM module (hence the repro repo being explicit about it in package.json and also having to use .cjs for some Cypress related setup to get the test runner working).
• I'm using @cypress/react and the component test runner for its delightful testing experience. That has been superb, it's just the last stretch of figuring out coverage that I'm hitting a wall. As a result, while React is not a dependency of the library I'm writing, it is a dev dependency for my tests. I definitely do not use create-react-app, and therefore have a custom .webpack.config.cjs and .babelrc in the cypress/ folder.

I don't fully understand why code coverage is not working, but some suspicions I have if they are helpful for the debugger:

• Maybe it's related to my package being ESM-only? (i.e. "type": "module" required in package.json).
• Maybe @cypress/react does not interoperate well with @cypress/code-coverage with ESM (as mentioned in the earlier point)?
• Maybe it's related to peer dependency versions that I have installed (I simply installed the latest).
• Maybe I'm completely missing something despite spending days iterating on variations from the official docs (which has a very simple setup!).

Thank you guys for the great work, and let me know if you need any other information if the repro repo is insufficient.

[twhoff]

Hi Chris, saw your comment.

I had a look at your code base and wonder if you might be missing babel-plugin-istanbul - I think you need to include it as a plugin as istanbul - from your webpack config (the one you are loading in the plugin index.js file?

Just a hunch - let me know how you go.

[chrisrzhou]

Hi @twhoff appreciate you taking the time to help here! 🤗

I have istanbul defined in .babelrc in https://github.com/chrisrzhou/cypress-code-coverage-issue-472/tree/main/cypress, and also have babel-plugin-istanbul installed as a dev dependency in package.json. I have to define my own webpack.config.cjs since this is not a CRA (it's a component library). The *.cjs cypress files are needed because the source code is written as ESM (i.e. type: "module" in package.json) and without specifying this, Cypress will complain.

[namespace-github]

Hello @chrisrzhou, hello all,

here is my solution, which from my point of view is a gap in the documentation.

In my case when using the new component tests, which was mentioned in the context of the question (open-ct or run-ct) window.__coverage__ was not evaluated from @cypress/code-coverage.

The reason was that, according to the documentation, import '@cypress/code-coverage/support'; only included in support/index.js. Cypress, however, uses in my environement instead /support/unit.js for component tests. Both support/index.js and support/unit.js files imports the support/commands.js.

I put the line there, then it works as well, because cypress can register the tasks in component tests.

[chrisrzhou]

Hey @namespace-github, thank you so much for getting back to me. I tried your solution and it does not seem to work. I might be missing something still and the online documentations have not been too helpful so really looking forward to your solution.

In my repro repo, I added in chrisrzhou/cypress-code-coverage-issue-472@f6e496d, a commit that adds ./support/unit.js with the import. I reran npm test and still observed .nyc_output/out.json to be an empty object.

If you have any insights or can reproduce a fix by cloning the repro repo, that would be sweet!
https://github.com/chrisrzhou/cypress-code-coverage-issue-472

[VaibhavP17]

Is there any update on this issue ? I am facing the same issue. While running the application using dev server on regular browser, I see window.__coverage__ object but while running the cypress test cases, I get the errors: cypress\.nyc_output\out.json has no coverage information and "Could not find any coverage information in your application by looking at the window coverage object. Did you forget to instrument your application? See [code-coverage#instrument-your-application](https://github.com/cypress-io/code-coverage#instrument-your-application) [@cypress/code-coverage]"

[kubijo]

Same here as @VaibhavP17 said…

[DuudeXX8]

Same...

[winjay-yu-awx]

Same here

[aautcq]

I have been struggling with an empty object in my .nyc_output/out.json, and I'd like to share the root cause in my code, it case it might be of use for someone else.

My code was correctly instrumented by istambul, and the Cypress code-coverage plugin was correctly set up as per the docs. I had an exclusion glob pattern in my Cypress config file, like this:

export default defineConfig({
  e2e: {
    specPattern: [
      '**/*.feature',
      '!**/ignored/**/*.feature',
    ],
  },
})

It turns out that

1. Even if this does not break Cypress, I should have been following the Cypress doc and use the excludeSpecPattern option instead to exclude test files
2. @cypress/code-coverage actually uses the specPattern option to ignore test files when computing the coverage, and the '!**/ignored/**/*.feature' pattern resulted in the plugin ignoring all my source files 🙃

TL;DR
Don't use negative patterns in specPattern, prefer excludeSpecPattern