Total regression in codecov coverage report looks suspicious

[puzpuzpuz]

Got an automated coverage report on #35753. The reported coverage regression is estimated as -8.50% which is very suspicious, considering that the changes made by the PR are relatively small and tests are included. Impacted files list also includes mostly unrelated files. Is it possible that there is something wrong with codecov configuration or the service itself?

See #35753 (comment) and https://codecov.io/gh/nodejs/node/pull/35753 for more details

[richardlau]

#35670 (comment) is probably relevant.

cc @bcoe

[bcoe]

@puzpuzpuz @richardlau I'm not 100% sure the reason, but codecov.io seems to get a bit confused about the order that commits land to our main branch:

☝️ it's expected that coverage will be about 87% now, but it doesn't look like codecov.io has caught up yet.

I believe the notifications will stabilize once it catches up.

[puzpuzpuz]

@bcoe thanks for the clarification. It makes sense. I'm going to hide the codecov message and close this issue. Feel free to reopen it if you think it's necessary.

[bcoe]

see also: #35779

[bcoe]

@thomasrockhu codecov's comments on PRs seem to behave somewhat strangely on the Node.js project:

The HEAD of our default branch has been at 87% coverage for several commits now, but the coverage graph oscillates:

Similarly, PRs are getting reports of seemingly unrelated drops in coverage.

---

I wonder if the problem might be dates on commits, if we look at the last N commits to the Node.js project:

Date: Sat Oct 17 19:38:35 2020 +0200
Date: Fri Oct 16 11:50:00 2020 -0600
Date: Sat Aug 29 16:20:44 2020 +0900
Date: Fri Oct 23 23:53:41 2020 +0200
Date: Sat Oct 17 04:49:05 2020 -0700

Note that the ordering of commit dates does not match the order in which they're landed.

I'm not 100% sure why our commit history looks like this (CC: @Trott, @nodejs/build).

[Trott]

Note that the ordering of commit dates does not match the order in which they're landed.

That's the date the commit was authored, not merged. That sort of thing will happen on any large codebase with lots of people working on it. For example, here are the last 10 timestamps as of this writing for https://github.com/microsoft/vscode:

Date:   Sat Oct 24 06:21:24 2020 +0000
Date:   Fri Oct 23 18:37:39 2020 -0700
Date:   Fri Oct 23 14:38:26 2020 -0700
Date:   Fri Oct 23 12:10:43 2020 -0700
Date:   Fri Oct 23 16:13:16 2020 -0700
Date:   Fri Oct 23 16:09:24 2020 -0700
Date:   Fri Oct 23 14:47:55 2020 -0700
Date:   Fri Oct 23 14:32:42 2020 -0700
Date:   Sat Oct 24 00:08:57 2020 +0300
Date:   Fri Oct 23 13:38:15 2020 -0700

If you run git log --format=fuller, you will get separate AuthorDate and CommitDate fields. I'm not sure the commit dates will necessarily be in order either, though, if there are force pushes or perhaps other things. But I suspect they will more likely be in the order you expect. This is, I think, what the GitHub interface uses.

[bcoe]

If you run git log --format=fuller, you will get separate AuthorDate and CommitDate fields

@Trott thanks for clarifying, and providing the one liner, I figured it was something along these lines.

@Trott @thomasrockhu, looking at the list of recent commits on the default branch. It jumps out at me that codecov.io seems to be sorting by AuthorDate, whereas for providing an accurate comparison report on a PR, we would want to compare against the greatest CommitDate.

[Qard]

In #35779, the c8 step spat out a whole bunch of errors. Might be related: https://github.com/nodejs/node/runs/1299567071#step:8:1

[bcoe]

In #35779, the c8 step spat out a whole bunch of errors. Might be related

@Qard these are warnings are noisy, but I believe are unrelated (they relate to the file path not being properly calculated for files in the deps/ folder, this information is then dropped in the final reports.).

Your coverage output is what I expect ~87% is the combined coverage for our C++ and JS dependencies. What seems to be broken is the comparison between this and the default branch, which consistently indicates drops which aren't accurate.

[thomasrockhu]

@bcoe, I'm off this week, but I'll do a deep dive on Monday. My suspicion is that there might be something in the differing number of uploads between the ~87% covered commits (2 uploads) and the ~96% covered commits (1 upload).

I would just double-check this article in case anything there is useful.