Simplify fuzzing coverage (CI) scripts somewhat #3925
Conversation
|
👋 Thanks for assigning @joostjager as a reviewer! |
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
.github/workflows/build.yml
Outdated
| ./contrib/generate_fuzz_coverage.sh --output-dir `pwd` | ||
| # Could you use this to fake the coverage report for your PR? Sure. | ||
| # Will anyone be impressed by your amazing coverage? No | ||
| # Maybe if codecov wasn't broken we wouldn't need to do this... | ||
| bash <(curl -s https://codecov.io/bash) -f fuzz-codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57" |
There was a problem hiding this comment.
There appears to be a path mismatch in the coverage file handling. The script is called with --output-dir \pwd`which will create the coverage file at`pwd`/fuzz-codecov.json, but the codecov upload command is looking for fuzz-codecov.json` in the current directory without a path prefix.
To fix this, either:
- Add the
--output-codecov-jsonflag to the script call, or - Update the file path in the bash command to
\pwd`/fuzz-codecov.json`
This will ensure the codecov upload can find the generated coverage file.
| ./contrib/generate_fuzz_coverage.sh --output-dir `pwd` | |
| # Could you use this to fake the coverage report for your PR? Sure. | |
| # Will anyone be impressed by your amazing coverage? No | |
| # Maybe if codecov wasn't broken we wouldn't need to do this... | |
| bash <(curl -s https://codecov.io/bash) -f fuzz-codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57" | |
| ./contrib/generate_fuzz_coverage.sh --output-dir `pwd` | |
| # Could you use this to fake the coverage report for your PR? Sure. | |
| # Will anyone be impressed by your amazing coverage? No | |
| # Maybe if codecov wasn't broken we wouldn't need to do this... | |
| bash <(curl -s https://codecov.io/bash) -f `pwd`/fuzz-codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57" |
Spotted by Diamond
Is this helpful? React 👍 or 👎 to let us know.
|
👋 The first review has been submitted! Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer. |
| --output-codecov-json) | ||
| OUTPUT_CODECOV_JSON=1 | ||
| shift 1 | ||
| ;; | ||
| *) | ||
| echo "Unknown option: $1" | ||
| echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY]" |
There was a problem hiding this comment.
The usage information is missing the newly added --output-codecov-json flag. Please update the help text to include this option:
echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY] [--output-codecov-json]"This will help users understand all available command-line options.
| echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY]" | |
| echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY] [--output-codecov-json]" |
Spotted by Diamond
Is this helpful? React 👍 or 👎 to let us know.
There was a problem hiding this comment.
Its a CI-only flag so not really worth documenting.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3925 +/- ##
==========================================
- Coverage 89.01% 88.99% -0.03%
==========================================
Files 166 166
Lines 119692 121374 +1682
Branches 119692 121374 +1682
==========================================
+ Hits 106546 108014 +1468
- Misses 10740 10952 +212
- Partials 2406 2408 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
TheBlueMatt
force-pushed
the
2025-07-3718-followups
branch
from
95df8a0 to
427d984
Compare
We recently introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. Here, we simplify some of the scripts that do so, especially removing the second `ci-fuzz.sh` file we added to the repo.
codecov recommends using the new CLI uploader rather than their (deprecated) bash uploader (which we use). It also appears to have a fail-on-error mode which we'd prefer over the bash one which silently swallows errors.
TheBlueMatt
force-pushed
the
2025-07-3718-followups
branch
from
427d984 to
08795f2
Compare
|
Okay, this seems to work, and codecov's report includes coverage for lines that are |
| # Could you use this to fake the coverage report for your PR? Sure. | ||
| # Will anyone be impressed by your amazing coverage? No | ||
| # Maybe if codecov wasn't broken we wouldn't need to do this... | ||
| bash <(curl -s https://codecov.io/bash) -f target/codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57" | ||
| ./codecov --verbose upload-process --disable-search --fail-on-error -f target/codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57" |
There was a problem hiding this comment.
From docs:
./codecov --verbose upload-process --disable-search -t ${{ secrets.CODECOV_TOKEN }} -n 'job-name'-${{ github.run_id }} -F <flag name> -f <covarage report file name>
Don't we need the job name?
And the flag, is that useful to keep apart fuzz and normal test coverage?
We recently introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. Here, we simplify some of the scripts that do so, especially removing the second
ci-fuzz.shfile we added to the repo.