chore: Single-shot benchmarking + continuous benchmarking #183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aborgna-q
changed the title
aborgna-q
force-pushed
the
ab/iai-benches
branch
from
fc7f000 to
d7e9605
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #183 +/- ##
=======================================
Coverage 83.27% 83.27%
=======================================
Files 24 24
Lines 6117 6117
Branches 6117 6117
=======================================
Hits 5094 5094
Misses 947 947
Partials 76 76 ☔ View full report in Codecov by Sentry. |
aborgna-q
force-pushed
the
ab/iai-benches
branch
8 times, most recently
from
f0fbe46 to
4a85932
Compare
aborgna-q
force-pushed
the
ab/iai-benches
branch
from
4a85932 to
0ea41d2
Compare
aborgna-q
force-pushed
the
ab/iai-benches
branch
from
0dbf375 to
7437319
Compare
Bencher Report
aborgna-q
force-pushed
the
ab/iai-benches
branch
2 times, most recently
from
ebb8365 to
ac2480a
Compare
aborgna-q
force-pushed
the
ab/iai-benches
branch
from
ac2480a to
0748032
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Setups
iai-callgrind-based benchmarks for single-shot benchmarking (by usingcallgrindinternally).Cleans up the benchmark cases and adds some structure so they can be executed both with
iai-callgrindandcriterion, depending on whether we want to measure instruction count or time.They can be called separatedly,
See DEVELOPMENT.md for instructions.
The instruction count benchmarks are now uploaded to bencher.dev, so we get an historical comparison of the performance and a CI check can alert about regressions.
I believe the "No thresholds found" error in this PR will go away once this gets run in
main.The service choice was mainly between bencher.dev and codspeed.io . I choose the former since it supports single-shot benchmarks natively.
See this issue in
ratatuiwhere the Bencher maintainer discusses some differences, ratatui/ratatui#1092.