Skip to content

Conversation

Jiloc
Copy link
Contributor

@Jiloc Jiloc commented Oct 1, 2025

Description

This PR is about simplifying the process of writing and updating our consensus tests.

the current drill: write a test, run it, see it fail, extract the data and then manually recreate the ExpectedResult struct from the terminal back into your code. It's easy to mess up.

This change introduces snapshot testing using a crate called insta.

Long story short: with this, we don't have to write the ExpectedResult by hand!

So, how does it work

Instead of stuffing the ExpectedResult into our test files, insta saves it automatically into a separate .snap file (or inline, I have an example). These new files live in a snapshots/ directory and we check them into Git. They become the "source of truth" for what the output should be.

Pros:

  • Writing the tests should be easier. You just write the test setup and let insta generate the correct output for you.
  • Our test files look cleaner. They're all about the logic, not giant blocks of data.
  • Easy Diffs for Reviews: When a change breaks consensus, the git diff on the .snap file will show exactly what changed in the output. Reviewing consensus-breaking changes should be easier.
  • Updating the fixtures on consensus-breaking releases can be automated

How do I use this?

Writing a new test:

  1. Set up your ConsensusTestVector like before, but just... don't add an expected_result.
  2. Run the test logic: let result = ConsensusTest::new(...).run();
  3. Add this magic line at the end: insta::assert_ron_snapshot!(result);
  4. Run cargo insta review to create the snapshot for you:
    cargo insta review
  5. Run the test
  6. Just commit the new .snap file along with your code.

Updating a test after you broke something (on purpose!):

  1. Run cargo test. It'll fail, and insta will print a beautiful diff showing you exactly what's different.
  2. If the new output is correct and expected, just run cargo insta review again.
  3. insta will automatically update the snapshot file for you.
  4. Commit your code and the updated snapshot. Your PR will now have a clear "before and after" of the consensus change.

Applicable issues

  • fixes #

Additional info (benefits, drawbacks, caveats)

Checklist

  • Test coverage for new or modified code paths
  • Changelog is updated
  • Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)
  • New clarity functions have corresponding PR in clarity-benchmarking repo
  • New integration test(s) added to bitcoin-tests.yml

@Jiloc Jiloc changed the title Feat/aac tests with insta feat: Improve consensus test ergonomics with snapshot testing Oct 1, 2025
@Jiloc Jiloc requested review from fdefelici, jferrant and kantai October 3, 2025 16:11
@Jiloc Jiloc self-assigned this Oct 3, 2025
@Jiloc Jiloc moved this to Status: In Review in Stacks Core Eng Oct 3, 2025
@Jiloc Jiloc added aac Avoiding Accidental Consensus aac-testing Avoiding Accidental Consensus Testing Specific Task labels Oct 3, 2025
@Jiloc Jiloc added this to the 3.2.0.0.2 milestone Oct 3, 2025
@Jiloc Jiloc marked this pull request as ready for review October 3, 2025 16:34
@Jiloc Jiloc requested review from a team as code owners October 3, 2025 16:34
@github-project-automation github-project-automation bot moved this from Status: In Review to Status: 💻 In Progress in Stacks Core Eng Oct 7, 2025
@jferrant jferrant added this pull request to the merge queue Oct 7, 2025
Merged via the queue into stacks-network:develop with commit 33d7113 Oct 7, 2025
297 of 301 checks passed
@github-project-automation github-project-automation bot moved this from Status: 💻 In Progress to Status: ✅ Done in Stacks Core Eng Oct 7, 2025
Copy link

codecov bot commented Oct 7, 2025

Codecov Report

❌ Patch coverage is 93.52518% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.83%. Comparing base (b5e3baf) to head (f34fde5).
⚠️ Report is 13 commits behind head on develop.

Files with missing lines Patch % Lines
stackslib/src/chainstate/tests/consensus.rs 93.52% 9 Missing ⚠️

❌ Your project status has failed because the head coverage (65.83%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff              @@
##           develop    #6555       +/-   ##
============================================
- Coverage    79.88%   65.83%   -14.05%     
============================================
  Files          568      568               
  Lines       347203   347125       -78     
============================================
- Hits        277377   228547    -48830     
- Misses       69826   118578    +48752     
Files with missing lines Coverage Δ
stackslib/src/chainstate/tests/consensus.rs 84.16% <93.52%> (-7.21%) ⬇️

... and 345 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5e3baf...f34fde5. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aac Avoiding Accidental Consensus aac-testing Avoiding Accidental Consensus Testing Specific Task

Projects

Status: Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

AAC Testing: Develop Integration Test Harness for append_block in stackslib

3 participants