Skip to content

testing: show extra information on failure #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

andrewliebenow
Copy link
Contributor

No description provided.

@andrewliebenow
Copy link
Contributor Author

Old:

---- tr::tr_octal_above_one_byte_value stdout ----
thread 'tr::tr_octal_above_one_byte_value' panicked at plib/src/testing.rs:85:5:
assertion `left == right` failed
  left: "(Ł)"
 right: "Ł)"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

New:

---- tr::tr_octal_above_one_byte_value stdout ----
thread 'tr::tr_octal_above_one_byte_value' panicked at plib/src/testing.rs:229:9:
------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Test failed with 2 total failure types
--------------------------------------------------------------------------------

Command executed
--------------------------------------------------------------------------------
tr -d \501
--------------------------------------------------------------------------------

Failure 1 of 2: stdout differs from what was expected
--------------------------------------------------------------------------------

Actual
--------------------------------------------------------------------------------
(Ł)
--------------------------------------------------------------------------------

Expected
--------------------------------------------------------------------------------
Ł)
--------------------------------------------------------------------------------

Failure 2 of 2: stderr differs from what was expected
--------------------------------------------------------------------------------

Actual
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

Expected
--------------------------------------------------------------------------------
tr: warning: the ambiguous octal escape \501 is being interpreted as the 2-byte sequence \050, 1

--------------------------------------------------------------------------------


stderr, for diagnosing failure
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

@andrewliebenow
Copy link
Contributor Author

The new format might look a bit verbose, but I've repeatedly run up against:

  1. Not remembering what left/right correspond to (actual/expected)
  2. Not knowing if the comparison is in stdout or stderr
  3. Not having access to any additional information beyond that which is provided by the first failed assertion

@andrewliebenow andrewliebenow force-pushed the testing-add-failure-information branch 2 times, most recently from f81b4ac to 7bde6c9 Compare October 27, 2024 09:42
@jgarzik jgarzik added the enhancement New feature or request label Apr 2, 2025
@jgarzik jgarzik requested a review from Copilot April 2, 2025 21:25
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances the test failure reporting by displaying additional information when tests do not pass as expected. The changes include improved formatting with consistent markers, detailed messages for differences in stdout, stderr, and exit code, and a consolidated failure summary report.

"
stderr, for diagnosing failure{}
{MARKER}
{stderr_cow}
Copy link
Preview

Copilot AI Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code computes a truncated version of stderr as 'stderr_truncated' but later prints the full 'stderr_cow' instead. Consider printing 'stderr_truncated' in the failure message when its length differs from 'stderr_cow'.

Suggested change
{stderr_cow}
{}

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants