report assertion errors as 'failures' and other runtime exceptions as 'errors', consistent with Junit terminology #371
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
Currently, assertion failures (e.g., AssertionError) are being reported as errors, and unexpected exceptions (e.g., NullPointerException, IOException, etc.) are being reported as failures.
This is inverted relative to JUnit’s standard terminology.
JUnit convention
Failures → Assertion failures (AssertionError) indicating the test ran but produced an unexpected result.
Errors → Unexpected exceptions, infrastructure issues, or anything other than an AssertionError.
Reference:
JUnit 5 Legacy Reporter
Logs AssertionError as FAILURE and everything else as ERROR.
Reference: JUnit Legacy XMLReportWriter.java (line 444)
Fix
Update error classification logic to correctly map:
AssertionError → failure
Any other exception → error
Impact
Test reports will now align with JUnit expectations.
Downstream tooling and dashboards that parse JUnit XML or test results will correctly distinguish between test failures (assertion mismatches) and infrastructure errors (unexpected exceptions).