False positives from tests/test_self.py

[greg-ww]

I noticed that some tests in test_self.py fail to validate the behavior they are meant to verify due to the way with pytest.raises(SystemExit): is being used blocking the asserts from ever being run.

For example in tests/test_self.py::TestCallbackOptions::test_errors_only you can change

    def test_errors_only() -> None:
        """Test the --errors-only flag."""
        with pytest.raises(SystemExit):
            run = Run(["--errors-only"])
            assert run.linter._error_mode

to

    def test_errors_only() -> None:
        """Test the --errors-only flag."""
        with pytest.raises(SystemExit):
            run = Run(["--an-invalid-option"])
            assert run.linter._error_mode

and it still passes perfectly happily but when you look at the output there were errors

$ pytest tests/test_self.py::TestCallbackOptions::test_errors_only -s
============================================================= test session starts =============================================================
platform win32 -- Python 3.12.3, pytest-8.3.5, pluggy-1.5.0
rootdir: D:\Repos\pylint
configfile: pyproject.toml
plugins: anyio-4.9.0, langsmith-0.4.1
collected 1 item                                                                                                                               

tests\test_self.py usage: pylint [options]
pylint: error: Unrecognized option found: an-invalid-option
.

============================================================== 1 passed in 0.12s ============================================================== 

You can even just change the assert line to assert False and it will still pass every time.

This affects multiple tests in test_self.py I confirmed this behavior in test_errorrs_only and test_verbose.

[Pierre-Sassoulas]

Thank you, nice catch !