Tests: move extra tests to the "risky" group

... as those are basically testing PHPCS itself and not the code in PHPCSUtils, so may start failing without notice.

Includes adding a couple of extra builds to make sure "risky" tests are run against both a variation of high/low PHP/PHPCS combinations.

Includes adding an extra build step and run the `compareWithPHPCS` group separately from the `xtra` group as the first should only be run against PHPCS `dev-master`, as it basically checks if functionality within PHPCSUtils is up-to-date with PHPCS `master`, while the second should be run against high/low supported PHPCS versions, as it basically tests functionality in PHPCS itself for which PHPCS does not have (sufficient) tests .

.github/workflows/test.yml

@@ -116,7 +116,22 @@ jobs:
             risky: true
             experimental: true
 
-          - php: '7.4'
+          - php: '5.4'
+            phpcs_version: '3.7.1'
+            risky: true
+            experimental: true
+
+          - php: '5.4'
+            phpcs_version: 'dev-master'
+            risky: true
+            experimental: true
+
+          - php: '8.1'
+            phpcs_version: '3.7.1'
+            risky: true
+            experimental: true
+
+          - php: '8.1'
             phpcs_version: 'dev-master'
             risky: true
             experimental: true
@@ -169,26 +184,36 @@ jobs:
           composer-options: --ignore-platform-reqs
 
       - name: Run the unit tests without caching (non-risky)
-        if: matrix.risky == false
+        if: ${{ matrix.risky == false }}
         run: vendor/bin/phpunit --no-coverage
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version == '4.0.x-dev' && '4.0.0' || matrix.phpcs_version }}
           PHPCSUTILS_USE_CACHE: false
 
       - name: Run the unit tests with caching (non-risky)
-        if: matrix.risky == false
+        if: ${{ matrix.risky == false }}
         run: vendor/bin/phpunit --testsuite PHPCSUtils --no-coverage --repeat 2
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version == '4.0.x-dev' && '4.0.0' || matrix.phpcs_version }}
           PHPCSUTILS_USE_CACHE: true
 
-      - name: Run the unit tests (risky)
-        if: ${{ matrix.risky }}
+      # Only run the "compare with PHPCS" group against dev-master as it ensures that PHPCSUtils
+      # functionality is up to date with `dev-master`, so would quickly fail on older PHPCS.
+      - name: Run the unit tests (risky, comparewithPHPCS)
+        if: ${{ matrix.risky && matrix.phpcs_version == 'dev-master' }}
         # "nothing" is excluded to force PHPUnit to ignore the <exclude> settings in phpunit.xml.dist.
         run: vendor/bin/phpunit --no-coverage --group compareWithPHPCS --exclude-group nothing
         env:
           PHPCS_VERSION: ${{ matrix.phpcs_version == '4.0.x-dev' && '4.0.0' || matrix.phpcs_version }}
 
+      # Run the "xtra" group against high and low PHPCS as these are tests safeguarding PHPCS itself.
+      - name: Run the unit tests (risky, xtra)
+        if: ${{ matrix.risky }}
+        # "nothing" is excluded to force PHPUnit to ignore the <exclude> settings in phpunit.xml.dist.
+        run: vendor/bin/phpunit --no-coverage --group xtra --exclude-group nothing
+        env:
+          PHPCS_VERSION: ${{ matrix.phpcs_version == '4.0.x-dev' && '4.0.0' || matrix.phpcs_version }}
+
 
   #### CODE COVERAGE STAGE ####
   # N.B.: Coverage is only checked on the lowest and highest stable PHP versions

Tests/Xtra/README.md

@@ -6,3 +6,5 @@ These tests safeguard functionality from PHP_CodeSniffer for which PHPCS itself
 
 In most cases, these type of tests should be pulled to PHP_CodeSniffer itself, but if testing this within the PHPCS native test framework would be difficult, tests can be placed here.
 Tests can also be added here for functionality for which PHPCS just doesn't have enough tests or temporarily, while waiting for a test PR to be merged in PHPCS itself.
+
+Note: these tests are run in CI in a separate test run called "risky" as these tests may start failing without notice due to changes in PHPCS itself.

phpunit.xml.dist

@@ -50,6 +50,8 @@
     <groups>
         <exclude>
             <group>compareWithPHPCS</group>
+            <!-- Extra tests are basically testing PHPCS itself, not PHPCSUtils. -->
+            <group>xtra</group>
         </exclude>
     </groups>