Skip to content

Add script to validate code examples in sniff documentation #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

rodrigoprimo
Copy link
Collaborator

@rodrigoprimo rodrigoprimo commented May 30, 2025

This PR introduces a new script that verifies code examples in PHPCS sniff documentation. The script checks that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff, ensuring code examples correctly demonstrate the behavior of the sniff.

Noteworthy implementation decisions:

  • The script will be available in the Composer vendor/bin directory as phpcs-check-doc-examples.
  • To check for syntax errors in the code examples, the code uses token_get_all() with the flag TOKEN_PARSE. This means that PHP >= 7.0 is required to run this script while the rest of the repository requires PHP >= 5.4. A requirement check was added to phpcs-check-doc-examples to alert users if they are running PHP < 7.0, and a warning was added to the top of the file informing that phpcs-check-doc-examples must remain compatible with PHP 5.4 for the requirement check to work.
  • The script is unable to differentiate between multiple code examples in the same <code> block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same <code> block, there is at least one example that does trigger the sniff.
  • The tests for this script run on a separate GitHub action job as they need to be tested against a different set of PHP versions than the rest of the tests in this repository, and they need to be tested against multiple PHPCS versions and OSes. The DebugSniff tests need to be tested against multiple PHPCS versions, but not multiple OSes, while the tests for the other tools need to be tested against multiple OSes, but not multiple PHPCS versions.

How it works

The script:

  1. Finds XML documentation files for PHPCS sniffs based on the parameters passed in the command line. If no path is specified, it searches the directory where the script was executed and its sub-directories.
  2. Extracts code blocks marked as "Valid" or "Invalid" from these files
  3. Programatically calls PHPCS on these code blocks to verify that:
    • Valid examples don't trigger the sniff (no violations)
    • Invalid examples do trigger the sniff (have violations)
  4. Reports any discrepancies

How to use the script

  • Run it from the root directory of a PHPCS standard:
$ /path/to/script/phpcs-check-doc-examples

The script will search for all sniff documentation files and check their code examples. For other ways to run the script check the help page:

$ /path/to/script/phpcs-check-doc-examples --help

Known limitations

  • Currently, it is not possible to use this script to check PHPCompatibility documentation as there is no way to pass the testVersion to the PHPCS run. My suggestion is to open a separate issue to address this after this PR is merged.

Related issues

Closes #142

@rodrigoprimo rodrigoprimo force-pushed the check-xml-documentation-code-examples branch from 8298e09 to cae0df1 Compare May 30, 2025 20:13
@rodrigoprimo rodrigoprimo force-pushed the check-xml-documentation-code-examples branch 2 times, most recently from e00380e to 7d7801e Compare June 12, 2025 18:00
This commit moves the code related to checking color and formatting the help text to a new helper class. This way it can be reused when the new DocCodeExamples script is introduced.

Tests were added for the new HelpTextFormatter::format() method.
@rodrigoprimo rodrigoprimo force-pushed the check-xml-documentation-code-examples branch 2 times, most recently from a34f9e5 to a2d0a64 Compare June 12, 2025 18:59
This commit introduces a new script that verifies code examples in PHPCS sniff documentation.
The script checks that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger
the sniff, ensuring code examples correctly demonstrate the behavior of the sniff.

Noteworthy implementation decisions:
- The script will be available in the Composer vendor/bin directory as `phpcs-check-doc-examples`.
- To check for syntax errors in the code examples, the code uses `token_get_all()` with the flag TOKEN_PARSE. This means that PHP >= 7.0 is required to run this script while the rest of the repository requires PHP >= 5.4. A requirement check was added to `phpcs-check-doc-examples` to alert users if they are running PHP < 7.0, and a warning was added to the top of the file informing that `phpcs-check-doc-examples` must remain compatible with PHP 5.4 for the requirement check to work.
- The script is unable to differentiate between multiple code examples in the same <code> block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same <code> block, there is at least one example that does trigger the sniff.
- The tests for this script run on a separate GitHub action job as they need to be tested against a different set of PHP versions than the rest of the tests in this repository, and they need to be tested against multiple PHPCS versions and OSes. The DebugSniff tests need to be tested against multiple PHPCS versions but not multiple OSes, while the tests for the other tools need to be tested against multiple OSes but not multiple PHPCS versions.

How to use the script:

- Run it from the root directory of a PHPCS standard:

```
$ /path/to/script/phpcs-check-doc-examples
```

The script will search for all sniff documentation files and check their code examples. For other ways to run the script check the help page:

```
$ /path/to/script/phpcs-check-doc-examples --help
```
@rodrigoprimo rodrigoprimo force-pushed the check-xml-documentation-code-examples branch from a2d0a64 to 8e1daa5 Compare June 12, 2025 19:12
@rodrigoprimo
Copy link
Collaborator Author

@jrfnl, I made the changes that we discussed during our call earlier this week:

  • Fixed the PHPCS autoloading logic to be able to run the script when installed via Composer as a dependency of an external standard (tested with PHPCSExtra).
  • Fixed the issue with the PHPCSUtils caching mechanism that could cause the script to return incorrect results for some sniffs as an instance of DummyFile was being created without using a unique filename.
  • Added support for colorized output.
  • Added a summary report at the end of the script execution, including feedback when no errors were found.
  • Moved the check for libxml and dom to the script itself, and the rest of this repo doesn't require those two extensions.
  • Changed the script name to be shorter.
  • Fixed PHPCS violations in the script file itself (since the script has no extension, PHPCS is not automatically executed against it).

This PR is ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

CLI tool to check the code samples used in XML docs against the sniff
2 participants