tests: look for regressions when converting PDFs #1089
Draft
+285
−19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stores a reference version of the converted PDFs and diffs them when the newly converted document during the tests.
almet
changed the title
almet
changed the title
This is useful to reduce the computation time when creating PDF visual diffs. Here is a comparison of the same operation using python arrays and numpy arrays + lookups: Python arrays: ``` diff took 5.094218431997433 seconds diff took 3.1553626069980965 seconds diff took 3.3721952960004273 seconds diff took 3.2134646750018874 seconds diff took 3.3410625500000606 seconds diff took 3.2893160990024626 seconds ``` Numpy: ``` diff took 0.13705662599750212 seconds diff took 0.05698924000171246 seconds diff took 0.15319590600120137 seconds diff took 0.06126453700198908 seconds diff took 0.12916332699751365 seconds diff took 0.05839455900058965 seconds
Which makes it easier to inspect after CI run failures.
This leverages a new flag that can be passed during the tests to regenerate the PDFs if needed.
This is to see how a failing CI would look like.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I convert all the documents we have in our test suite and store them in a
referencefolder, and then compare this bit to bit, using pymupdf pixel buffers.We could use another diffing tool and tell it what's acceptable (if that exists), but I believe that in the end what matters most is the developer experience when we have an output that changes.
Two things come to mind:
Inspecting the diff
There are multiple tools that allow to do that, but I found that diff-pdf good and able to generate an output we can look at without having to run a GUI.
Produces this diff.pdf file for the changes between the 0.8.1 release and this commit.
Update the reference version
We should have a command to bump all the reference documents (or a specific one).
Status:
This PR currently only fails tests when there a change in the output. I plan to do the following:
Fixes #321