docs: Add outline of unit testing recommendations #619

lundybernard · 2025-07-12T22:07:33Z

This PR contains an outline of a proposed Development Guide / Principles / Unit Testing Recommendations section.

I am opening this PR now to allow discussion of the outline, and key-points, before fleshing out the complete page.

📚 Documentation preview 📚: https://scientific-python-cookie--619.org.readthedocs.build/

henryiii · 2025-07-13T05:26:32Z

Over, looks good, just one overall issue that can be solved mostly by moving stuff around. I do not think we should push anyone toward unittest over pytest for unit tests. The reasons:

pytest is the most popular framework. unittest is mostly there so CPython can test itself, and has a few specific uses it's good for. It's particularly bad for scientific work.
Keeping the tests as easy to write as possible means more tests are written. unittest has a ton of boiler plate.
Fixtures, parametrization, etc. are all much better in pytest than in any xtest style framework.
Installing a dependency to run unit tests is fine. Developers run unit tests. Developers can install dependencies.

That last point is important: unit tests are for developers to develop the code. End users do not need to run unit tests. Once the units work correctly, then you don't need to verify that again.

However, I think we should instead emphasize a new category of tests, the one we mentioned (I don't remember your term, "smoke test" is what I thought of). That's a separate category from unit tests, and for that one you don't want dependencies; it's a great place to use unittest. So:

Integration tests: runs the entire app. Smaller projects often put them in /tests, larger projects might place them in a separate repository. Uses pytest or a command line runner. Can be slow.
Unit tests: verifies the various parts work individually. Usually in /tests. Uses pytest. Should be reasonable to run often by the developer.
Smoke test. Should be in the package, and runnable from the package. Uses unittest. Most common in compiled packages. Should verify the build works, and any architecture specific behavior works. Should be very fast.

So I'd basically recommend moving the unittest parts to a new section about smoke tests; the actual content is fine, just I think it needs reordering.

(And replace "smoke" with whatever you mentioned today, I've just forgotten what it was. Validity? Verification? ...)

docs/pages/principles/testing.md

henryiii · 2025-07-13T05:51:45Z

The only other thing I'd add is maybe a mention somewhere that this is guidelines for good design, but even poorly structured tests are better than no tests. I don't want to discourage someone from writing tests, but instead give them ways to improve their test design if they want to, and help as users scale up to larger projects.

henryiii · 2025-07-13T07:13:42Z

Here's an example I added to boost-histogram: scikit-hep/boost-histogram#1022

lundybernard · 2025-07-15T01:16:21Z

The term I use is "Diagnostic Tests",
The audience for diagnostic tests is people checking or troubleshooting live / production deployments who need to verify that an installation is correct.

FFR:
"Smoke tests" are a sub-set of test cases, that are run to quickly verify that key features of a system work.
We some times resort to creating a smoke test suite out of desperation, when the runtime of the full test suite becomes a burden.

docs/pages/principles/testing.md

henryiii · 2025-08-01T18:44:32Z

Let me know when it's ready for another review!

lundybernard · 2025-08-13T18:57:43Z

Thank you for your patience @henryiii,
I have a proof-reader who is helping me with revisions to the structure, and grammar/punctuation.
We should be ready for a full review soon.

I have one question we could use help with.
In the current version, we have a lot of detailed info about how to write unit tests, which I think would fit better in a Topical Guide. Should I go ahead and split those sections out into a new topical guide on unit tests now?

lundybernard · 2025-08-21T22:00:10Z

@henryiii this is ready for another review

henryiii · 2025-08-22T21:43:24Z

For now, keeping it on one page makes sense, as there's a discussion on restructuring going on anyway. I'm about to leave for a week-long vacation, so might not be able to get to this. I've got a couple of sections I'd like to work on a bit, and I'd like to compare this with the current pytest page to see if I can reduce overlap or cross-link.

One solution could be to merge in more-or-less the current state (I could do a quick review), then I could make PRs to update it later. Or we can wait till I get back - do you have a preference?

lundybernard · 2025-08-22T23:15:03Z

This went through several rounds of proof-reading and revision, so I'm happy to merge it, and continue working on it in the future through new Issues/PRs.

I don't mind if you prefer to wait and do a more thorough final review when you get back. Whatever works best for you.

henryiii

Added a few comments; some of it is fine for followup. Just the key stuff like renaming the src. import matters. Take it out of draft when you are ready!

henryiii · 2025-08-23T02:55:53Z

docs/pages/principles/testing.md

+```
+
+```python
+from src.lib import say_hello, dangerous_sideffects


Please don't name anything src. That's a folder name we use specially intending it to never show up in imports. It's not good to make people think it's okay to write "src" in their import statements. Select some other name. :)

henryiii · 2025-08-23T02:57:43Z

docs/pages/principles/testing.md

+namespace into an imported dependency's namespace, like so:
+
+```python
+def test_(mocker):


I can do it in a followup, but we should use monkeypatch where possible. Mocking should be used if you have to make a thing to monkeypatch in, but the patching process is much better with monkeypatch.

Also, I like keeping test_func or some similar name, I've seen people actually write def test(...) tests, which is really annoying, as it's not usually clear what it's testing, and it's hard to select with -k.

I changed test_ to test_func in those examples.

I think there is a lot of ground to cover regarding patching with mocks and static values, probably worth its own topical guide.
I'm not very familiar with the MonkeyPatch plugin, but I'm happy to learn more and incorporate it into these docs.

Personally, I prefer to patch out dependencies using autospec Mocks, so that the tests fail if the interface of the dependency changes. I'd like to work out the details in another issue or on the scientific-python discord.

henryiii · 2025-08-23T03:01:29Z

docs/pages/principles/testing.md

+Consider the benefits of refactoring your imports like so:
+
+```python
+from numpy import sum as np_sum, Array as NpArray


This should not be done except very special circumstances. Normally, you should monkeypatch the import and not use import ... from. There are some circumstances where you need to avoid this sort of import, like to avoid circular import issues. You can't import a unittest TestCase this way or it triggers the runner, in fact! Some projects strongly recommend avoiding from imports. There are special cases where you need this, for example if you need to patch in something but you it must only affect the local code, so it's good as a tip, but not the general recommendation for design. Especially renaming imports like this. :) Happy to address in a followup, though.

Also full imports are completely unambiguous. :)

Sure, I'd love to follow up on best practices for imports.
The main reason I prefer this style is that it makes patching, drop-in replacements, and refactoring a little easier. I agree that this may be an advanced-use tip, more appropriate for very polished code than code in the early stages of development.

side-note:
I checked importing a TestCase, because that sounds really weird... You can import a TestCase without executing it, unless you're using a test runner; python test.py does not execute an imported TestCase, but both pytest test.py and python -m unittest test.py do!

docs/pages/principles/testing.md

henryiii · 2025-08-23T03:07:37Z

docs/pages/principles/testing.md

+
+## Testing Edgecases
+
+While writing unit tests, you may be tempted to test edgecases. You may have a


I'd clarify that this is about long, trick things to test. Generally, you should always test edge cases like minimum, maximum, None, etc. I think the problem is the name - "edge cases" I think implies the wrong thing.

Maybe this needs a little more elaboration.

I think excessive edge case testing in unit tests is bad, we do want to test a reasonable subset of expected, and supported, valid inputs. This may be a good place to mention the TDD rule about not testing against invalid input until it actually becomes a problem. That said, I would not complain if someone showed me a huge test suite for a critical function that still runs in less than a second.

I'll work on the wording in this section.

I rewrote this section, and changed the title to "Extensive Input Testing".
The rewrite emphasizes readability, and recommend organizing edge case and extensive input testing in a way that preserves it.

henryiii · 2025-08-23T03:10:37Z

docs/pages/principles/testing.md

+`import numpy as np`. However, as we develop our unit tests, this can cause
+difficulty with mocking, and complicate refactoring.
+
+To patch out numpy.sum in your test, you either need to patch the _global_ numpy


Suggested change

To patch out numpy.sum in your test, you either need to patch the _global_ numpy

To patch out `numpy.sum` in your test, you either need to patch the _global_ numpy

henryiii · 2025-08-23T03:12:09Z

docs/pages/principles/testing.md

+difficulty with mocking, and complicate refactoring.
+
+To patch out numpy.sum in your test, you either need to patch the _global_ numpy
+module, which can have unintended side-effects, or specifically patch numpy.sum


FYI, monkey patching is isolated to just the function (and can even be reduced to just a line) so it normally works fine. It's only in special cases where something gets called multiple times and you only want to change your usage, which is pretty rare.

This needs some clarification. Yes, the lifetime of the patch is limited to the lifetime of the function, but the patch can target the namespace of the file-under-test, or the namespace of the dependency.

I can provide some examples to illustrate, but they are a bit long. This detail probably bares some detailed explanation, along with a discussion on when to use a Mock, vs when to substitute in a fake.

henryiii · 2025-08-23T03:12:27Z

docs/pages/principles/testing.md

+- Test files should be named `test_{file under test}.py`, so that test runners
+  can find them easily.
+
+- test\_.py files should match your source files (file-under-test) one-to-one,


Suggested change

- test\_.py files should match your source files (file-under-test) one-to-one,

- `test_*.py` files should match your source files (file-under-test) one-to-one,

Co-authored-by: Lauren Moore <[email protected]>

henryiii

Just had a quick last look, I think it's good! We can make PRs for followups as needed. Thanks for all the work on this!

henryiii reviewed Jul 13, 2025

View reviewed changes

docs/pages/principles/testing.md Show resolved Hide resolved

henryiii reviewed Jul 13, 2025

View reviewed changes