You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/pages/principles/testing.md
+86-59Lines changed: 86 additions & 59 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,36 +10,41 @@ parent: Principles
10
10
11
11
# Testing recommendations
12
12
13
-
In this guide, we will provide a roadmap and best-practices for creating test suites for python projects.
13
+
In this guide, we will provide a roadmap and best-practices for creating test
14
+
suites for python projects.
15
+
16
+
We will describe the most important types of test suites, the purposes they
17
+
serve and differences between them. They will be presented in OutSide -> In
18
+
order, which is our recommend approach. Starting with
19
+
[Public Interface tests](#user-interface-or-public-api-testing), which test your
20
+
code from the perspective of your users, focusing on the behavior of the public
21
+
interface and the Features that your project provides. Then we will cover
22
+
[Package Level Integration tests](#package-level-integration-tests), which test
23
+
that the various parts of your package work together, and work with the other
24
+
packages it depends on. Finally we will cover the venrable
25
+
[Unit Test](#unit-tests), which test the correctness of your code from a
26
+
perspective internal to your codebase, tests individual units in isolation, and
27
+
are optimized to run quickly and often.
28
+
29
+
These 3 test suites will cover the bulk of your testing needs and help get your
30
+
project to a reliable and maintainable state. We will also discuss some more
31
+
specialized and advanced types of test cases in our
32
+
[Taxonomy of Test Cases](#taxonomy-of-test-cases) section.
14
33
15
-
We will describe the most important types of test suites, the purposes they serve
16
-
and differences between them.
17
-
They will be presented in OutSide -> In order, which is our recommend approach.
18
-
Starting with [Public Interface tests](#user-interface-or-public-api-testing),
19
-
which test your code from the perspective of your users,
20
-
focusing on the behavior of the public interface and the Features that your project provides.
21
-
Then we will cover [Package Level Integration tests](#package-level-integration-tests), which test that the various parts of your package
22
-
work together, and work with the other packages it depends on.
23
-
Finally we will cover the venrable [Unit Test](#unit-tests), which test the correctness of your code from a perspective
24
-
internal to your codebase, tests individual units in isolation, and are optimized to run quickly and often.
25
-
26
-
These 3 test suites will cover the bulk of your testing needs and help get your project to a reliable
27
-
and maintainable state. We will also discuss some more specialized and advanced types of test cases
28
-
in our [Taxonomy of Test Cases](#taxonomy-of-test-cases) section.
34
+
## Advantages of Testing
29
35
36
+
- Trustworthy code: Well tested code, is code that you can trust to behave as
37
+
expected.
38
+
- Living Documentation: A good test is a form of documentation, which tells us
39
+
how the code is expected to behave, communicates the intent of the author, and
40
+
is validated every time the test is run.
41
+
- Preventing Failure: Tests provide safety against many ways code can fail, from
42
+
errors in implementation, to unexpected changes in upstream dependencies.
43
+
- Confidence when making changes: A thorough suite of tests allows developers to
44
+
add features, fix bugs, and refactor code, with a degree of confidence that
45
+
their changes do not break existing features, or cause unexpected
46
+
side-effects.
30
47
31
-
## Advantages of Testing
32
-
- Trustworthy code: Well tested code, is code that you can trust to behave as expected.
33
-
- Living Documentation: A good test is a form of documentation,
34
-
which tells us how the code is expected to behave, communicates the intent of the author,
35
-
and is validated every time the test is run.
36
-
- Preventing Failure: Tests provide safety against many ways code can fail, from errors in implementation,
37
-
to unexpected changes in upstream dependencies.
38
-
- Confidence when making changes: A thorough suite of tests allows developers to add features, fix bugs,
39
-
and refactor code, with a degree of confidence that their changes do not break existing features,
40
-
or cause unexpected side-effects.
41
-
42
-
43
48
## Any test case is better than none
44
49
45
50
When in doubt, write the test that makes sense at the time.
@@ -77,11 +82,11 @@ guide]({% link pages/guides/pytest.md %}).
77
82
78
83
- These test cases live outside of your source code.
79
84
- Test the code as you expect your users to interact with it.
80
-
- Keep these tests simple, and easily readable, so that they provide good documentation when
81
-
a user asks "how should I use this feature"
85
+
- Keep these tests simple, and easily readable, so that they provide good
86
+
documentation when a user asks "how should I use this feature"
82
87
- Focus on the supported use-case, and avoid extensive edge-case testing
83
-
(edge-case and exhaustive input testing will be handled in a separate test suite)
84
-
88
+
(edge-case and exhaustive input testing will be handled in a separate test
89
+
suite)
85
90
86
91
{: .highlight-title }
87
92
@@ -94,19 +99,23 @@ guide]({% link pages/guides/pytest.md %}).
94
99
95
100
## Project Level Integration Testing
96
101
97
-
The term "Integration Test" is unfortunately overloaded, and used to describe testing that various components
98
-
integrate with each other, at many levels of the system. These tests will loosly follow the "Detroit School" of test design.
102
+
The term "Integration Test" is unfortunately overloaded, and used to describe
103
+
testing that various components integrate with each other, at many levels of the
104
+
system. These tests will loosely follow the "Detroit School" of test design.
99
105
100
-
- Write tests which view the code from an outside-in perspective, like [Public Interface]() tests
106
+
- Write tests which view the code from an outside-in perspective, like
107
+
[Public Interface]() tests
101
108
- Avoid Mocks/Fakes/Patches as much as possible
102
-
- Test that the components of your code all work together (inner-package integration)
109
+
- Test that the components of your code all work together (inner-package
110
+
integration)
103
111
- Test that your code works with its dependencies (dependency integration)
104
112
105
-
These tests can be a good place for more extensive edge-case, and fuzzy input testing.
106
-
107
-
The intended audience for these tests developers working on the project, or debugging issues they encounter
108
-
as opposed to Public Interface tests, which should be helpful for users of the package.
113
+
These tests can be a good place for more extensive edge-case, and fuzzy input
114
+
testing.
109
115
116
+
The intended audience for these tests developers working on the project, or
117
+
debugging issues they encounter as opposed to Public Interface tests, which
118
+
should be helpful for users of the package.
110
119
111
120
## Unit Tests
112
121
@@ -407,40 +416,58 @@ production-like system, and run tests against it.
407
416
- Processing data from a pre-loaded test database
408
417
- Manual QA testing
409
418
410
-
411
419
### Other Kinds of Internal Tests
412
-
The thing that distinguishes Internal tests is their perspective on the code,
413
-
where External tests focus on the way users will interact with the package (or the public API)
414
-
and "avoid testing implementation details".
415
-
Internal tests exist to test that those critical implementation details work correctly.
420
+
421
+
The thing that distinguishes Internal tests is their perspective on the code,
422
+
where External tests focus on the way users will interact with the package (or
423
+
the public API) and "avoid testing implementation details". Internal tests exist
424
+
to test that those critical implementation details work correctly.
416
425
417
426
#### Testing Edgecases
418
-
While writing unit tests, you may be tempted to test edgecases. You may have a critical private function or algorithm, which is not part of the public API, so not a good candidate for External tesing, and you are concerned about many edgecases that you want to defend against using tests.
419
427
420
-
It is perfectly valid to write extensive edgecase testing for private code, but these tests should be kept separate from the unit test suite. Extensive edgecase testing makes tests long, and difficult to read (tests are documentation). They can slow down execution, we want unit tests to run first, fast, and often.
421
-
* Place them in separate files from unit tests, to improve readability
422
-
*[mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they can be run as a separate test suite, after your unit test pass
428
+
While writing unit tests, you may be tempted to test edgecases. You may have a
429
+
critical private function or algorithm, which is not part of the public API, so
430
+
not a good candidate for External tesing, and you are concerned about many
431
+
edgecases that you want to defend against using tests.
432
+
433
+
It is perfectly valid to write extensive edgecase testing for private code, but
434
+
these tests should be kept separate from the unit test suite. Extensive edgecase
435
+
testing makes tests long, and difficult to read (tests are documentation). They
436
+
can slow down execution, we want unit tests to run first, fast, and often.
437
+
438
+
- Place them in separate files from unit tests, to improve readability
439
+
-[mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they
440
+
can be run as a separate test suite, after your unit test pass
423
441
424
442
#### Fuzz Tests and other slow tests
425
-
Testing random input, using tools like Hypothesis, is similar to testing edge cases, but running these tests can take a very long time, and they can often be much more complex and difficult to read for new developers.
426
-
* Place them in their own test files
427
-
*[mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they can be run as a separate test suite, once all of the faster test suites have succeeded.
428
443
444
+
Testing random input, using tools like Hypothesis, is similar to testing edge
445
+
cases, but running these tests can take a very long time, and they can often be
446
+
much more complex and difficult to read for new developers.
447
+
448
+
- Place them in their own test files
449
+
-[mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they
450
+
can be run as a separate test suite, once all of the faster test suites have
451
+
succeeded.
429
452
430
453
## Diagnostic Tests
431
454
432
455
Diagnostic tests are used to verify the installation of a package. They should
433
456
be runable on production systems, like when we need to ssh into a live server to
434
457
troubleshoot problems.
435
458
436
-
A diagnostic test suite may contain any combination of tests you deem pertinent. You could include all the unit tests, or a specific subset of them. You may want to include some integration tests, and feature tests.
437
-
Consider them Smoke Tests, a select sub-set of tests, meant to catch critical errors quickly, not perform a full system check of the package.
438
-
439
-
* Respect the user's environment!
440
-
* Diagnostic tests should not require additional dependencies beyond what the package requires.
441
-
* Do not create files, alter a database, or change the state of the system
442
-
* Run quickly, select tests that can be run in a few moments
443
-
* provide meaningful feedback
459
+
A diagnostic test suite may contain any combination of tests you deem pertinent.
460
+
You could include all the unit tests, or a specific subset of them. You may want
461
+
to include some integration tests, and feature tests. Consider them Smoke Tests,
462
+
a select sub-set of tests, meant to catch critical errors quickly, not perform a
463
+
full system check of the package.
464
+
465
+
- Respect the user's environment!
466
+
- Diagnostic tests should not require additional dependencies beyond what the
467
+
package requires.
468
+
- Do not create files, alter a database, or change the state of the system
469
+
- Run quickly, select tests that can be run in a few moments
0 commit comments