You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/client/testing/System-Testing-with-Squish-BDD.md
+67-20Lines changed: 67 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,19 +2,37 @@
2
2
3
3
## BDD and Gherkin Overview
4
4
5
-
Behaviour-Driven Development (BDD) is a software development process that builds on the lessons learnt from Test-Driven Development (TDD). BDD focuses on developing a common understanding of the behaviour of an application. More details can be found at https://cucumber.io/docs/bdd/ and https://www.agilealliance.org/glossary/bdd/.
5
+
Behaviour-Driven Development (BDD) is a software development process that builds on the lessons learnt from Test-Driven
6
+
Development (TDD). BDD focuses on developing a common understanding of the behaviour of an application. More details can
7
+
be found at https://cucumber.io/docs/bdd/ and https://www.agilealliance.org/glossary/bdd/.
6
8
7
-
BDD works by describing the behaviours of an application in a language that can be understood by developers and users, the Gherkin language https://cucumber.io/docs/gherkin/reference/, enabling a conversation on detailed application behaviour. This language helps to accurately describe behaviour and provides a way to carefully consider what to build. Gherkin is most effective for designing software when combined with other techniques such as low-fidelity prototypes.
9
+
BDD works by describing the behaviours of an application in a language that can be understood by developers and users,
10
+
the Gherkin language https://cucumber.io/docs/gherkin/reference/, enabling a conversation on detailed application
11
+
behaviour. This language helps to accurately describe behaviour and provides a way to carefully consider what to build.
12
+
Gherkin is most effective for designing software when combined with other techniques such as low-fidelity prototypes.
8
13
9
-
Gherkin also helps to automate testing. The steps in a `.feature` (gherkin) file can be linked to the code that runs the test step on the application. Squish supports this with its BDD tools https://www.froglogic.com/squish/features/bdd-behavior-driven-development-testing/. We have made use of this in the script generator for all its Squish testing. These tests can now act as documentation of the behaviour of the application and can be used to discuss the intricacies of the behaviour with scientists.
14
+
Gherkin also helps to automate testing. The steps in a `.feature` (gherkin) file can be linked to the code that runs the
15
+
test step on the application. Squish supports this with its BDD tools
16
+
https://www.froglogic.com/squish/features/bdd-behavior-driven-development-testing/. We have made use of this in the
17
+
script generator for all its Squish testing. These tests can now act as documentation of the behaviour of the
18
+
application and can be used to discuss the intricacies of the behaviour with scientists.
10
19
11
20
## Structure of our tests
12
21
13
-
Squish is split up into test suites, the script generator tests are all in the `suite_script_gen_tests`. The test cases in this suite are the `.feature` files that describe the behaviour in Gherkin. In general, we define one or more features in each file with the `Feature:` tag and each feature has a collection of scenarios denoted by the `Scenario:` tag. The feature will have a title and a description and a scenario will have a title that acts as its description.
22
+
Squish is split up into test suites, the script generator tests are all in the `suite_script_gen_tests`. The test cases
23
+
in this suite are the `.feature` files that describe the behaviour in Gherkin. In general, we define one or more
24
+
features in each file with the `Feature:` tag and each feature has a collection of scenarios denoted by the
25
+
`Scenario:` tag. The feature will have a title and a description and a scenario will have a title that acts as its
26
+
description.
14
27
15
-
Each scenario is made up of a set of `Given`, `When`, `Then` steps. These steps can take parameters including whole tables and can be ordered in lots of different ways. `Given` generally describes the state the application should be in before a user action, `When` describes a user action and `Then` describes verification of the state of the application after a user action. More details can be found at https://cucumber.io/docs/gherkin/ and in the Squish tutorials on https://www.froglogic.com/squish/features/bdd-behavior-driven-development-testing/.
28
+
Each scenario is made up of a set of `Given`, `When`, `Then` steps. These steps can take parameters including whole
29
+
tables and can be ordered in lots of different ways. `Given` generally describes the state the application should be in
30
+
before a user action, `When` describes a user action and `Then` describes verification of the state of the application
31
+
after a user action. More details can be found at https://cucumber.io/docs/gherkin/ and in the Squish tutorials on
The `Given`, `When` and `Then` steps are linked to code which is stored in the test suite resources steps area. For example, the `then_tooltip.py` file contains `Then` steps related to tooltip behaviour. Steps are defined like this:
34
+
The `Given`, `When` and `Then` steps are linked to code which is stored in the test suite resources steps area. For
35
+
example, the `then_tooltip.py` file contains `Then` steps related to tooltip behaviour. Steps are defined like this:
18
36
19
37
```python
20
38
@Then("the following actions have a tooltip |word|")
@@ -23,34 +41,63 @@ def step(context, status):
23
41
do_test_code()
24
42
```
25
43
26
-
This step takes a table as a parameter (accessed through `context.table`) and a word parameter (described by the decorator with `|word|` and passed to the step function as `status`). A step can have multiple decorators of `@Given`, `@When` and `@Then`. There are also more abilities such as using `From` sections and passing data between tests through the context variable - details at https://doc.froglogic.com/squish/latest/api.bdt.functions.html.
44
+
This step takes a table as a parameter (accessed through `context.table`) and a word parameter (described by the
45
+
decorator with `|word|` and passed to the step function as `status`). A step can have multiple decorators of
46
+
`@Given`, `@When` and `@Then`. There are also more abilities such as using `From` sections and passing data between
47
+
tests through the context variable - details at https://doc.froglogic.com/squish/latest/api.bdt.functions.html.
27
48
28
-
We do not generally edit the test.py file in the test case resources scripts. This script is created by squish and handles starting up tests and setting up the hooks. This could be used to start up the client instead of using `@OnFeatureStart` to speed up tests and enable us to better structure our features without concern for lengthening our tests.
49
+
We do not generally edit the test.py file in the test case resources scripts. This script is created by squish and
50
+
handles starting up tests and setting up the hooks. This could be used to start up the client instead of using
51
+
`@OnFeatureStart` to speed up tests and enable us to better structure our features without concern for lengthening
52
+
our tests.
29
53
30
54
## Scenario and Feature hooks
31
55
32
-
In the script section of the test suite resources, there is a file named `bdd_hooks.py`. This file contains a number of functions hooked into the tests to run at the beginning and end of features. For example:
56
+
In the script section of the test suite resources, there is a file named `bdd_hooks.py`. This file contains a number of
57
+
functions hooked into the tests to run at the beginning and end of features. For example:
33
58
34
59
```python
35
60
@OnFeatureStart
36
61
defhook(context):
37
62
do_hook_code()
38
63
```
39
64
40
-
Before the scenarios of a feature are run this hook is called by Squish. We utilise `@OnFeatureStart`, `@OnScenarioStart`, `@OnScenarioEnd` and `@OnFeatureEnd` to carry out the test setup and cleanup activities. More details at `6.19.10. Performing Actions During Test Execution Via Hooks` of https://doc.froglogic.com/squish/latest/api.bdt.functions.html.
65
+
Before the scenarios of a feature are run this hook is called by Squish. We utilise `@OnFeatureStart`,
66
+
`@OnScenarioStart`, `@OnScenarioEnd` and `@OnFeatureEnd` to carry out the test setup and cleanup activities.
67
+
More details at `6.19.10. Performing Actions During Test Execution Via Hooks` of
Have you already agreed to a gherkin description of the behaviour required? If so then add it to the test cases either as a new feature (there's a little button labelled BDD in the test cases section of the Squish GUI) or if your feature fits well into a feature already in the test suite then include the scenarios in there - this avoids running feature start and end code which restarts the client and lengthens the tests, though don't be afraid to add a new feature if it doesn't fit.
45
-
46
-
If you haven't already agreed on the behaviour, consider creating a design for the feature with gherkin and a low-fidelity prototype and reviewing it with other developers and script generator users - this depends on how major and well defined the feature is. If you're not sure, ask the team!
47
-
48
-
Now that you have your feature, scenario and steps laid out it's time to write the test code. Are any of the steps already defined in the test suite resources? They may not have the same name but they may have the same functionality - you can add a new `@Given`, `@When` or `@Then` decorator to the step or rename the step in the test case. Any steps that don't have an already defined test function for you will need to implement, these steps will be annotated by Squish in the test case.
49
-
50
-
To add the steps you want you can either code them directly by writing a test step function or you can record it. If you right-click on a feature or scenario in the test case and click `Record Missing Steps in Feature/Scenario` then the test will execute the steps it knows and then pauses on the steps it doesn't know to record any button clicks and do any verification steps as according to the squish recording tools - see the tutorial on https://www.froglogic.com/squish/features/recording-and-playback/. After recording, this will insert the step function into a file in the steps section of the test suite resources - often this is in a file that doesn't make sense so please move it to somewhere it does make sense.
51
-
52
-
Although the recording is useful, it often produces brittle step functions, please make use of the utilities in the global scripts area to improve the robustness of the test and see [here](System-Testing-with-Squish) for some hints, tips and gotchas.
72
+
Have you already agreed to a gherkin description of the behaviour required? If so then add it to the test cases either
73
+
as a new feature (there's a little button labelled BDD in the test cases section of the Squish GUI) or if your feature
74
+
fits well into a feature already in the test suite then include the scenarios in there - this avoids running feature
75
+
start and end code which restarts the client and lengthens the tests, though don't be afraid to add a new feature if
76
+
it doesn't fit.
77
+
78
+
If you haven't already agreed on the behaviour, consider creating a design for the feature with gherkin and a
79
+
low-fidelity prototype and reviewing it with other developers and script generator users - this depends on how major
80
+
and well defined the feature is. If you're not sure, ask the team!
81
+
82
+
Now that you have your feature, scenario and steps laid out it's time to write the test code. Are any of the steps
83
+
already defined in the test suite resources? They may not have the same name but they may have the same
84
+
functionality - you can add a new `@Given`, `@When` or `@Then` decorator to the step or rename the step in the test
85
+
case. Any steps that don't have an already defined test function for you will need to implement, these steps will be
86
+
annotated by Squish in the test case.
87
+
88
+
To add the steps you want you can either code them directly by writing a test step function or you can record it.
89
+
If you right-click on a feature or scenario in the test case and click `Record Missing Steps in Feature/Scenario` then
90
+
the test will execute the steps it knows and then pauses on the steps it doesn't know to record any button clicks and
91
+
do any verification steps as according to the squish recording tools - see the tutorial on
92
+
https://www.froglogic.com/squish/features/recording-and-playback/. After recording, this will insert the step function
93
+
into a file in the steps section of the test suite resources - often this is in a file that doesn't make sense so
94
+
please move it to somewhere it does make sense.
95
+
96
+
Although the recording is useful, it often produces brittle step functions, please make use of the utilities in the
97
+
global scripts area to improve the robustness of the test and see [here](System-Testing-with-Squish) for some hints, tips and gotchas.
53
98
54
99
## Running the tests
55
100
56
-
You can run the whole test suite with the play button located above the test cases next to the new BDD test case button. You can also run test cases with the run button next to the case in the case list. You can run features and scenarios by right-clicking and pressing run.
101
+
You can run the whole test suite with the play button located above the test cases next to the new BDD test case button.
102
+
You can also run test cases with the run button next to the case in the case list. You can run features and scenarios
When writing tests, it is important that the purpose and expected outcome of the test is clear from the name. This has several advantages:
3
+
When writing tests, it is important that the purpose and expected outcome of the test is clear from the name.
4
+
This has several advantages, for example:
4
5
5
6
- It helps other developers debug failing tests
6
7
- It helps clarify your own thoughts as to the purpose of the test
7
-
- It defines the code itself by explicitly stating the expected interface. Ideally you should be able to reconstruct a functional class entirely from its tests
8
-
- And more
8
+
- It defines the code itself by explicitly stating the expected interface. Ideally you should be able to reconstruct a
9
+
functional class entirely from its tests
9
10
10
-
It isn't expected that all tests will follow this guide, many tests pre-date this guide. Sometimes as well, these guidelines may prove overly restrictive or add unnecessary bulk to the test name. As ever, discretion is advised, but be clear why you're not following the guidelines if you choose not to. For example "'Initializes_ok' is fine as a test name because I'm just checking that I can initialize the class" would be a bad reason.
11
+
While new tests should aim to follow the conventions documented here, it is allowable to deviate in cases such as:
12
+
- Older tests which predate this naming convention; although if you are doing significant refactoring, consider updating
13
+
the old test names to match this convention.
14
+
- Adding tests to existing/external code which uses a different convention; prefer to adopt the convention in the
15
+
surrounding code for consistency.
16
+
- The convention is too restrictive for a particular test. Use discretion here; only deviate from the guidelines if you
17
+
have a strong reason to do so.
11
18
12
-
There are many test naming formats out there, each with pros and cons, and each with people who get far too passionate about their personal favourite. We have opted to go with the GIVEN_WHEN_THEN format.
19
+
There are many test naming formats out there, each with pros and cons. We have opted to go with the
20
+
[Given When Then format](https://martinfowler.com/bliki/GivenWhenThen.html). The given-when-then scheme is broadly
21
+
comparable with other common test naming & structuring schemes such as arrange-act-assert, but with different names.
GIVEN, WHEN, and THEN are in capitals with the rest of the test name in lower case, words are separated by underscores.
21
30
22
-
In some cases, there are no preconditions, or the preconditions are truly trivial (be wary of jumping to that conclusion though). In those cases given may be omitted.
31
+
In cases where there are no preconditions, or where the preconditions are very simple, the "given" section may be
32
+
omitted.
23
33
24
-
Where possible don't include the method being tested name in the test name as that could change over time.
34
+
Where possible, don't include the method being tested name in the test name, as that could change over time.
25
35
26
-
Given that java methods usually use CamelCase, it is useful to tell CheckStyle to ignore the name format add a warning suppression to the top of the class:
36
+
Given that java methods usually use CamelCase, it is useful to tell Checkstyle to ignore the name format add a warning
0 commit comments