Definition of test instructions #1679
Conversation
|
ajanec01 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
ajanec01
requested review from
WilcoFiers,
Jym77,
annethyme,
carlosapaduarte,
colabottles,
daniel-montalvo,
EmmaJP and
HelenBurge
| > - Enter "technology" into the textbox; | ||
| > - Click the button. | ||
| > | ||
| > This `p` element with `role="status"` has an implicit `aria-live` value of "polite". |
There was a problem hiding this comment.
Suggest we put the description above the test instructions.
| @@ -151,6 +151,19 @@ The description should: | |||
|
|
|||
| For a detailed description on what to write test cases for see [test case design](../test-cases). For more details, see [ACT Rules Format: Test Cases](https://www.w3.org/TR/act-rules-format/#test-cases). | |||
|
|
|||
| ### Test instructions | |||
There was a problem hiding this comment.
We'll need to think about how to expose the test instructions in the testcases file. We may need to write regular expressions to parse the test instructions.
| > - Start the test when the window is loaded; | ||
| > - Enter "technology" into the textbox; | ||
| > - Click the button. | ||
| > |
There was a problem hiding this comment.
I suggest adding a "run the test" instruction. Here the test should be run at the end, but maybe there could be cases where several states should be tested along the way.
There was a problem hiding this comment.
After discussion in ACT-R meeting: we suggest having a final step that is something like "end the test after..."
| > **Test instructions**: | ||
| > - Start the test when the window is loaded; | ||
| > - Enter "technology" into the textbox; | ||
| > - Click the button. |
There was a problem hiding this comment.
| > - Click the button. | |
| > - Click the button. | |
| > - End the test after the status message is updated |
|
|
||
| Such instructions should be explicit to make sure that the desired state of a test case is obtained. The steps necessary to get the test case into the desired state need to be listed out. The instructions for evaluating the rule should not be included in the list of instructions to improve readability. For example, a test case description with a set of instructions could look like the following: | ||
|
|
||
| > **Test instructions**: |
There was a problem hiding this comment.
My main concern is the lack of "programmatic" association between the instructions and the test case itself.
With the current setup:
- As a tool vendor, I can look at the list of ACT test cases, see that, e.g., the case at URL https://act-rules.github.io/testcases/5f99a7/9a417788dfd68b83820b01deb71e427f8d8edc3a.html should Pass rule N°5f99a7 then run my tool at this URL and see if the rule corresponding rule is Passing or Failing.
- As an end user, I can double check implementation reports by doing the same. I only need to know what is the name of the rule in the tool.
With such instructions:
- As a tool vendor, I need to add some code on top of grabbing the test case and running the tool, in order to implement these instructions. That code might be wrong independently of the tool; or worse, the instructions might be unclear/ambiguous/subjective and my tool doesn't get the correct result only because the code I write for the instructions is wrong. Moreover, finding which test cases need some adhoc code is effectively a manual process (hence not scalable).
- As an end user, I can't really double check implementations either. Again, that need some adhoc code writing (or the tool vendor to also open the code they use to implement the instructions). And again, there might be discrepancies between the intention of the instruction, the code written by the tool vendor to produce a report, and the code written by the end user to double check a report…
|
I'm also considering whether the test instructions should (also) be included in the example itself. So that a tester that navigates to the example's page (without going through the ACT rules page) can still see them and act accordingly. See also #1661 (comment) where giving access to the computer's microphone seems fairly similar to a test instruction. |
| @@ -151,6 +151,19 @@ The description should: | |||
|
|
|||
| For a detailed description on what to write test cases for see [test case design](../test-cases). For more details, see [ACT Rules Format: Test Cases](https://www.w3.org/TR/act-rules-format/#test-cases). | |||
|
|
|||
| ### Test instructions | |||
|
|
|||
| At times, it may be necessary for the user to follow a set of instructions to get a test case into a state in which it can be evaluated against the expectation. It happens when the pass, failure, or inapplicability of a test case can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself. | |||
There was a problem hiding this comment.
| At times, it may be necessary for the user to follow a set of instructions to get a test case into a state in which it can be evaluated against the expectation. It happens when the pass, failure, or inapplicability of a test case can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself. | |
| At times, **pass**, **fail** and **inapplicable** can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself. The tester may need to follow a set of instructions to evaluate a test case in a specific state against the expectation. |
|
|
||
| At times, it may be necessary for the user to follow a set of instructions to get a test case into a state in which it can be evaluated against the expectation. It happens when the pass, failure, or inapplicability of a test case can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself. | ||
|
|
||
| Such instructions should be explicit to make sure that the desired state of a test case is obtained. The steps necessary to get the test case into the desired state need to be listed out. The instructions for evaluating the rule should not be included in the list of instructions to improve readability. For example, a test case description with a set of instructions could look like the following: |
There was a problem hiding this comment.
| Such instructions should be explicit to make sure that the desired state of a test case is obtained. The steps necessary to get the test case into the desired state need to be listed out. The instructions for evaluating the rule should not be included in the list of instructions to improve readability. For example, a test case description with a set of instructions could look like the following: | |
| Explicit, step-by-step instructions should be provided to ensure the desired state of a test case is obtained. For example, a test case description with a set of instructions could look like the following: |
The instructions for evaluating the rule should not be included in the list of instructions to improve readability.
I can't relate this sentence to any of these, other than "Don't use these instructions to improve rule readability". This seems like good advice, but probably not that relevant. Maybe I am missing something...
Defining "Test instructions" for rules that require the tester to get a test case into a specific state.
Need for Final Call: none
How to Review And Approve