Skip to content

Definition of test instructions #1679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: develop
Choose a base branch
from
+14 −0
Draft

Definition of test instructions #1679

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
93b502c
defining test instructions
Aug 3, 2021
a834765
adding the list condition
Aug 3, 2021
cb5dadc
Update spelling-ignore.yml
ajanec01 Aug 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions __tests__/spelling-ignore.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@
- Namespaced
- namespaceURI
- testability
- inapplicability

# Parts of Unicode
- 000A
Expand Down
13 changes: 13 additions & 0 deletions pages/design/rule-design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,19 @@ The description should:

For a detailed description on what to write test cases for see [test case design](../test-cases). For more details, see [ACT Rules Format: Test Cases](https://www.w3.org/TR/act-rules-format/#test-cases).

### Test instructions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to think about how to expose the test instructions in the testcases file. We may need to write regular expressions to parse the test instructions.


At times, it may be necessary for the user to follow a set of instructions to get a test case into a state in which it can be evaluated against the expectation. It happens when the pass, failure, or inapplicability of a test case can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
At times, it may be necessary for the user to follow a set of instructions to get a test case into a state in which it can be evaluated against the expectation. It happens when the pass, failure, or inapplicability of a test case can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself.
At times, **pass**, **fail** and **inapplicable** can only be determined after a change of the state of a test case that is triggered by the user's action and not the content itself. The tester may need to follow a set of instructions to evaluate a test case in a specific state against the expectation.


Such instructions should be explicit to make sure that the desired state of a test case is obtained. The steps necessary to get the test case into the desired state need to be listed out. The instructions for evaluating the rule should not be included in the list of instructions to improve readability. For example, a test case description with a set of instructions could look like the following:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Such instructions should be explicit to make sure that the desired state of a test case is obtained. The steps necessary to get the test case into the desired state need to be listed out. The instructions for evaluating the rule should not be included in the list of instructions to improve readability. For example, a test case description with a set of instructions could look like the following:
Explicit, step-by-step instructions should be provided to ensure the desired state of a test case is obtained. For example, a test case description with a set of instructions could look like the following:

The instructions for evaluating the rule should not be included in the list of instructions to improve readability.

I can't relate this sentence to any of these, other than "Don't use these instructions to improve rule readability". This seems like good advice, but probably not that relevant. Maybe I am missing something...


> **Test instructions**:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is the lack of "programmatic" association between the instructions and the test case itself.

With the current setup:

With such instructions:

  • As a tool vendor, I need to add some code on top of grabbing the test case and running the tool, in order to implement these instructions. That code might be wrong independently of the tool; or worse, the instructions might be unclear/ambiguous/subjective and my tool doesn't get the correct result only because the code I write for the instructions is wrong. Moreover, finding which test cases need some adhoc code is effectively a manual process (hence not scalable).
  • As an end user, I can't really double check implementations either. Again, that need some adhoc code writing (or the tool vendor to also open the code they use to implement the instructions). And again, there might be discrepancies between the intention of the instruction, the code written by the tool vendor to produce a report, and the code written by the end user to double check a report…

> - Start the test when the window is loaded;
> - Enter "technology" into the textbox;
> - Click the button.
Copy link
Member

@WilcoFiers WilcoFiers Aug 26, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> - Click the button.
> - Click the button.
> - End the test after the status message is updated

>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding a "run the test" instruction. Here the test should be run at the end, but maybe there could be cases where several states should be tested along the way.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussion in ACT-R meeting: we suggest having a final step that is something like "end the test after..."

> This `p` element with `role="status"` has an implicit `aria-live` value of "polite".
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest we put the description above the test instructions.


## Listed conditions

Applicability and expectations, often have multiple conditions that need to be met for something to be applicable / passed. When there are more than three conditions, listing them all in the running text can be difficult to read and understand. Instead, use a bullet-point list following this format:
Expand Down