Skip to content

Commit 1b81dff

Browse files
SanjulaGanepolaSanjulaGanepola
authored
Merge pull request #74 from codefori/docs/itest-cli
Add docs on `itest` CLI and automated test usage
2 parents d989b08 + 9ed67eb commit 1b81dff

File tree

9 files changed

+202
-46
lines changed

9 files changed

+202
-46
lines changed

‎.astro/astro/content.d.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,13 @@ declare module 'astro:content' {
327327
collection: "docs";
328328
data: InferEntrySchema<"docs">
329329
} & { render(): Render[".mdx"] };
330+
"developing/testing/cli.mdx": {
331+
id: "developing/testing/cli.mdx";
332+
slug: "developing/testing/cli";
333+
body: string;
334+
collection: "docs";
335+
data: InferEntrySchema<"docs">
336+
} & { render(): Render[".mdx"] };
330337
"developing/testing/configuring.mdx": {
331338
id: "developing/testing/configuring.mdx";
332339
slug: "developing/testing/configuring";

‎astro.config.mjs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,10 @@ export default defineConfig({
8888
label: 'Configuring Tests',
8989
link: 'developing/testing/configuring',
9090
},
91+
{
92+
label: 'CLI & Automated Tests',
93+
link: 'developing/testing/cli',
94+
},
9195
{
9296
label: 'Troubleshooting',
9397
link: 'developing/testing/troubleshooting',

‎src/content/docs/developing/testing/Testing_CLI.png‎

176 KB
Loading

‎src/content/docs/developing/testing/Testing_GitHubAction.png‎

133 KB
Loading

‎src/content/docs/developing/testing/Testing_TestProfiles.png‎

39.3 KB
Loading

‎src/content/docs/developing/testing/cli.mdx‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
---
2+
title: CLI & Automated Tests
3+
---
4+
5+
import { Aside, CardGrid, Card, Steps } from '@astrojs/starlight/components';
6+
7+
<div style="display: flex; gap: 5px; align-items: center;">
8+
<a href="https://www.npmjs.com/package/@ibm/itest">
9+
<img src="https://img.shields.io/npm/v/@ibm/itest.svg?label=version" alt="Version"/>
10+
</a>
11+
<a href="https://www.npmjs.com/package/@ibm/itest">
12+
<img src="https://img.shields.io/npm/dm/@ibm/itest.svg" alt="Installs"/>
13+
</a>
14+
</div>
15+
16+
The [IBM i Testing CLI](https://www.npmjs.com/package/@ibm/itest) (`itest`) is a companion to the [IBM i Testing](https://marketplace.visualstudio.com/items?itemName=IBM.vscode-ibmi-testing) VS Code extension, which allows you to run unit tests and generate code coverage results for RPG and COBOL programs on IBM i. With this CLI, you can run tests in your terminal on your local PC or in PASE on IBM i. This enables you to even script the execution of tests in a CI/CD pipeline.
17+
18+
![](Testing_CLI.png)
19+
20+
## Installation
21+
22+
This CLI can be used on your local PC or directly on IBM i. In either case, it can be installed globally via NPM:
23+
24+
```sh
25+
npm i -g itest
26+
```
27+
28+
## Usage
29+
30+
Once installed, you can run the `itest` command in your terminal along with various [options](#options) which configure what tests are run, the environment they are run in, how to run them, and what logs to generate. At a minimum, there are a few options that must be specified based on where you are running the command from and where your tests are located:
31+
32+
1. **Local PC + Local Tests**: Specify the `--ld, --local-directory [path]` option to point to a local directory containing your test files and the `--id, --ifs-directory [path]` option to point to the IFS directory on your IBM i where the project will be deployed.
33+
2. **Local PC + Tests in Library**: Specify the `--l, --library <library>` option to point to a library on your IBM i containing your test members.
34+
3. **IBM i + Tests in IFS**: Specify the `--id, --ifs-directory [path]` option to point to an IFS directory on your IBM i containing your test files.
35+
4. **IBM i + Tests in Library**: Specify the `--l, --library <library>` option to point to a library on your IBM i containing your test members.
36+
37+
Check out the [examples](#examples) section below for more details.
38+
39+
<Aside type="note">
40+
If you are running the command on your local PC, you will be prompted to enter your IBM i connection details. Alternatively, you can set the following environment variables: `IBMI_HOST`, `IBMI_USER`, and `IBMI_PASSWORD` (or `IBMI_PRIVATE_KEY` if using private key authentication).
41+
</Aside>
42+
43+
## Options
44+
45+
Listed below are all the available options for the `itest` command. For convenience, use the `itest --help` command to see all these options and their descriptions directly in your terminal.
46+
47+
| Flag | Description |
48+
|------------------------------------------------|------------------------------------------------------------------------------|
49+
| `--v, --version` | Display the version number |
50+
| `--ld, --local-directory [path]` | Local directory containing tests (preset: ".") |
51+
| `--id, --ifs-directory [path]` | IFS directory containing containing tests or deploy directory (preset: ".") |
52+
| `--l, --library <library>` | Library containing tests. |
53+
| `--sf, --source-files <sourceFiles...>` | Source files to search for tests. (default: ["QTESTSRC"]) |
54+
| `--ll, --library-list <libraries...>` | Libraries to add to the library list. |
55+
| `--cl, --current-library <library>` | The current library to use for the test run. |
56+
| `--cc, --code-coverage [ccLvl]` | Run with code coverage (choices: "*LINE", "*PROC", preset: "*LINE") |
57+
| `--ct, --coverage-thresholds <threshholds...>` | Set the code coverage thresholds (yellow and green). (default: ["60","90"]) |
58+
| `--sc, --skip-compilation` | Skip compilation |
59+
| `--sr, --summary-report [path]` | Save summary report (preset: "./.itest/summary-report.md") |
60+
| `--tr, --test-result [path]` | Save test result logs (preset: "./.itest/test-result.log") |
61+
| `--to, --test-output [path]` | Save test output logs (preset: "./.itest/test-output.log") |
62+
| `--co, --command-output [path]` | Save command output logs (preset: "./.itest/command-output.log") |
63+
| `--h, --help` | Display help for command |
64+
65+
<Aside type="caution">
66+
If you are using Git Bash on Windows, POSIX-style paths are automatically converted to Windows paths. This conversion can break the path passed to the `--ifs-directory` option. To prevent this, you have two options:
67+
1. Add an additional `/` to the start of any relative or absolute POSIX path:
68+
```sh
69+
itest --ld . --id //home/USER/builds/ibmi-company_system --ll RPGUNIT QDEVTOOLS --cl MYLIB --cc
70+
```
71+
2. Set the environment variable `MSYS_NO_PATHCONV` to `1`:
72+
```sh
73+
# Option A: Set the variable for your shell session
74+
export MSYS_NO_PATHCONV=1
75+
itest --ld . --id /home/USER/builds/ibmi-company_system --ll RPGUNIT QDEVTOOLS --cl MYLIB --cc
76+
77+
# Option B: Set the variable only for this command
78+
MSYS_NO_PATHCONV=1 itest --ld . --id /home/USER/builds/ibmi-company_system --ll RPGUNIT QDEVTOOLS --cl MYLIB --cc
79+
```
80+
</Aside>
81+
82+
## Examples
83+
84+
1. Run tests in local directory:
85+
```sh
86+
itest --ld . --id /home/USER/builds/ibmi-company_system --ll RPGUNIT QDEVTOOLS --cl MYLIB --cc
87+
```
88+
89+
*This first example command will deploy the contents of the current directory on your PC to the specified IFS directory, setup both the library list and current library, and finally run all tests with code coverage.**
90+
91+
2. Run tests in IFS directory:
92+
```sh
93+
itest --id /home/USER/builds/ibmi-company_system --ll RPGUNIT QDEVTOOLS --cl MYLIB --cc
94+
```
95+
96+
*This second example is the exact same as the first one except it is run on your on IBM i because no local directory option is specified.*
97+
98+
3. Run tests in library:
99+
```sh
100+
itest --l RPGUTILS --ll RPGUNIT QDEVTOOLS --cl RPGUTILS --cc
101+
````
102+
103+
*This third example is similar to the second except it will run tests in the specified library. This command can be run both on your local PC or IBM i.*
104+
105+
4. Run tests in library (more customized):
106+
```sh
107+
itest --l RPGUTILS --ll RPGUNIT QDEVTOOLS --cl RPGUTILS --cc *PROC --ct 70 95 --sr --tr --to --co
108+
````
109+
110+
*This fourth example is similar to the third one except it generates procedure coverage, customizes the yellow and green coverage thresholds, and saves all logs to the default log directory (`.itest`).*
111+
112+
## Logs
113+
114+
There are various logs that can be generated by the CLI to help you understand the results of your tests and diagnose any failures or issues. By default, all logs are saved to the `.itest` in the current working directory, but you can customize the path for each log via the corresponding option. The following logs can be generated:
115+
116+
* **Summary Report**: Markdown file that with a summary of the test results and code coverage results in a human readable format. This report is particularly useful when running tests in a CI/CD pipeline.
117+
* **Test Result**: Log file with a copy of the standard output of this command.
118+
* **Test Output**: Log file with information about which tests are discovered, what test configuration is used, all standard output/errors for any commands, and more. It will have the equivalent of what is shown in the [IBM i Testing output channel](../troubleshooting/#extension-logs) when running tests in VS Code. This log file is the most useful for diagnosing issues.
119+
* **Command Output**: Log file with standard output/errors and error codes for all commands run on the IBM i. It will have the equivalent of what is shown in the `Code for IBM i` output channel when running tests in VS Code.
120+
121+
## Automated Tests
122+
123+
Given that this CLI tool can be used to run tests from the command line, it can be easily integrated into any CI/CD pipeline. As a reference, use the below GitHub Action workflow snippet which uses the `itest` command to run tests and generate code coverage for the `my-sample-application` project:
124+
125+
```yaml
126+
- name: Run Unit Tests
127+
id: test
128+
run: |
129+
itest \
130+
--ld . \
131+
--id /home/$IBMI_USER/builds/my-sample-application \
132+
--ll MYAPP RPGUNIT QDEVTOOLS \
133+
--cl MYAPP \
134+
--cc --sr --tr --to --co
135+
env:
136+
IBMI_HOST: ${{ secrets.IBMI_HOST }}
137+
IBMI_USER: ${{ secrets.IBMI_USER }}
138+
IBMI_PASSWORD: ${{ secrets.IBMI_PASSWORD }}
139+
IBMI_SSH_PORT: ${{ secrets.IBMI_SSH_PORT }}
140+
```
141+
142+
This step can be used in a workflow that is triggered on pull requests to ensure that all tests pass and code coverage is acceptable before allowing the pull request to be merged. The markdown summary report can even be rendered in the PR comments making it easier for reviewers to see the results.
143+
144+
![](Testing_GitHubAction.png)
145+
146+
Check out the full [pr.yaml](https://github.com/IBM/ibmi-company_system/blob/main/.github/workflows/pr.yaml) file example in the [ibmi-company_system](https://github.com/IBM/ibmi-company_system) repository. This workflow showcases an end-to-end CI/CD pipeline that consists of build, impact analysis, unit tests, and code coverage!

‎src/content/docs/developing/testing/configuring.mdx‎

Lines changed: 42 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -189,50 +189,49 @@ Here is an example with all execution parameters set in the `testing.json` file:
189189
</TabItem>
190190
</Tabs>
191191

192-
<Aside type="note">
193-
The `wrapperCmd` parameter in the `RUCRTRPG`/`RUCRTCBL`/`RUCALLTST` specifications are not actual parameters of the RPGUnit commands, but rather are provided by the extension to allow wrappering them. This is primarily used when integrating RPGUnit with other vendor tools.
194-
195-
**Example `testing.json` configuration**:
196-
```json
197-
{
198-
"rpgunit": {
199-
"rucrtrpg": {
200-
"tgtCcsid": "*JOB",
201-
"dbgView": "*SOURCE",
202-
"cOption": [
203-
"*EVENTF"
204-
],
205-
"wrapperCmd": {
206-
"cmd": "MYLIB/MYCMD TSTCMD",
207-
"params": {
208-
"desc": "My description"
209-
}
192+
### Vendor Integration
193+
The `wrapperCmd` parameter in the `RUCRTRPG`/`RUCRTCBL`/`RUCALLTST` specifications are not actual parameters of the RPGUnit commands, but rather are provided by the extension to allow wrappering them. This is primarily used when integrating RPGUnit with other vendor tools.
194+
195+
**Example `testing.json` configuration**:
196+
```json
197+
{
198+
"rpgunit": {
199+
"rucrtrpg": {
200+
"tgtCcsid": "*JOB",
201+
"dbgView": "*SOURCE",
202+
"cOption": [
203+
"*EVENTF"
204+
],
205+
"wrapperCmd": {
206+
"cmd": "MYLIB/MYCMD TSTCMD",
207+
"params": {
208+
"desc": "My description"
210209
}
211-
},
212-
"rucalltst": {
213-
"wrapperCmd": {
214-
"cmd": "MYLIB/MYCMD CALLCMD",
215-
"params": {
216-
"desc": "My description"
217-
}
210+
}
211+
},
212+
"rucalltst": {
213+
"wrapperCmd": {
214+
"cmd": "MYLIB/MYCMD CALLCMD",
215+
"params": {
216+
"desc": "My description"
218217
}
219218
}
220219
}
221-
}
222-
```
223-
**Resolved compile command**:
224-
```js
225-
MYLIB/MYCMD TSTCMD(RPGUNIT/RUCRTRPG TSTPGM(MYUSER/TEMPDET)
226-
SRCSTMF('/home/MYUSER/builds/ibmi-company_system/qtestsrc/empdet.test.sqlrpgle')
227-
TGTCCSID(*JOB) DBGVIEW(*SOURCE) COPTION(*EVENTF) RPGPPOPT(*LVL2)
228-
INCDIR('''/home/MYUSER/builds/ibmi-company_system''')) DESC('My description')
229-
```
230-
231-
**Resolved execution command**:
232-
```js
233-
MYLIB/MYCMD CALLCMD(RPGUNIT/RUCALLTST TSTPGM(MYUSER/TEMPDET) ORDER(*REVERSE)
234-
DETAIL(*ALL) OUTPUT(*ALLWAYS) LIBL(*CURRENT) JOBD(*DFT) RCLRSC(*NO)
235-
XMLSTMF('/tmp/testing/vscode-ibmi-testing/RPGUNIT/TEMPDET_1751646914682.xml'))
236-
DESC('My description')
237-
```
238-
</Aside>
220+
}
221+
}
222+
```
223+
**Compile Command**:
224+
```js
225+
MYLIB/MYCMD TSTCMD(RPGUNIT/RUCRTRPG TSTPGM(MYUSER/TEMPDET)
226+
SRCSTMF('/home/MYUSER/builds/ibmi-company_system/qtestsrc/empdet.test.sqlrpgle')
227+
TGTCCSID(*JOB) DBGVIEW(*SOURCE) COPTION(*EVENTF) RPGPPOPT(*LVL2)
228+
INCDIR('''/home/MYUSER/builds/ibmi-company_system''')) DESC('My description')
229+
```
230+
231+
**Execution Command**:
232+
```js
233+
MYLIB/MYCMD CALLCMD(RPGUNIT/RUCALLTST TSTPGM(MYUSER/TEMPDET) ORDER(*REVERSE)
234+
DETAIL(*ALL) OUTPUT(*ALLWAYS) LIBL(*CURRENT) JOBD(*DFT) RCLRSC(*NO)
235+
XMLSTMF('/tmp/testing/vscode-ibmi-testing/RPGUNIT/TEMPDET_1751646914682.xml'))
236+
DESC('My description')
237+
```

‎src/content/docs/developing/testing/overview.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ The [IBM i Testing](https://marketplace.visualstudio.com/items?itemName=IBM.vsco
1919

2020
* 👨‍💻 **Run Tests**: Visualize and run tests suites out of local files or source members.
2121
* ✍️ **Generate Stubs**: Generate test stubs for individual test cases or an entire test suite.
22-
* ⚙️ **Configure Tests**: Configure parameters to compile (`RUCRTRPG`/`RUCRTCLB`) and run (`RUCALLTST`) tests.
22+
* ⚙️ **Configure Tests**: Configure parameters to compile (`RUCRTRPG`/`RUCRTCBL`) and run (`RUCALLTST`) tests.
2323
* 📋 **View Test Results**: View detailed test results along with inline failure/error messages.
2424
* 🎯 **Generate Code Coverage**: View line and procedure level code coverage results as an overlay in the editor.
2525

‎src/content/docs/developing/testing/running.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ From here you can hover on any test case, test suite, folder, library, or source
2323
<Steps>
2424
1. **Deployment *(Local Files Only)***: The contents of your workspace folder will be deployed to a directory in the IFS. The default deployment directory suggested will be of the following format: `/home/<user>/builds/<workspace-folder>/`. This is the same deployment process that takes place when you compile any local files via Code for IBM i.
2525
2. **Read Compile Configuration**: To customize the parameters used to compile your tests, a global and directory specific test configuration file will be read. Learn more about this in the [Compile Configuration](../configuring/#compile-configuration) section.
26-
3. **Compilation**: Given the compile configuration and tests selected for execution, the tests will be compiled via the `RUCRTRPG` or `RUCRTCLB` command. This step will be skipped for tests which have had no change since the last compilation.
26+
3. **Compilation**: Given the compile configuration and tests selected for execution, the tests will be compiled via the `RUCRTRPG` or `RUCRTCBL` command. This step will be skipped for tests which have had no change since the last compilation.
2727
4. **Execution**: The compiled test program will be executed via the `RUCALLTST` command. The parameters for this command are configured in the VS Code settings or the test configuration file. Learn more about this in the [Execution Configuration](../configuring/#execution-configuration) section.
2828
</Steps>
2929

@@ -56,7 +56,7 @@ The relevant files under test will also be highlighted in the editor where the e
5656

5757
## Test Profiles
5858

59-
The extension provides two kinds of test profiles (**Run** and **Coverage**) which represent different ways of executing tests. The standard profiles only compile the test if they have not already been compiled since loading them in. Variants of these profiles are also available that always re-compile the test (refer to the profiles with the `(Compile)` suffix).
59+
The extension provides two kinds of test profiles (**Run** and **Coverage**) which represent different ways of executing tests. The standard profiles only compile the test if they have not already been compiled since loading them in. Variants of these profiles are also available which can always re-compile the test (refer to the profiles with the `(Force Compile)` suffix) or skip compilation entirely (refer to the profiles with the `(Skip Compile)` suffix).
6060

6161
To run a set of tests with a specific profile:
6262

0 commit comments

Comments
 (0)