Skip to content

Commit

Permalink
Adapt Test Runner README.md to AI Ref Models Requirements (#107)
Browse files Browse the repository at this point in the history
Signed-off-by: Clayne Robison <[email protected]>
Co-authored-by: Tyler Titsworth <[email protected]>
  • Loading branch information
claynerobison and Tyler Titsworth authored Jun 12, 2024
1 parent e13b13e commit 75a2562
Showing 1 changed file with 46 additions and 5 deletions.
51 changes: 46 additions & 5 deletions test-runner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,29 @@ Test Runner is a simple automated testing framework for replicating and validati

These steps assume you are using a modern Linux OS like [Ubuntu 22.04](https://www.releases.ubuntu.com/jammy/) with [Python 3.10+](https://www.python.org/downloads/release/python-3100/).

### Install Test Runner

```bash
git clone https://github.com/intel/ai-containers
pip install -r ai-containers/test-runner/requirements.txt
```

### Create Tests Definition File

```bash
touch tests.yaml
```

### Run test file

```bash
python ai-containers/test-runner/test_runner.py -f tests.yaml
```

1. Install the python requirements file
2. Create a `tests.yaml` file in your project's test directory
2. Create a `tests.yaml` [file](#create-tests-definition-file) in your project's test directory
3. Add a [test](#test-definition) to the `tests.yaml` file
4. Run the Test Runner application command line interface to execute the test
4. [Run the Test Runner application](#run-test-file) command line interface to execute the test

### Test Definition

Expand Down Expand Up @@ -47,6 +59,15 @@ A test is defined as a set of commands to be executed along with their associate
> [!TIP]
> To find more information about each property, click on the attribute link to see it's OCI spec definition. To see the list of default values used for each property, see the [class definition](utils/test.py#L18).
#### Tests Definition Example

A sample minimal tests.yaml file will look like this:

```yaml
test_name:
cmd: something -you <want|the|test_runner> --todo
```
### Advanced Tests
There are two more complex properties that can be enabled to provide additional functionality to your tests.
Expand All @@ -55,20 +76,40 @@ There are two more complex properties that can be enabled to provide additional
Environment variables can be set for a test by providing a dictionary of key-value pairs in the `env` property of the test definition. These environment variables will be set when the test is executed using the [`expandvars`](https://pypi.org/project/expandvars/) library.

Here's an example of a test definition with environment variables:
Here's an example of a test definition using environment variables:

```yaml
test:
cmd: echo "${VAR1:-hello}"
```

Execution:

```bash
python test-runner/test_runner.py -f path/to/tests.yaml
VAR=world python test-runner/test_runner.py -f path/to/tests.yaml
VAR1=world python test-runner/test_runner.py -f path/to/tests.yaml
```

In the example above, the first output will be `hello`, and the second output will be `world`.

Modifying the above example to use the `env` dictionary option:

```yaml
test:
cmd: echo "${VAR1:-hello}"
env:
VAR1: "Yo!"
```

Execution:

```bash
python test-runner/test_runner.py -f path/to/tests.yaml
VAR1=world python test-runner/test_runner.py -f path/to/tests.yaml
```

In this instance, the default value of `hello` is overridden first by the value in the `env` variable dictionary, and then by the value passed on the test command line. Thus, the first output will be `Yo!`, and the second output will be `world`.

> [!TIP]
> It is a best practice to set a default value in the case where `VAR1` is not set with `:-<value>`, if the environment variable is not set for whatever reason, the test runner application will fail the test.

Expand Down Expand Up @@ -164,7 +205,7 @@ There are two modes for running the tests: baremetal and container.

#### Baremetal

In baremetal mode, the test command is executed directly on the host system. This mode is used when the img attribute is not provided in the test definition.
In baremetal mode, the test command is executed directly on the host system. This mode is implied when the `img` key/value pair is not provided in the test definition.

Example of a test definition for baremetal mode:

Expand Down

0 comments on commit 75a2562

Please sign in to comment.