Skip to content

Proposal for "ATF Script Runner" #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Proposal for "ATF Script Runner" #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
43d670d
Initial version
dboltovskyi Nov 7, 2017
852bb8e
Update according to comments in review
dboltovskyi Nov 7, 2017
4dee3fc
Add component diagram
dboltovskyi Dec 12, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added assets/proposals/NNNN-atf-script-runner/atf_script_runner_diagram.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
49 changes: 49 additions & 0 deletions proposals/NNNN-atf-script-runner.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# ATF Script Runner

* Proposal: [SDL-NNNN](NNNN-atf-script-runner.md)
* Author: [Dmytro Boltovskyi](https://github.com/dboltovskyi)
* Status: **Awaiting review**
* Impacted Platforms: [ATF]

## Introduction

ATF is a great tool for both developers and testers that helps to test SDL functionality.
But it lacks a powerful test script runner.

## Motivation

Currently ATF is unable to:

- run multiple test scripts
- run test script in parallel
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dboltovskyi run test scripts in separate threads

- create test reports in standard format
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dboltovskyi standart format
BTW what does it mean standart format? Do we have some standarts for that?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dboltovskyi I think better to say "- create test reports with formats used in continuous integration systems" to avoid questions about standards.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AKalinich-Luxoft Yep, e.g.: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd


The purpose of this proposal is to develop extended script runner as part of ATF package.

## Proposed solution

Script runner is a command line tool with the following features:

- An option to run single test script
- An option to run batch of test scripts
- An option to run test set
- Possibility to run test scripts in parallel (in separate threads)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dboltovskyi
My previous comment on the same point is still actual even if you'll run tasks in separate threads. How are you going to use SDL in that case so we can avoid concurrent usage of devices or SDL internal state changes due to multiple tests run?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dev-gh Maybe my comment was missed. Let me duplicate it here:

Yep - with the same instance of SDL it's impossible. But the idea is to use SDL Watchdog (new feature in separate proposal) that will provide a new SDL instance for test script from the pull of available instances.

So each test script from test set will be run within separate SDL instance of a certain type (e.g.: RC=ON, PROPRIETARY)

- Ability to create reports with logs collected in a format used by continuous integration system

Input data could be defined as a set of input parameters and options.

![Component relations](../assets/proposals/NNNN-atf-script-runner/atf_script_runner_diagram.png)

Functionality of 'SDL Watchdog' and 'ATF Facade' is described in other proposals.

## Potential downsides

N/A

## Impact on existing code

No impact on existing code is observed since script runner is a new module.

## Alternatives considered

Refactoring of existing runner (bash script), but it will negotiate current big base of test artifacts.