Proposal for "ATF Script Runner" #2

dboltovskyi · 2017-11-07T12:54:13Z

Proposal for "ATF Script Runner"

dev-gh · 2017-11-07T13:11:05Z

proposals/NNNN-atf-script-runner.md

+
+Develop script runner with the following features:
+
+  - Ability to run single test script


@dboltovskyi
I think 'option' is more appropriate in that context than 'ability'

dev-gh · 2017-11-07T13:14:16Z

proposals/NNNN-atf-script-runner.md

+  - Ability to run single test script
+  - Ability to run batch of test scripts
+  - Ability to run test set
+  - Possibility to run scripts in parallel


@dboltovskyi
Is that really possilble w/o side effects? I mean if several test scripts will be run simultaneously and use same SDL instance they may have affect on each other. In case there will be several SDL instances again we may face with same problem due to connection parameters, communication channels simultaneous usage etc.

@dev-gh Yep - with the same instance of SDL it's impossible. But the idea is to use SDL Watchdog (new feature in separate proposal) that will provide a new SDL instance for test script from the pull of available instances.

dev-gh · 2017-11-07T13:15:18Z

proposals/NNNN-atf-script-runner.md

+
+## Potential downsides
+
+Existing CI jobs needs to be reconfigured in order to use new script runner.


@dboltovskyi
I think this is our local pain in the back and not something that committee should worry about.

@dev-gh ,
not agree, we don't know who can use SDL and ATF and all other open source repos.
The change could affect not only us :)

dev-gh · 2017-11-07T13:16:01Z

proposals/NNNN-atf-script-runner.md

+
+## Alternatives considered
+
+Use one of the existing test runners, e.g. Python Nose, PyTest etc.


@dboltovskyi
Than why we shouldn't do that? What are the advantages of yet another test runner?

@dboltovskyi , suggest to keep silence about 3rd party if we dpon't want to adjust 3rd party of course.
as alternative, total refactoring of existing runner, but it will negotiate current big base of test artifacts

@dev-gh Using of test runner from Python seems can be more complex than implementation of our own. But this has to be investigated.
Anyway any existing runner can be adopted in scope of this proposal.

dev-gh · 2017-11-07T13:19:01Z

proposals/NNNN-atf-script-runner.md

+## Introduction
+
+Originally ATF was developed for SDL developers and thats why it lacks a few features of modern Test Frameworks
+(Java jUnit, Python Nose, etc.)


@dboltovskyi
Not sure whether it is proper motivation since these are like general purpose frameworks whereas ATF is specifically tailored for SDL.

AKalinich-Luxoft · 2017-11-07T13:07:09Z

proposals/NNNN-atf-script-runner.md

+
+## Introduction
+
+Originally ATF was developed for SDL developers and thats why it lacks a few features of modern Test Frameworks


@dboltovskyi for SDL developers? Not for testers? :)

@dboltovskyi As I remember the original customer who requested ATF was testing team

@dboltovskyi ,
no matter for who ATF is, but I suggest to do not mention other modern framwork.
It could birth bad thoughts about usage of some other tool instead of ATF.
Please rephrase it in manner like - ATF is a great tool fpor booth developers and testers that help test any change in SDL.

@AKalinich-Luxoft , @AByzhynar Actually I'm not aware about the history of ATF, but this particular phrase was provided by Alex Kutsan :)

AKalinich-Luxoft · 2017-11-07T13:09:02Z

proposals/NNNN-atf-script-runner.md

+Currently ATF is unable to:
+
+  - run multiple test scripts
+  - run test script in parallel


@dboltovskyi run test scripts in separate threads

AKalinich-Luxoft · 2017-11-07T13:10:34Z

proposals/NNNN-atf-script-runner.md

+
+  - run multiple test scripts
+  - run test script in parallel
+  - create test reports in standard format


@dboltovskyi standart format
BTW what does it mean standart format? Do we have some standarts for that?

@dboltovskyi I think better to say "- create test reports with formats used in continuous integration systems" to avoid questions about standards.

@AKalinich-Luxoft Yep, e.g.: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd

AKalinich-Luxoft · 2017-11-07T13:12:16Z

proposals/NNNN-atf-script-runner.md

+
+  - Ability to run single test script
+  - Ability to run batch of test scripts
+  - Ability to run test set


@dboltovskyi what is a difference between batch of test scripts and test set?

@AKalinich-Luxoft Test set means running of test script that are defined in particular text file.
Batch of test script means running scripts from defined folder (recursively).

AKalinich-Luxoft · 2017-11-07T13:20:48Z

proposals/NNNN-atf-script-runner.md

+  - Ability to run single test script
+  - Ability to run batch of test scripts
+  - Ability to run test set
+  - Possibility to run scripts in parallel


@dboltovskyi here also could be used "Ability" word as in other items

AKalinich-Luxoft · 2017-11-07T13:21:01Z

proposals/NNNN-atf-script-runner.md

+  - Ability to run batch of test scripts
+  - Ability to run test set
+  - Possibility to run scripts in parallel
+  - Ability to create reports in standard format with logs collected


@dboltovskyi standart

AKalinich-Luxoft · 2017-11-07T13:31:32Z

proposals/NNNN-atf-script-runner.md

+
+## Impact on existing code
+
+No impact on existing code is observed.


@dboltovskyi as I understand in terms of this proposal it is planned to improve test script runner so I think there will be impact at least on existing runner implementation. Isn't it?

@AKalinich-Luxoft Existing runner (start.sh or run.sh) remains untouched

aderiabin · 2017-11-07T15:49:53Z

proposals/NNNN-atf-script-runner.md

+
+  - run multiple test scripts
+  - run test script in parallel
+  - create test reports in standard format


@dboltovskyi I think better to say "- create test reports with formats used in continuous integration systems" to avoid questions about standards.

AByzhynar · 2017-11-07T15:37:00Z

proposals/NNNN-atf-script-runner.md

+
+## Introduction
+
+Originally ATF was developed for SDL developers and thats why it lacks a few features of modern Test Frameworks


@dboltovskyi thats-> that's

AByzhynar · 2017-11-07T15:37:56Z

proposals/NNNN-atf-script-runner.md

+
+## Introduction
+
+Originally ATF was developed for SDL developers and thats why it lacks a few features of modern Test Frameworks


@dboltovskyi As I remember the original customer who requested ATF was testing team

Itileda · 2017-11-07T16:46:44Z

proposals/NNNN-atf-script-runner.md

+
+## Introduction
+
+Originally ATF was developed for SDL developers and thats why it lacks a few features of modern Test Frameworks


@dboltovskyi ,
no matter for who ATF is, but I suggest to do not mention other modern framwork.
It could birth bad thoughts about usage of some other tool instead of ATF.
Please rephrase it in manner like - ATF is a great tool fpor booth developers and testers that help test any change in SDL.

Itileda · 2017-11-07T16:52:24Z

proposals/NNNN-atf-script-runner.md

+
+## Potential downsides
+
+Existing CI jobs needs to be reconfigured in order to use new script runner.


@dev-gh ,
not agree, we don't know who can use SDL and ATF and all other open source repos.
The change could affect not only us :)

Itileda · 2017-11-07T16:55:16Z

proposals/NNNN-atf-script-runner.md

+
+## Alternatives considered
+
+Use one of the existing test runners, e.g. Python Nose, PyTest etc.


@dboltovskyi , suggest to keep silence about 3rd party if we dpon't want to adjust 3rd party of course.
as alternative, total refactoring of existing runner, but it will negotiate current big base of test artifacts

dboltovskyi · 2017-11-07T17:13:53Z

@Itileda @AByzhynar @AKalinich-Luxoft @aderiabin @dev-gh Please check new commit 852bb8e

dev-gh · 2017-11-08T08:21:02Z

proposals/NNNN-atf-script-runner.md

+
+## Introduction
+
+ATF is a great tool for both developers and testers that help test any change in SDL.


@dboltovskyi

helps with SDL testing

dev-gh · 2017-11-08T08:24:12Z

proposals/NNNN-atf-script-runner.md

+  - An option to run single test script
+  - An option to run batch of test scripts
+  - An option to run test set
+  - Possibility to run test scripts in parallel (in separate threads)


@dboltovskyi
My previous comment on the same point is still actual even if you'll run tasks in separate threads. How are you going to use SDL in that case so we can avoid concurrent usage of devices or SDL internal state changes due to multiple tests run?

@dev-gh Maybe my comment was missed. Let me duplicate it here:

Yep - with the same instance of SDL it's impossible. But the idea is to use SDL Watchdog (new feature in separate proposal) that will provide a new SDL instance for test script from the pull of available instances.

So each test script from test set will be run within separate SDL instance of a certain type (e.g.: RC=ON, PROPRIETARY)

* SendLocation for Mobile Nav This proposal is about extending the capability for apps to receive and service SendLocation requests from other apps which is currently limited to the Head Unit's embedded navigation system. * Updating links to assets * Updated links to assets * Create a * Asset #1 * asset #2 * Delete a * Delete SendLocationForMobileNav_RAI.jpg * Add files via upload * Delete SendLocationForMobileNav_RequestResponse.jpg * Add files via upload * Addressed comments

…vicelink#613) * Create NNNN-Avoid_custom_button_subscription_when_HMI_does_not_support.md * Changes after review * Changes after review #2 * Changes after review #3 * Update NNNN-Avoid_custom_button_subscription_when_HMI_does_not_support.md

* Revision fixes fixing typos Updates after review #1 Update 0206-remote_atf_testing.md Minor updates Minor updates #2 minor text formatting * Fixes after Laura review * Update 0206-remote_atf_testing.md * Update 0206-remote_atf_testing.md * Update 0206-remote_atf_testing.md

Initial version

43d670d

dboltovskyi requested review from AByzhynar, aderiabin, AKalinich-Luxoft, AStasiuk, Itileda and vkushnirenko-luxoft November 7, 2017 12:55

dev-gh reviewed Nov 7, 2017

View reviewed changes

AKalinich-Luxoft requested changes Nov 7, 2017

View reviewed changes

aderiabin requested changes Nov 7, 2017

View reviewed changes

AByzhynar requested changes Nov 7, 2017

View reviewed changes

Itileda requested changes Nov 7, 2017

View reviewed changes

Update according to comments in review

852bb8e

dboltovskyi force-pushed the atf_script_runner branch from 39ad423 to 852bb8e Compare November 7, 2017 17:13

AKalinich-Luxoft approved these changes Nov 8, 2017

View reviewed changes

dev-gh reviewed Nov 8, 2017

View reviewed changes

AByzhynar approved these changes Nov 30, 2017

View reviewed changes

dboltovskyi force-pushed the atf_script_runner branch 2 times, most recently from 2682667 to 5170bb8 Compare December 12, 2017 10:34

Add component diagram

4dee3fc

dboltovskyi force-pushed the atf_script_runner branch from 5170bb8 to 4dee3fc Compare December 12, 2017 10:37

aderiabin approved these changes Dec 20, 2017

View reviewed changes


		Develop script runner with the following features:

		- Ability to run single test script


		## Potential downsides

		Existing CI jobs needs to be reconfigured in order to use new script runner.


		## Alternatives considered

		Use one of the existing test runners, e.g. Python Nose, PyTest etc.


		## Introduction

		Originally ATF was developed for SDL developers and thats why it lacks a few features of modern Test Frameworks


		## Impact on existing code

		No impact on existing code is observed.


		## Introduction

		ATF is a great tool for both developers and testers that help test any change in SDL.

Proposal for "ATF Script Runner" #2

Are you sure you want to change the base?

Proposal for "ATF Script Runner" #2

Uh oh!

Conversation

dboltovskyi commented Nov 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dboltovskyi Nov 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dboltovskyi commented Nov 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dboltovskyi Nov 7, 2017 •

edited

Loading