Architectural changes in v1.8

[svanoort]

Rollup of the top-level architecture changes in v1.8

Batch A: Internals

1. Move execution logic into the test/benchmark object and out of the resttest.py class - they provide an execute method, and work on callbacks to log/return results (greatly simplifies all the below, quickly).
   • Signature: execute_macro(self, context=None, testConfig=None, cmdlineArgs=None, callbacks=DefaultLogging, _args, *_kwargs)
     • Lets you later first look for variables in self, then testConfig, then cmdlineArgs (which are a dictionary).
     • With this, we can remove the default value setting for testConfigs
     • Parsers (post-refactoring) can determine which elements are allowed for each item.
   • Allows for xUnit output: v2 Part 4: Jenkins / pytest integration or xUnit output #43
   • Allows for the macro steps to optimize dynamic evaluation, per Make all test properties dynamic, and support much easier ways to set them #101
   • Pre-work for dynamic macros/subcomponents, from v2 Part 2: Execution Macros And Steps #92
   • Challenge: lots of command-line & test-set config options need to be consumed here. Lots of logic and dependencies on resttest.py
   • Challenge: cyclic dependency issues, need to move outputs from macro into objects too, because tests/benchmarks can't import resttest (cyclic dependency issue, since it depends on them)
2. Fix the nasty parsing of configuration, per v2 Part 1: Refactor nightmarish parsing configuration to be DRY and elegant #75
   • Parsing class gets built out with shared extensions, which will do testing and type conversion and docs generation given a list of Options
   • Define an "Optionable" class:

  get_all_options(self)  # Returns a collection of the options allowed here, maybe for docs
  set_option(name, value)  # vs. object.option = value ? (with properties if needed), returns the object itself for fluent use
  parse(self, config)  # parse yaml and return the object
  add_option(self, arguments)  # Add a new config option

• SharedOptions class created, that cmdline, testConfig, and test/Benchmark can use for common settings
• cmdLine, testConfig, and test/Benchmark then can provide their own additional options beyond that (shared options are checked first, then internal options)

Batch B: Major features built on top of internals

1. Do some sort of mapping to unittest framework, per v2 Part 4: Jenkins / pytest integration or xUnit output #43 and provide coupling to allow xUnit output
2. Clever dynamic binding of properties/settings: Make all test properties dynamic, and support much easier ways to set them #101
   • Relies on property-based getters and setters I think, plus either using the "with" syntax to do context dynamic substitution, or setting it on the macro object.
   • Parser/option handler is responsible for creating these dynamic bindings in object, using dynamic binding names

Item A1 can occur in a single branch and is fairly isolated. B1 completely depends on it and needs investigation to determine if this approach will work.
Item A2 will block development of new features while in progress. B2 is completely dependent on it.