Capture global scope

[pdawyndt]

TESTed could capture the global scope before and after running a test (main call, expression, statement) such that this global scope could also be evaluated by a custom oracle (the default behavior being that any global scope is accepted; no expected value can be set and no built-in oracle is needed). The global scope could be represented as a mapping of variable names (string) onto strongly typed objects (maybe also the storage location of the objects must be captured to allow testing identity). Custom oracles receive the state of the global scope before and after running the test, so the two can be compared, for example to see that no new global variables have been defined as a side-effect of running a test, that glopbal variables have not been modified, or that global variables have been created or modified).

[pdawyndt]

Think about how this could enable simple exercise like:

The variable x is assigned an array [1, 2, 3, 4]. Add the integer 5 (int) to the end of this array.

So a possible test suite might look like:

- statement: 'x = [1, 2, 3, 4]'  # setup: storage location of list must be captured in global scope
- <execute submisson>            # insert submission at this position in test script
- expression: 'x'                # test script
  return: [1, 2, 3, 4, 5]
  oracle: 'custom'               # checks both equality and identity of list

This requires some form of setup (code executed before the submission itself is executed). It also requires a way to explicitly put the submission at some place in the test script (by default it is automatically inserted as the first statement/expressoin in the script). This may also serve some other purposes:

• embed the submission in skeleton code (i.e. add a prefix and or suffix to the submission) provided in the test suite (e.g. student is only asked to complete a commando line: prefix is given, submission is used as a prefix; so-called fill-in-the-gaps exercise; we could also allow to modify the submission in a programmed way, as we do with test oracles)
• allow multi-part submissions where individual parts can be inserted, executed and tested separately