Refactor/user function handling modules + Manager can run additional worker on thread

[jlnav]

Addresses #1166

User - Manager runs local worker on thread - Gen Worker 0

The manager can start an additional worker thread with the worker ID 0.

This worker is, via W["gen_worker"] = True, and filtering options, made to run gens only.

Tentative interface: libE_specs["gen_on_manager"] = True

Internal - Refactor runners.py

Largely refactors utils/runners.py so additional "Runners" for user functions can be more easily developed/extended.

GlobusComputeRunner and ThreadRunner inherit from Runner.

Runner inspects the contents of the input sim/gen specs and instantiates the correct subclass on the left-hand-side:

>>> sim_specs = {... , "globus_compute_endpoint": "1234}
>>> runner = Runner(sim_specs)
>>> runner
<libensemble.utils.runners.GlobusComputeRunner object at 0x1490ec190>
>>> runner = Runner({"sim_f": print})
>>> runner
<libensemble.utils.runners.Runner object at 0x104e024d0>

Internal - _WorkerIndexer class for indexing into worker arrays

Internal - Workers can be assigned gen_worker.

[codecov]

Codecov Report

Attention: Patch coverage is 94.26230% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 93.50%. Comparing base (801b6ed) to head (8a50e60).
Report is 23 commits behind head on develop.

Files	Patch %	Lines
libensemble/comms/comms.py	78.57%	3 Missing ⚠️
libensemble/alloc_funcs/fast_alloc.py	81.81%	2 Missing ⚠️
libensemble/alloc_funcs/only_one_gen_alloc.py	86.66%	2 Missing ⚠️
libensemble/manager.py	96.49%	1 Missing and 1 partial ⚠️
libensemble/utils/runners.py	95.55%	1 Missing and 1 partial ⚠️
libensemble/alloc_funcs/inverse_bayes_allocf.py	83.33%	0 Missing and 1 partial ⚠️
libensemble/alloc_funcs/persistent_aposmm_alloc.py	83.33%	0 Missing and 1 partial ⚠️
libensemble/ensemble.py	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #1216       +/-   ##
============================================
+ Coverage    72.59%   93.50%   +20.91%     
============================================
  Files           90       90               
  Lines         8153     8212       +59     
  Branches      1457     1469       +12     
============================================
+ Hits          5919     7679     +1760     
+ Misses        1998      323     -1675     
+ Partials       236      210       -26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shuds13]

How different from self._W = W

[jlnav]

I overloaded __getattr__, which is called for .attribute access even in the constructor. This is a workaround.

[shuds13]

From a quick look, I don't understand the number of calls to the _Worker constructor. Dont we just need to create one wrapper object? This could use the __getitem__ method to set indexing. e.g.

class _Worker:
    def __init__(self, W, manager_as_worker=False):
        self.W = W
        self.manager_as_worker = manager_as_worker

    def __getitem__(self, key):
        if self.manager_as_worker:
            return self.W[key]
        else:
            return self.W[key - 1]

This object can be an attribute of manager class, dont need to pass around.

[shuds13]

@jlnav

• In start_only_persistent why not gen_workers=False on the sim
• What about fast_alloc_and_pausing and give_pregenerated_work?