Runner-Pool

[elrodrigues]

Build a Trainer/Runner pool (where each runner would have 1 environment and 1 job associated) for parallel training.

[elrodrigues]

The objective of this issue has changed because of design of the middleware. Pool is probably the wrong word to use here since I'm talking about lazily spinning up Trainers/Runners when a new job is added and strapping a manager to these runners, but it's the closest word I have in my vocabulary to describe this.

This is a certified schizo moment.

[elrodrigues]

The runners/trainers will have a 'hook' to sync their models to the manager's master model every couple episodes. The manager will also periodically 'down'-sync its master-model to its trainers.

I haven't yet decided Taus for the up-sync/hook and down-sync. These will be set in config.

[elrodrigues]

This has changed a little now. The trainer's 'hook' is no longer an up-sync but instead a down-sync. The up-sync is handled internally by the trainer after its master model is set by the manager.

I imagine trainers implementing their form of soft_update_agent(local, target). For BDQTrainer for example, this function would contain something along the lines of:

BDQAgent.soft_update(self.pre_net, self.pre_target, self.tau)
BDQAgent.soft_update(self.state_net, self.state_target, self.tau)
for i in range(self.num_actions):
    BDQAgent.soft_update(self.adv_targets[i], self.adv_nets[i], self.tau)