Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "You are welcome to use this coding environment to train your agent for the project.  Follow the instructions below to get started!\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "Run the next code cell to install a few packages.  This line will take a few minutes to run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mtensorflow 1.7.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mipython 6.5.0 has requirement prompt-toolkit<2.0.0,>=1.0.15, but you'll have prompt-toolkit 3.0.18 which is incompatible.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# !pip -q install ./python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environments corresponding to both versions of the environment are already saved in the Workspace and can be accessed at the file paths provided below.  \n",
    "\n",
    "Please select one of the two options below for loading the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_size -> 5.0\n",
      "\t\tgoal_speed -> 1.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "from unityagents import UnityEnvironment\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# select this option to load version 2 (with 20 agents) of the environment\n",
    "# env = UnityEnvironment(file_name='/data/Reacher_Linux_NoVis/Reacher.x86_64')\n",
    "env = UnityEnvironment(file_name='Reacher.app')\n",
    "TRAIN = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]\n",
    "print(torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Note that **in this coding environment, you will not be able to watch the agents while they are training**, and you should set `train_mode=True` to restart the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  A few **important notes**:\n",
    "- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```\n",
    "- To structure your work, you're welcome to work directly in this Jupyter notebook, or you might like to start over with a new file!  You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.\n",
    "- In this coding environment, you will not be able to watch the agents while they are training.  However, **_after training the agents_**, you can download the saved model weights to watch the agents on your own machine! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ReplayBuffer and OrnsteinUhlenbeckProcess CLASSES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import copy\n",
    "from collections import namedtuple, deque\n",
    "\n",
    "import torch\n",
    "\n",
    "\n",
    "class ReplayBuffer:\n",
    "    \"\"\"Fixed-size buffer to store experience tuples.\"\"\"\n",
    "\n",
    "    def __init__(self, action_size, buffer_size, batch_size, seed, device):\n",
    "        \"\"\"Initialize a ReplayBuffer object.\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            action_size (int): dimension of each action\n",
    "            buffer_size (int): maximum size of buffer\n",
    "            batch_size (int): size of each training batch\n",
    "            seed (int): random seed\n",
    "        \"\"\"\n",
    "        self.action_size = action_size\n",
    "        self.memory = deque(maxlen=buffer_size)  \n",
    "        self.batch_size = batch_size\n",
    "        self.experience = namedtuple(\"Experience\", field_names=[\"state\", \"action\", \"reward\", \"next_state\", \"done\"])\n",
    "        self.seed = random.seed(seed)\n",
    "        self.device = device\n",
    "    \n",
    "    def add(self, state, action, reward, next_state, done):\n",
    "        \"\"\"Add a new experience to memory.\"\"\"\n",
    "        e = self.experience(state, action, reward, next_state, done)\n",
    "        self.memory.append(e)\n",
    "    \n",
    "    def sample(self):\n",
    "        \"\"\"Randomly sample a batch of experiences from memory.\"\"\"\n",
    "        experiences = random.sample(self.memory, k=self.batch_size)\n",
    "\n",
    "        states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(self.device)\n",
    "        actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).float().to(self.device)\n",
    "        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(self.device)\n",
    "        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(self.device)\n",
    "        dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(self.device)\n",
    "  \n",
    "        return (states, actions, rewards, next_states, dones)\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"Return the current size of internal memory.\"\"\"\n",
    "        return len(self.memory)\n",
    "    \n",
    "class OrnsteinUhlenbeckProcess:\n",
    "    def __init__(self, size, seed, mu=0, std=.2, theta=.15, dt=1):\n",
    "        self.theta = theta\n",
    "        self.mu = mu * np.ones(size)\n",
    "        self.std = std\n",
    "        self.dt = dt\n",
    "        self.size = size\n",
    "        self.seed = random.seed(seed)\n",
    "        self.reset()\n",
    "\n",
    "    def sample(self):\n",
    "        dx = self.theta * (self.mu - self.x_prev) * self.dt + self.std * np.sqrt(\n",
    "            self.dt) * np.random.standard_normal(self.size)\n",
    "        self.x_prev = self.x_prev + dx\n",
    "        return self.x_prev\n",
    "\n",
    "    def reset(self):\n",
    "        self.x_prev = self.mu * np.ones(self.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Actor and Critic CLASSES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from collections import OrderedDict\n",
    "\n",
    "\n",
    "def hidden_init(layer):\n",
    "    fan_in = layer.weight.data.size()[0]\n",
    "    lim = 1. / np.sqrt(fan_in)\n",
    "    return (-lim, lim)\n",
    "\n",
    "class Actor(nn.Module):\n",
    "    def __init__(self, obs_size, act_size, fc1_units=256, fc2_units=128, fc3_units=64):\n",
    "        super(Actor, self).__init__()\n",
    "\n",
    "        self.net = nn.Sequential(OrderedDict([\n",
    "            ('fc1', nn.Linear(obs_size, fc1_units)),\n",
    "            ('relu1', nn.ReLU()),\n",
    "            ('fc2', nn.Linear(fc1_units, fc2_units)),\n",
    "            ('relu2', nn.ReLU()),\n",
    "            ('fc3', nn.Linear(fc2_units, fc3_units)),\n",
    "            ('relu3', nn.ReLU()),\n",
    "            ('fc4', nn.Linear(fc3_units, act_size)),\n",
    "            ('tanh', nn.Tanh())\n",
    "        ]))\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)\n",
    "    \n",
    "    def reset_parameters(self):\n",
    "        self.net.fc1.weight.data.uniform_(*hidden_init(self.net.fc1))\n",
    "        self.net.fc2.weight.data.uniform_(*hidden_init(self.net.fc2))\n",
    "        self.net.fc3.weight.data.uniform_(*hidden_init(self.net.fc3))\n",
    "        self.net.fc4.weight.data.uniform_(-3e-3, 3e-3)\n",
    "\n",
    "\n",
    "class Critic(nn.Module):\n",
    "    def __init__(self, obs_size, act_size, fc1_units=256, fc2_units=128, fc3_units=64):\n",
    "        super(Critic, self).__init__()\n",
    "\n",
    "        self.obs_net = nn.Sequential(OrderedDict([\n",
    "             ('fcs1', nn.Linear(obs_size, fc1_units)),\n",
    "             ('relu1', nn.ReLU()),\n",
    "        ]))\n",
    "\n",
    "        self.out_net = nn.Sequential(OrderedDict([\n",
    "            ('fc2', nn.Linear(fc1_units + act_size, fc2_units)),\n",
    "            ('relu2', nn.ReLU()),\n",
    "            ('fc3', nn.Linear(fc2_units, fc3_units)),\n",
    "            ('relu3', nn.ReLU()),\n",
    "            ('fc4', nn.Linear(fc3_units, 1))\n",
    "        ]))\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def forward(self, states, actions):\n",
    "        obs = self.obs_net(states)\n",
    "        return self.out_net(torch.cat([obs, actions], dim=1))\n",
    "    \n",
    "    def reset_parameters(self):\n",
    "        self.obs_net.fcs1.weight.data.uniform_(*hidden_init(self.obs_net.fcs1))\n",
    "        self.out_net.fc2.weight.data.uniform_(*hidden_init(self.out_net.fc2))\n",
    "        self.out_net.fc3.weight.data.uniform_(*hidden_init(self.out_net.fc3))\n",
    "        self.out_net.fc4.weight.data.uniform_(-3e-3, 3e-3)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AGENT CLASS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "from collections import namedtuple, deque\n",
    "\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "BUFFER_SIZE = int(1e6)      # replay buffer size\n",
    "BATCH_SIZE = 1024           # minibatch size\n",
    "REPLAY_INITIAL = BATCH_SIZE # initial memory before updatting the network\n",
    "GAMMA = 0.99                # discount factor\n",
    "LR_ACTOR = 1e-4             # learning rate \n",
    "LR_CRITIC = 1e-3            # learning rate of the critic\n",
    "UPDATE_EVERY = 4            # how often to update the network\n",
    "TAU = 1e-3                  # soft update\n",
    "WEIGHT_DECAY = 0            # L2 weight decay\n",
    "NET_BODY = (256, 128, 64)       # hidden layers\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "class Agent():\n",
    "    \"\"\"Interacts with and learns from the environment.\"\"\"\n",
    "    \n",
    "    def __init__(self,\n",
    "                 state_size,\n",
    "                 action_size,\n",
    "                 random_seed,\n",
    "                 num_agents,\n",
    "                 net_body=NET_BODY,\n",
    "                 gamma=GAMMA,\n",
    "                 buffer_size=BUFFER_SIZE,\n",
    "                 batch_size=BATCH_SIZE,\n",
    "                 replay_initial=REPLAY_INITIAL,\n",
    "                 update_every=UPDATE_EVERY,\n",
    "                 lr_actor=LR_ACTOR,\n",
    "                 lr_critic=LR_CRITIC,\n",
    "                 weight_decay=WEIGHT_DECAY,\n",
    "                 tau=TAU\n",
    "    ):\n",
    "        \"\"\"Initialize an Agent object.\n",
    "        \n",
    "        Params\n",
    "        ======\n",
    "            state_size (int): dimension of each state\n",
    "            action_size (int): dimension of each action\n",
    "            random_seed (int): random seed\n",
    "        \"\"\"\n",
    "        self.state_size = state_size\n",
    "        self.action_size = action_size\n",
    "        self.seed = random.seed(random_seed)\n",
    "        self.gamma = gamma\n",
    "        self.batch_size = batch_size\n",
    "        self.replay_initial = replay_initial\n",
    "        self.update_every = update_every\n",
    "        self.tau = tau\n",
    "        self.device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "        \n",
    "        fc1_units, fc2_units, fc3_units = net_body\n",
    "        # Actor Network (w/ Target Network)\n",
    "        self.actor_local = Actor(state_size, action_size, fc1_units, fc2_units, fc3_units).to(self.device)\n",
    "        self.actor_target = Actor(state_size, action_size, fc1_units, fc2_units, fc3_units).to(self.device)\n",
    "        self.actor_optimizer = optim.Adam(self.actor_local.parameters(), lr=LR_ACTOR)\n",
    "\n",
    "        # Critic Network (w/ Target Network)\n",
    "        self.critic_local = Critic(state_size, action_size, fc1_units, fc2_units, fc3_units).to(self.device)\n",
    "        self.critic_target = Critic(state_size, action_size, fc1_units, fc2_units, fc3_units).to(self.device)\n",
    "        self.critic_optimizer = optim.Adam(self.critic_local.parameters(), lr=LR_CRITIC, weight_decay=WEIGHT_DECAY)\n",
    "\n",
    "        # Noise process\n",
    "        self.noise = OrnsteinUhlenbeckProcess((num_agents, action_size), random_seed)\n",
    "\n",
    "        # Replay memory\n",
    "        self.memory = ReplayBuffer(\n",
    "            action_size=action_size,\n",
    "            buffer_size=buffer_size,\n",
    "            batch_size=batch_size,\n",
    "            seed=random_seed,\n",
    "            device=self.device\n",
    "        )\n",
    "   \n",
    "    \n",
    "    def step(self, states, actions, rewards, next_states, dones, step):\n",
    "        \"\"\"Save experience in replay memory, and use random sample from buffer to learn.\"\"\"\n",
    "        # Save experience / reward\n",
    "        for state, action, reward, next_state, done in zip(states, actions, rewards, next_states, dones):\n",
    "            self.memory.add(state, action, reward, next_state, done) \n",
    "\n",
    "        # Learn every UPDATE_EVERY time steps after reaching the minimal sample size.\n",
    "        if (len(self.memory) > self.replay_initial) and (step % self.update_every == 0):\n",
    "            experiences = self.memory.sample()\n",
    "            self.update(experiences, self.gamma)\n",
    "       \n",
    "    \n",
    "    def act(self, state, add_noise=True):\n",
    "        \"\"\"Returns actions for given state as per current policy.\"\"\"\n",
    "\n",
    "        state = torch.from_numpy(state).float().to(device)\n",
    "\n",
    "        self.actor_local.eval()\n",
    "        with torch.no_grad():\n",
    "            action = self.actor_local(state).cpu().data.numpy()\n",
    "        self.actor_local.train()\n",
    "\n",
    "        if add_noise:\n",
    "            action += self.noise.sample()\n",
    "            \n",
    "        return np.clip(action, -1, 1)\n",
    "\n",
    "    def reset(self):\n",
    "        self.noise.reset()\n",
    "\n",
    "    def update(self, experiences, gamma):\n",
    "        \"\"\"Update policy and value parameters using given batch of experience tuples.\n",
    "        Q_targets = r + γ * critic_target(next_state, actor_target(next_state))\n",
    "        where:\n",
    "            actor_target(state) -> action\n",
    "            critic_target(state, action) -> Q-value\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            experiences (Tuple[torch.Tensor]): tuple of (s, a, r, s', done) tuples \n",
    "            gamma (float): discount factor\n",
    "        \"\"\"\n",
    "        states, actions, rewards, next_states, dones = experiences\n",
    "        \n",
    "        # Update Critic\n",
    "        # Get expected Q values from critic model\n",
    "        curr_Q = self.critic_local.forward(states, actions)\n",
    "        # Get predicted Q values (for next states) from target model\n",
    "        next_actions = self.actor_target.forward(next_states)\n",
    "        next_Q = self.critic_target.forward(next_states, next_actions)\n",
    "        expected_Q = rewards + (self.gamma * next_Q * (1 - dones))\n",
    "        # Compute critic loss\n",
    "        critic_loss = F.mse_loss(curr_Q, expected_Q)\n",
    "\n",
    "        self.critic_optimizer.zero_grad()\n",
    "        critic_loss.backward()\n",
    "        torch.nn.utils.clip_grad_norm_(self.critic_local.parameters(), 1)\n",
    "        self.critic_optimizer.step()\n",
    "        \n",
    "        # Update Actor\n",
    "        # Get expected actions from actor model\n",
    "        actions_pred = self.actor_local(states)\n",
    "        actor_loss = -self.critic_local(states, actions_pred).mean()\n",
    "        \n",
    "        self.actor_optimizer.zero_grad()\n",
    "        actor_loss.backward()\n",
    "        self.actor_optimizer.step()\n",
    "        \n",
    "        # Update target networks\n",
    "        self.update_target(self.critic_local, self.critic_target, self.tau)  \n",
    "        self.update_target(self.actor_local, self.actor_target, self.tau)  \n",
    "    \n",
    "\n",
    "\n",
    "    def update_target(self, local_model, target_model, tau):\n",
    "        \"\"\"Soft update model parameters.\n",
    "        θ_target = τ*θ_local + (1 - τ)*θ_target\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            local_model: PyTorch model (weights will be copied from)\n",
    "            target_model: PyTorch model (weights will be copied to)\n",
    "            tau (float): interpolation parameter \n",
    "        \"\"\"\n",
    "        for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):\n",
    "            target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(agent, env, n_episodes=2000, max_t=1000, min_score=30, num_agents=20, print_every=100):\n",
    "    \"\"\"DDPG-Learning.\n",
    "    Params\n",
    "    ======\n",
    "        agent: An instance of Agent class\n",
    "        env: Unity environment\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "    \"\"\"\n",
    "    score_list = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)      # last 100 scores\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        average_score = 0\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        scores = np.zeros(num_agents)\n",
    "        score = 0\n",
    "        agent.reset()\n",
    "        for t in range(max_t):\n",
    "            actions = agent.act(states)\n",
    "            env_info = env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations      # get the next state\n",
    "            rewards = env_info.rewards                     # get the reward\n",
    "            dones = env_info.local_done                    # see if episode has finished\n",
    "            agent.step(states, actions, rewards, next_states, dones, t)\n",
    "            scores += rewards                                # update the score\n",
    "            states = next_states                             # roll over the state to next time step\n",
    "            if np.any(dones):\n",
    "                break\n",
    "        mean_score = np.mean(scores)\n",
    "        scores_window.append(mean_score)           # save most recent score\n",
    "        score_list.append(mean_score)              # save most recent score\n",
    "            \n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "\n",
    "        if i_episode % print_every == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            \n",
    "        if np.mean(scores_window) >= min_score:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_env_solved_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_env_solved_critic.pth')\n",
    "            break\n",
    "    return score_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define Agent Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "start_time = datetime.now()\n",
    "\n",
    "# VARS\n",
    "config = {\n",
    "    \"buffer_size\": int(1e6),    # replay buffer size\n",
    "    \"batch_size\": 1024,         # minibatch size\n",
    "    \"replay_initial\": 1024,     # initial memory before updating the network\n",
    "    \"gamma\": 0.99,              # discount factor\n",
    "    \"lr_actor\": 1e-4,           # learning rate \n",
    "    \"lr_critic\": 1e-3,          # learning rate of the critic\n",
    "    \"update_every\": 2,          # how often to update the network\n",
    "    \"tau\": 1e-3,                # soft update\n",
    "    \"weight_decay\": 0,          # l2 weight decay\n",
    "    \"net_body\": (256, 128, 64), # hidden layers    \n",
    "}\n",
    "agent = Agent(\n",
    "        state_size=state_size,\n",
    "        action_size=action_size,\n",
    "        random_seed=0,\n",
    "        num_agents=num_agents,\n",
    "        **config\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Execute Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Agent\n",
      "Episode 100\tAverage Score: 28.13\n",
      "Episode 105\tAverage Score: 30.01\n",
      "Environment solved in 105 episodes!\tAverage Score: 30.01\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd4XOWZ9/Hvrd5sS7JkW+690FxQjMEUx4QAIQspJIQ0NiGQugn7pkH23Tdkr81uspu2m2RJCBDIJguEksACS+KlGYxtkMFdtmXLTbJsq/c6c79/zNgYI9uK0cyRZn6f69KlmTNn5tyHY+an8zznPI+5OyIikrxSgi5ARESCpSAQEUlyCgIRkSSnIBARSXIKAhGRJKcgEBFJcgoCEZEkpyAQEUlyCgIRkSSXFnQBA1FUVORTp04NugwRkWFl3bp1de5efKr1Yh4EZpYKlAHV7v5eM5sGPAAUAq8Bn3D3npN9xtSpUykrK4t1qSIiCcXM9g5kvXg0DX0FKD/m+feBH7v7LKARuDEONYiIyAnENAjMbCJwFXBX9LkBy4GHo6vcB7wvljWIiMjJxfqM4CfAN4Bw9PlooMnd+6LPq4AJMa5BREROImZBYGbvBQ67+7pjF/ezar/jYJvZzWZWZmZltbW1MalRRERie0awFLjazPYQ6RxeTuQMId/MjnRSTwQO9Pdmd7/T3UvdvbS4+JSd3iIicppiFgTufpu7T3T3qcBHgGfd/WPAc8C10dVuAB6LVQ0iInJqQdxQ9k3g/5jZTiJ9BncHUIOIiETF5YYyd38eeD76uBJYHI/tiiSjvlCYhvYexozMCrqUIcvd2VXbRvGILEZlpw/KZ5btaWDHoTaWzSlmfH72oHxmvAyLO4tFhit3p7M3RF/YCYWcnlCYzp4QHT0hALIzUslKTyE7PZWs9FQy01KIXGU98M+vae6i4nAb5TUtrKmsp2xPI23dfcwrGcn7F47n6vkTGDfq1KHg7tz78h5e2d1ARloKmWkpnDVhFNeeO5GcjMhXRU9fmLW765k5Jo+SUW/+slu/v4nX9jaytaaFqsYOPrFkKledU3LK7bZ09VLd2MmEgmxGZp3+l/KqnXW8treRA82d1LZ289HzJrN87th+9/O7T5Zz10u7MYMzSkZSOqWAorxMRmSlkZORRncoTFdPiNQUo3RqAWeOH0VqyomPS8WhVj55zytHj+sZJSP5wKIJfHzJFLLSU097n+LFhsPk9aWlpa47iyVW6tq6GZWdTnpq/y2l1U2d3P3iblq7eukNhQk5pKcY6akpZKWnMDovk6K8TKaMzmHxtMKjn7OxqolvP76F1/c1DbgWM5hcmMPZE0Yxf2I+M8fmMaUwh/H52ZTXtPDyrnpe3dNAXVs3TR29NLT3HP3yAZhRnMuS6aOZWJDD01sOsmF/E6kpxgcWTuBLy2cyZXQuAM0dvfSEwhSPyASgvbuPrz20gf/ZfJApo3MwoKMnxOHWbvJz0vnkkil09oZ49LVq6tt7yMtM4x+uOZP3L5xAY0cvtz++hcc3RK77KMrLICcjjX0NHXxw0URuv/oMapq7eGRdFat21VGQk8GE/Gwy01JYt6+RrQdaCEe/hvJz0pk1Jo/zZxRx4cwiFkzKJyPtjePS0tXLD/+0nbbuEDdfPJ0540bQ0hXZ/qOvVR/dfooZTZ293H/TeZw7pfDo+/tCYW57dBMPravi+sWTGDsyi7WVDWyoanrTf8fjjcpOZ8n0Qs6fPprzZxQxe2ze0cBu7erlmp+toqWrl59/dBGv72/iz1sO8tq+JiYWZPP1y+cwozjvaEiPGZnJu88Yx3nTC2np7GXVrnpW76qnqrGD2tZuGtp7KMrLZHJhDpNH5/Dh0knMHJM34H9Db/73ZOvcvfSU6ykIJBm1d/fx1KYaHiqr4pU9DeTnpHP5GeN4zzklLJleSGZa5K+4P205yDce3khnb4ii3AzSUlNITTH6wmF6+5yOnj5auvqOfm5BTjpXnVNCKOw88Op+Rudm8oklU8jNTCU1Gh45GankZEQ+v6s3TEdPiK7eEF19ITq6Q+w83Mam6maqmzr7rX322DwmFuSQn51OQW4G04pymTkmj9ljR1CYm/GmdXfXtXPfy3u4/5V99IWdRZPz2d/QycGWLgDmjhvBJbOLeX57LRWHW7ntynl85qJpR7/kyvY08MuVlazYeoi0FONd88by3vkl3PfyHl7d08jyuWPYWNVEc2cvX3rnLK4/bxJjRmTRGwrz02cq+NlzO8nJSKOtu4+06F/XnT0hqpu6aOvuZcGkfM6bNpoZY/Koaepkb0MHW6qb2VTdTNgj/z2vXzyZT5w/hb31HXz19xs42NJFZloKHT0h3n3GWLYcaOFgSxdfXDaDzy+bSXZGKo3tPXzgjpdp7uzlD1+4gCmjc9ld1853nyznf8sPccu7ZvGVS2e96eyrpy9Ma1cvHT0hMqNnaR09IdZU1rNqZx2rdtYfPSYlo7L4cOkkrnvHJG5/fAvPbDvM7z5zHkumjz76eS9V1PFPT5Wztabl6LLJhTkcbu2iqzdMTkbq0fAZmZXGtKJcikdkUZibTl1bD3vr29nf2Ml9n1rM+TPe+Ny/hIJA5ARaunq58icvUt3UybSiXK5ZMJ49de2s2HqI9p4QmWkplE4toDA3k//ecICzJozkZ9cvYmpRbr+f19MXpr69m83VLTy+4QArth6kN+T89QVT+cq7Zp12c0d9Wze769rZW99BdVMn06N/7RflZf7Fn3W4pYtfrqykbG8jM4pymT1uBAAvbK+lbG8DORlp/OyjC7loVv+Xalc3dZKVFjn7AQiFnV+8sIsfr9jB3JIR/Ou185lXMvIt7yvb08CvV+1h0ZQCrlkwfsC1N3f2snpXPX94vYoVWw9hZoTdmVKYw4+uW8C00bncs2o3967aQ9GITH704fksnFzwps/YXdfO+/9jFQU5GYzPz2LVznrSUoz/e9U8/nrptL/kP99R+xs6WF1Zz5Mba1hZUcuRr8+/f+8Z3HjhWz8zHHb+tOUgPaEw500bzbhRWXT2hHhpZx0rd9QydmQmF84q5uwJ/Tc9hcOOw0mbpU5GQSByAv/4xFbuXrWbuz5ZyvK5Y47+VdjVG+KlijpW7apj9a56dh5u45PnT+WbV845eoYwEO3dfZEziNP4wg7Ckb/WT6ctu66tm/zsdNJO0Kw2GPY3dPDbtXtJNeOL75xJbuYbXZt9oTCpKXbCfpVXdjfw8bvXUpSbwfWLJ3PdOyYNWif6/oYOHnh1H6lm/O1ls/+ivp14URCI9GPn4Vau+MmLXHvuRL73wXNOum4o7Kf9l5gMHY3tPYzMTk/KYznQINBVQ5KQdtW28VBZFY+tr2Z8fjbf+8DZzByTx3f+eyvZGal8/fI5p/yMZPziSEQFx/WbyFspCCShuDu3PbqJB17dT2qKcdGsIjZWNXPVv7/EVeeU8GJFHd/+qzOOtnWLiIJAEszD66p44NX9fGLJFP5m+UzGjMyitrWbv//jZv7wejWzx+bx8SVTgi5TZEhREEjC2Fvfzu2Pb+G8aYXcfvWZR5t2ikdkcsfHF/FiRR1TR+ee8H4BkWSlIJCE0BcKc8uD60lJMX503YK3tO+bGRfP1ii2Iv1REMiwdrC5ixcranlyUw2v72vip9cvZMIwG+dFJGgKAhm2/vVP2/j5c7sAKMrL5JZ3zeKv5o8PuCqR4UdBIMPSmsp6fv7cLv5q/ng+f8kM5o4bQYou9xQ5Leo1kyGp4lArtz6ykdau3re81t7dxzce3siU0Tl8/4Nnc8b4kQoBkbdBQSBD0u3/vYUHXt1/tOnnWN9/ehv7Gzv412vnHx0eWUROn4JAhpwjIz2OHZnJPS/tZm99+9HXXthRy29W7+XTS6exeFrhST5FRAYqZkFgZllm9oqZbTCzLWb2nejye81st5mtj/4siFUNMvy4O//yp+2MH5XFQ5+9gNQU45+eKgciA4h9/rfrmD02j6+9+9RDRIjIwMTyvLobWO7ubWaWDrxkZv8Tfe3r7v5wDLctw9SKrYfYsL+J73/wbCaPzuELy2bwwxU7uOP5Xfzs2QrGjcrit585j+yMoT/rk8hwEbMzAo9oiz5Nj/4M/aFOJTChsPODP29nelEuH1w0EYCbLp7OhPxsvv/0NsaMzOL+m5YwZoTm4hUZTDHtIzCzVDNbDxwGVrj72uhL3zWzjWb2YzPrd/QvM7vZzMrMrKy2tjaWZcoQ8cKOw+w41MYtl80+Or59Vnoq//yBs7l4djH/ddN5jNWE7CKDLqZB4O4hd18ATAQWm9lZwG3AXOAdQCHwzRO89053L3X30uJiDQ2QDFbtrCczLYV3n/HmCccvnl3Mbz69+C2TpYvI4IjLVUPu3gQ8D1zh7jXRZqNu4NfA4njUIEPf2t31LJycf1ozZYnI6YvlVUPFZpYffZwNvAvYZmYl0WUGvA/YHKsaZPho7uxly4GWN03+LSLxEcurhkqA+8wslUjg/N7dnzCzZ82sGDBgPfC5GNYgw0TZngbc4bxpCgKReItZELj7RmBhP8uXx2qbMnytqawnIzWFhZPzgy5FJOnozmIZEtbubmCB+gdEAqEgkMC1dPWyubqZJRoyQiQQCgIJ3Lo9jYQddRSLBERBIIFbU1lPeqqxcHJB0KWIJCUFgQRuze4GFkzK1/hBIgFREEig2rr72FzdrMtGRQKkIJBArdh6kFDYuWCmgkAkKAoCCYy786uVu5k1Jo8lOiMQCYyCQALz8q56tta0cNNF0zXnsEiAFAQSmDtXVlKUl8k1C8cHXYpIUlMQSCC2H2zlhR21fGrpVDLTdLWQSJAUBBKIX71YSXZ6Kh87b3LQpYgkPQWBxF3FoVYeW1/Nh0snkp+TEXQ5IklPQSBxVdvazafufZX8nAw+v2xm0OWICAoCiaOu3hA3/aaMurZu7vpkKeNGaf5hkaEglhPTiBzl7nztoQ1sqGrijo8tYv4kzTsgMlTEcqrKLDN7xcw2mNkWM/tOdPk0M1trZhVm9qCZqZE4Cew83MYTG2v48vJZXHFWSdDliMgxYtk01A0sd/f5wALgCjNbAnwf+LG7zwIagRtjWIMMEasr6wH44KKJAVciIseLWRB4RFv0aXr0x4HlwMPR5fcRmcBeEtzqXfVMyM9mUmF20KWIyHFi2llsZqlmth44DKwAdgFN7t4XXaUKmBDLGiR44bCzprKeJdNHY6ahJESGmpgGgbuH3H0BMBFYDMzrb7X+3mtmN5tZmZmV1dbWxrJMibEdh1tp7Ojl/BkaWE5kKIrL5aPu3gQ8DywB8s3syNVKE4EDJ3jPne5e6u6lxcXF8ShTYmT1rkj/wJLpmpNYZCiK5VVDxWaWH32cDbwLKAeeA66NrnYD8FisapChYfWueiYVZjOxICfoUkSkH7G8j6AEuM/MUokEzu/d/Qkz2wo8YGb/CLwO3B3DGiRg4bCzdncDl585NuhSROQEYhYE7r4RWNjP8koi/QWSBLbWtNDcqf4BkaFMQ0xITK2pPNI/oCAQGaoUBBJTayrrmTo6h5JRun9AZKhSEEjMNLb3sLayQc1CIkOcgkBiIhx2bnlwPd19YT523pSgyxGRk1AQSEz89NmdvLCjlv/3V2dw1oRRQZcjIiehIJBBt3JHLT95ZgcfWDhBU1GKDAMKAhlU4bDz9Yc3MHvMCL77/rM1tpDIMKAgkEG1taaFQy3dfG7ZdLIzUoMuR0QGQEEgg+rFijoAls4oCrgSERkoBYEMqlU765gzdgRjRmo+YpHhQkEgg6arN8QrexpYOlNnAyLDiYJABk3ZnkZ6+sJcNEtBIDKcKAhk0Ly4s5b0VGPxNM07IDKcKAhk0KzaWcfCyQXkZsZydHMRGWwKAhkUDe09bDnQwkXqHxAZdhQEMihW7azDHZaqf0Bk2FEQyKBYtbOOEVlpnKNxhUSGnVjOWTzJzJ4zs3Iz22JmX4kuv93Mqs1sffTnPbGqQeLnpZ11nD99NGmp+ttCZLiJZa9eH/BVd3/NzEYA68xsRfS1H7v7D2K4bYmj/Q0dVDV28pkLpwVdioichljOWVwD1EQft5pZOTAhVtuT4Kzd3QDAEk1AIzIsxeU83symEpnIfm100ZfMbKOZ3WNmBfGoQWJn9a56CnLSmT1mRNCliMhpiHkQmFke8Ahwi7u3AHcAM4AFRM4YfniC991sZmVmVlZbWxvrMuVtWFNZz3nTRpOSoiGnRYajmAaBmaUTCYHfufujAO5+yN1D7h4GfgUs7u+97n6nu5e6e2lxcXEsy5S3YX9DB9VNnZqXWGQYi+VVQwbcDZS7+4+OWV5yzGrvBzbHqgaJvTWV9QAsma4gEBmuYnnV0FLgE8AmM1sfXfYt4HozWwA4sAf4bAxrkBhbU9lAYW4Gs8bkBV2KiJymWF419BLQX6PxU7HapsSXu0f7BwrVPyAyjOnuHzltVY2dVDd1qllIZJhTEMhpWx3tH1BHscjwpiCQ07amsl79AyIJQEEgp8XdWb2rniXTC4lcICYiw5WCQE5LZV07Nc1dmp9YJAEoCOS0rNpZB8CFCgKRYU9BIKflpYo6JhZkM7kwJ+hSRORtUhDIKf3oz9v50C9ext0BCIWd1ZX1LJ1RpP4BkQSgIJBT+t/yw7y6p5GVFZHmoE3VzbR29WlaSpEEoSCQk+rqDbHjUCsAd71YCbzRP3CB7h8QSQgDDgIzu9DMPhV9XGxmmo4qCZTXtNAXdhZMyufFijq2HWzhpYo65pWMpCgvM+jyRGQQDCgIzOzbwDeB26KL0oHfxqooGTo2VzcD8E/vP5vs9FR+9uxO1u1tZKnOBkQSxkDPCN4PXA20A7j7AUDTUSWBjVXNjM7NYF7JCD5UOpEnNtbQEwqrf0AkgQw0CHo8csmIA5hZbuxKkqFkU3UzZ08chZnx6aXTMIP0VGPx1MKgSxORQTLQYah/b2a/BPLN7Cbg00RmF5ME1tkTouJwG5edMRaAqUW5XLtoIh09IXIzYzmVhYjE04D+b3b3H5jZZUALMAf4f+6+IqaVSeC21rQQCjtnTxh1dNm/fmh+gBWJSCycMgjMLBX4k7u/C9CXfxI50lF89sRRp1hTRIazU/YRuHsI6DCzv+jbwMwmmdlzZlZuZlvM7CvR5YVmtsLMKqK/C06zdomxjVXNFOVlMm5kVtCliEgMDbSht4vI3MMriF45BODuXz7Je/qAr7r7a2Y2AlgXff9fA8+4+/fM7FbgViKXpsoQs7m6mbMnjNQwEiIJbqBB8GT0Z8DcvQaoiT5uNbNyYAJwDbAsutp9wPMoCIacjp4+Kg63cvlZ44IuRURibKCdxfeZWQYwO7pou7v3DnQjZjYVWAisBcZGQwJ3rzGzMSd4z83AzQCTJ08e6KZkkGw90ELYeVNHsYgkpoHeWbwMqAB+DvwHsMPMLh7ge/OAR4Bb3L1loIW5+53uXurupcXFxQN9mwySTdGO4nPUUSyS8AbaNPRD4N3uvh3AzGYD9wPnnuxNZpZOJAR+5+6PRhcfMrOS6NlACXD49EqXWHptXxNjR2YyVh3FIglvoHcWpx8JAQB330FkvKETskgP491Aubv/6JiXHgduiD6+AXhs4OVKPLwxH7HGExJJBgM9Iygzs7uB/4w+/xiw7hTvWQp8gsjVRuujy74FfI/Inco3AvuAD/1lJUus7apto66tm/MVBCJJYaBB8Hngi8CXAQNWEukrOCF3fym6bn8uHWiBEn+rKxsAdEYgkiQGGgRpwL8daeKJ3m2swegT1Jpd9ZSMymLKaM1HLJIMBtpH8AyQfczzbOB/B78cCZq7s6Yy0j+gG8lEksNAgyDL3duOPIk+1p+LCajicBv17T3qHxBJIgMNgnYzW3TkiZmVAp2xKUmCtHpXPQDnawYykaQx0D6CW4CHzOwAkclpxgPXxawqCczqXfVMyM9mYkH2qVcWkYRw0jMCM3uHmY1z91eBucCDRAaTexrYHYf6JI7CYWftbvUPiCSbUzUN/RLoiT4+n8h9AD8HGoE7Y1iXBGD7oVYaO3rVLCSSZE7VNJTq7g3Rx9cBd7r7I8Ajx9wkJgniSP/Akumaj1gkmZzqjCDVzI6ExaXAs8e8pklrE8xz2w8zvSiXiQW6IEwkmZzqy/x+4AUzqyNyldCLAGY2E2iOcW0SR23dfaytbOCGC6YEXYqIxNlJg8Ddv2tmzwAlwJ/d3aMvpQB/E+viJH5eqqijJxRm+dyxQZciInF2yuYdd1/Tz7IdsSlHgvLstkOMyEqjdKqmkBZJNgO9oUwSWDjsPLutlktmF5Oeqn8SIslG/9cLm6qbqWvr5tJ5/c4aKiIJTkEgPLPtMCkGy2YrCESSkYJAeHbbIc6dUkBBbkbQpYhIAGIWBGZ2j5kdNrPNxyy73cyqzWx99Oc9sdq+DMzB5i42V7foaiGRJBbLM4J7gSv6Wf5jd18Q/XkqhtuXAXhi4wEA9Q+IJLGYBYG7rwQaTrmiBKatu487nt/FkumFzBqTF3Q5IhKQIPoIvmRmG6NNR7poPUC/WllJfXsPt105T6ONiiSxeAfBHcAMYAFQA/zwRCua2c1mVmZmZbW1tfGqL2nUtnbzqxcruersEuZPyg+6HBEJUFyDwN0PuXvI3cPAr4DFJ1n3TncvdffS4uLi+BWZJH76bAXdfWG+dvmcoEsRkYDFNQjMrOSYp+8HNp9oXYmdffUd/NfafVy/eBLTinKDLkdEAhazoaTN7H5gGVBkZlXAt4FlZraAyHSXe4DPxmr7cmLPbDtEX9j53CUzgi5FRIaAmAWBu1/fz+K7Y7U9GbjymhZG52YwIV/zEouI7ixOStsOtjKvZKSuFBIRQEGQdPpCYbYfbGXuuBFBlyIiQ4SCIMnsqW+nuy/MvJKRQZciIkOEgiDJlNe0AjC3RGcEIhKhIEgy5TUtpKUYMzWkhIhEKQiSzLaDrcwoziMzLTXoUkRkiFAQJJnymhbmqVlIRI6hIEgiTR091DR3qaNYRN5EQZBE3ugoVhCIyBsUBEmkvKYFQE1DIvImCoIkcmRoieK8zKBLEZEhREGQRDS0hIj0R0GQJPpCYbYfalWzkIi8hYIgSeyua6enL8zcceooFpE3UxAkiSc31QBw7hRNEy0ib6YgSAKdPSF+s3ovl84dw1TNSCYix1EQJIGHX6uiob2Hmy+eHnQpIjIExSwIzOweMztsZpuPWVZoZivMrCL6W+0UMRYKO3e9WMn8SfksnlYYdDkiMgTF8ozgXuCK45bdCjzj7rOAZ6LPJYb+vOUge+s7+OzF03XZqIj0K2ZB4O4rgYbjFl8D3Bd9fB/wvlhtX8Dd+eXKSqaMzuHyM8cFXY6IDFHx7iMY6+41ANHfY+K8/aSyoaqZ9fubuPHCaaSm6GxARPo3ZDuLzexmMyszs7La2tqgyxmW/vh6NRlpKbxv4YSgSxGRISzeQXDIzEoAor8Pn2hFd7/T3UvdvbS4uDhuBSaKUNh5clMN75xTzMis9KDLEZEhLN5B8DhwQ/TxDcBjcd5+QurqDbH0e8/y4Kv7ji5bW1lPbWs3V8/X2YCInFwsLx+9H1gNzDGzKjO7EfgecJmZVQCXRZ/L27T9YCvVTZ1873+20dzZC8DjGw6Qm5HKpfPUDSMiJ5cWqw929+tP8NKlsdpmsjoyz0BjRy//8fxOvnrZHP5n80HefeY4stI1N7GInFzMgkDip7ymhdyMVC4/cxy/fmkPJSOzaO7s5er544MuTUSGgSF71ZAMXHlNK3NLRvL1K+ZgBv/wxFbyc9JZOrMo6NJEZBhQEAxz7k75wRbmlYygZFQ2N100nbDDlWeVkJGmwysip6amoWGuqrGT1q4+5kUnpP/cshlUNXZw44VTgy1MRIYNBcEw98aE9JEgyMtM4ycfWRhkSSIyzKjtYJgrr2nFDOaM1RSUInJ6FATDXHlNC1MKc8jN1MmdiJweBcEwt+1gy9FmIRGR06EgGGZ+/+p+1u2NjO7d3t3H3oYOBYGIvC1qTxhGWrt6+dYfNlGQm8H//u0l7Kxtwx0FgYi8LTojGEbWVDbQF3ZqW7v5p6fKj7liSB3FInL6dEYwjKzcUUtORiofecdk7lm1m03VzYzMSmNCfnbQpYnIMKYzgmHkxYpazp8+mm9cMYepo3PYWtPC3JKRmotYRN4WBcEwsa++gz31HVw0q4is9FT++QPnAHCG+gdE5G1S09AwsbIiMl3nxbMjs7WdP2M09316MfPGqX9ARN4eBcEwsXJHLRPys5lWlHt02SWzNYWniLx9ahoaBnpDYV7eVc/Fs4vVHyAigy6QMwIz2wO0AiGgz91Lg6hjuFi/v4m27j4unqX5BURk8AXZNPROd68LcPtDWm8oTGtXH9npqbywvZYUgws00YyIxID6CIYgd+ejv1rDq3sajy5bNDmfUdnpAVYlIokqqCBw4M9m5sAv3f3O41cws5uBmwEmT54c5/KC9eSmGl7d08gnlkxhQkE2nT0hLp03JuiyRCRBBRUES939gJmNAVaY2TZ3X3nsCtFwuBOgtLTUgygyCN19Ib7/9DbmjhvB7VefSWqKOodFJLYCuWrI3Q9Efx8G/gAsDqKOoeg/V+9lf0Mnf3fVPIWAiMRF3IPAzHLNbMSRx8C7gc3xrmMoauro4d+fqeCS2cVcNEv3CIhIfATRNDQW+EP0evg04L/c/ekA6hhS3J1/fLKctu4+vvWeeUGXIyJJJO5B4O6VwPx4b3coC4edv39sMw+vq+KL75zBHA0bISJxpMtHAxYOO7c9uokHy/bzuUtm8LV3zwm6JBFJMgqCADR19HD/K/vZfKCZTVXN7Gvo4MvLZ/K3l83WEBIiEncKggDc+sgmnt5ykIkF2Zw5fiRfWj6TD5dOCrosEUlSCoI421PXzp+2HuQLy2bwjSvmBl2OiIhGH423e1btJj0lhb++YGrQpYiIAAqCuGps7+GhsiquWTCeMSOzgi5HRARQEMTV79bupbM3xGcumh50KSIiR6mPIIb2N3SwoaqJyYU5TCrI4d6X93Lx7GLdJyAiQ4qC4DRVHGrllgfXc+VZ47jp4ulkpqUCkUtDH1t/gMfWV/Pavqa3vO+mi6bFu1QRkZNSEJx3cSCrAAAI6klEQVSGxvYePvObMg42d/GDP+/g4XVVfGHZTNbsrueJjTX09IWZO24E37hiDhfOLOJAUxe7atswgws1uYyIDDEKgr9QbyjMF373GjVNXTzw2SW0dfVx+39v4RuPbCQ3I5UPl07k+sWTOXP8qKPvOWdigAWLiJyCgqAfNc2d3LtqD3vrO2jv6aOzJ0RRXiZTi3LZ39jB6sp6fvih+SyaXADA01+5mHV7Gzl74ijyMvWfVESGF31rHeNgcxf/8fxOHnhlP2F3phfnkpORRlZ6CjsOt/LMtkP0hpzPXTKDD577xp/5GWkpnD9jdICVi4icvqQJgprmTsr2NDI+P4sJ+TmMGZFJSnTil46ePn7xQiV3rtxFX8j5UOlEvrBsJpMKc970GaGw09zZS2FuRhC7ICISE0kRBIdaunjfz1dxqKX76LKcjFTmjBvBnLEjeH57LQdbunjvOSV884q5bwmAI1JTTCEgIgkn4YOgqzfEzf+5jtauPu791Dtwh6qmTnYdbqO8poWnNtUwrTiPn350Ie+YWhh0uSIicRdIEJjZFcC/AanAXe7+vVhsx9259ZGNbNjfxC8+fi7L5oyJxWZERIa1IOYsTgV+DlwJnAFcb2ZnxGJbv1xZyR/XH+Crl83mirPGxWITIiLDXhBjDS0Gdrp7pbv3AA8A18RiQ+Pzs7n23Il8afnMWHy8iEhCCKJpaAKw/5jnVcB5sdjQ1fPHc/X88bH4aBGRhBHEGUF/czH6W1Yyu9nMysysrLa2Ng5liYgkpyCCoAo4dl7GicCB41dy9zvdvdTdS4uLi+NWnIhIsgkiCF4FZpnZNDPLAD4CPB5AHSIiQgB9BO7eZ2ZfAv5E5PLRe9x9S7zrEBGRiEDuI3D3p4Cngti2iIi8maaqFBFJcgoCEZEkpyAQEUly5v6WS/iHHDOrBfae5tuLgLpBLGcoS5Z9TZb9hOTZ12TZT4jvvk5x91Nefz8sguDtMLMydy8Nuo54SJZ9TZb9hOTZ12TZTxia+6qmIRGRJKcgEBFJcskQBHcGXUAcJcu+Jst+QvLsa7LsJwzBfU34PgIRETm5ZDgjEBGRk0joIDCzK8xsu5ntNLNbg65nsJjZJDN7zszKzWyLmX0lurzQzFaYWUX0d0HQtQ4GM0s1s9fN7Ino82lmtja6nw9GBy8c9sws38weNrNt0WN7fgIf07+N/tvdbGb3m1lWohxXM7vHzA6b2eZjlvV7HC3i36PfURvNbFEQNSdsEMRzSswA9AFfdfd5wBLgi9F9uxV4xt1nAc9EnyeCrwDlxzz/PvDj6H42AjcGUtXg+zfgaXefC8wnss8Jd0zNbALwZaDU3c8iMvjkR0ic43ovcMVxy050HK8EZkV/bgbuiFONb5KwQUAcp8SMN3evcffXoo9biXxhTCCyf/dFV7sPeF8wFQ4eM5sIXAXcFX1uwHLg4egqibKfI4GLgbsB3L3H3ZtIwGMalQZkm1kakAPUkCDH1d1XAg3HLT7RcbwG+I1HrAHyzawkPpW+IZGDoL8pMScEVEvMmNlUYCGwFhjr7jUQCQtgTHCVDZqfAN8AwtHno4Emd++LPk+U4zodqAV+HW0Gu8vMcknAY+ru1cAPgH1EAqAZWEdiHtcjTnQch8T3VCIHwYCmxBzOzCwPeAS4xd1bgq5nsJnZe4HD7r7u2MX9rJoIxzUNWATc4e4LgXYSoBmoP9H28WuAacB4IJdIE8nxEuG4nsqQ+PecyEEwoCkxhyszSycSAr9z90ejiw8dOa2M/j4cVH2DZClwtZntIdK0t5zIGUJ+tEkBEue4VgFV7r42+vxhIsGQaMcU4F3Abnevdfde4FHgAhLzuB5xouM4JL6nEjkIEnZKzGg7+d1Aubv/6JiXHgduiD6+AXgs3rUNJne/zd0nuvtUIsfvWXf/GPAccG10tWG/nwDufhDYb2ZzoosuBbaSYMc0ah+wxMxyov+Wj+xrwh3XY5zoOD4OfDJ69dASoPlIE1JcuXvC/gDvAXYAu4C/C7qeQdyvC4mcPm4E1kd/3kOk/fwZoCL6uzDoWgdxn5cBT0QfTwdeAXYCDwGZQdc3SPu4ACiLHtc/AgWJekyB7wDbgM3AfwKZiXJcgfuJ9H30EvmL/8YTHUciTUM/j35HbSJyJVXca9adxSIiSS6Rm4ZERGQAFAQiIklOQSAikuQUBCIiSU5BICKS5BQEktDMLGRm64/5Oendumb2OTP75CBsd4+ZFZ3G+y43s9vNrMDMnnq7dYgMRNqpVxEZ1jrdfcFAV3b3X8SymAG4iMiNVRcDqwKuRZKEgkCSUnTYigeBd0YXfdTdd5rZ7UCbu//AzL4MfI7IsN9b3f0jZlYI3EPk5qcO4GZ332hmo4ncSFRM5KYoO2ZbHycy7HIGkcEBv+DuoePquQ64Lfq51wBjgRYzO8/dr47FfwORI9Q0JIku+7imoeuOea3F3RcDPyMyhtHxbgUWuvs5RAIBInfEvh5d9i3gN9Hl3wZe8siAcY8DkwHMbB5wHbA0emYSAj52/Ibc/UEiYwttdvezidxxu1AhIPGgMwJJdCdrGrr/mN8/7uf1jcDvzOyPRIZ8gMjwHh8EcPdnzWy0mY0i0pTzgejyJ82sMbr+pcC5wKuRYXXI5sQDx80iMtQAQI5H5poQiTkFgSQzP8HjI64i8gV/NfD3ZnYmJx82uL/PMOA+d7/tZIWYWRlQBKSZ2VagxMzWA3/j7i+efDdE3h41DUkyu+6Y36uPfcHMUoBJ7v4ckYlx8oE8YCXRph0zWwbUeWQuiGOXX0lkwDiIDDB2rZmNib5WaGZTji/E3UuBJ4n0D/wLkUESFygEJB50RiCJLjv6l/URT7v7kUtIM81sLZE/iK4/7n2pwG+jzT5GZC7dpmhn8q/NbCORzuIjQwt/B7jfzF4DXiAy1DLuvtXM/i/w52i49AJfBPb2U+siIp3KXwB+1M/rIjGh0UclKUWvGip197qgaxEJmpqGRESSnM4IRESSnM4IRESSnIJARCTJKQhERJKcgkBEJMkpCEREkpyCQEQkyf1/Nfqh1FtHejsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcb6b08dc88>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed (hh:mm:ss.ms) 1:26:07.014596\n"
     ]
    }
   ],
   "source": [
    "from workspace_utils import active_session\n",
    "\n",
    "if TRAIN:\n",
    "    with active_session():\n",
    "        print(\"Train Agent\")\n",
    "        scores = train(agent, env, n_episodes=200, max_t=1000)\n",
    "\n",
    "    # Plot the scores\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    plt.plot(np.arange(len(scores)), scores)\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Episode #')\n",
    "    plt.show()\n",
    "\n",
    "    time_elapsed = datetime.now() - start_time \n",
    "    print('Time elapsed (hh:mm:ss.ms) {}'.format(time_elapsed))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def execute_policy(env, agent, num_agents, step):\n",
    "    brain_name = env.brain_names[0]\n",
    "    env_info = env.reset(train_mode=False)[brain_name]  # reset the environment\n",
    "    states = env_info.vector_observations  # get the current state (for each agent)\n",
    "    scores = np.zeros(num_agents)  # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = agent.act(states)  # select an action (for each agent)\n",
    "        env_info = env.step(actions)[brain_name]  # send the action to the environment\n",
    "        next_states = env_info.vector_observations  # get next state (for each agent)\n",
    "        rewards = env_info.rewards  # get reward (for each agent)\n",
    "        dones = env_info.local_done  # see if episode finished\n",
    "        scores += rewards  # update the score (for each agent)\n",
    "        states = next_states  # roll over states to next time step\n",
    "        if np.any(dones):  # exit loop if episode finished\n",
    "            break\n",
    "    print(f\"Total score (averaged over agents) for episode {step}: {np.mean(scores)}\")\n",
    "    return np.mean(scores)\n",
    "\n",
    "\n",
    "def evaluate_agent(\n",
    "    agent, env, num_agents, checkpoint_pth=\".\", num_episodes=100, min_score=30\n",
    "):\n",
    "    score_lst = []\n",
    "    agent.actor_local.load_state_dict(\n",
    "        torch.load(\n",
    "            f\"{checkpoint_pth}/checkpoint_env_solved_actor.pth\",\n",
    "            map_location=torch.device('cpu')\n",
    "        ),\n",
    "    )\n",
    "    agent.critic_local.load_state_dict(\n",
    "        torch.load(\n",
    "            f\"{checkpoint_pth}/checkpoint_env_solved_critic.pth\",\n",
    "            map_location=torch.device('cpu')\n",
    "        ),\n",
    "    )\n",
    "    for i in range(num_episodes):\n",
    "        score_lst.append(execute_policy(env, agent, num_agents, i))\n",
    "\n",
    "    assert (\n",
    "        np.mean(score_lst) >= min_score\n",
    "    ), f\"Environment not solved: Expected score >= {min_score}, found {np.mean(score_lst)}\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) for episode 0: 38.4824991398491\n"
     ]
    }
   ],
   "source": [
    "evaluate_agent(agent, env, num_agents, num_episodes=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}