<jupyter_start><jupyter_text>Continuous Control---Train an agent to reach its robot arm to the target!        Single Agent      20 Agents               ---  Table of Contents---* Table of Contents0. [Import Necessary Modules](0.-Import-Necessary-Modules)1. [Start and Explore the Environment](1.-Start-and-Explore-the-Environment)    * 1.1. [Start the Environment](1.1.-Start-the-Environment)    * 1.2. [Examine the State and Action Spaces](1.2.-Examine-the-State-and-Action-Spaces)    * 1.3. [Take Random Actions in the Environment](1.3.-Take-Random-Actions-in-the-Environment)2. [It's Your Turn!](2.-It's-Your-Turn!)    * 1.1. [Tune Your Hyperparameters!](2.1.-Tune-Your-Hyperparameters!)    * 1.2. [Instantiate Your Agent](2.2.-Instantiate-Your-Agent)    * 1.3. [Train Your Agent](2.3.-Train-Your-Agent)    * 1.4. [Plot the Training Scores](2.4.-Plot-the-Training-Scores)    * 1.5. [Save the Agent](2.5.-Save-the-Agent)3. [Load & Evaluate the Agent](3.-Load-&-Evaluate-Your/Pretrained-Agent)    * 1.1. [Load the Agent](3.1.-Load-the-Agent)    * 1.2. [Watch the Agent!](3.2.-Watch-the-Agent!)    * 1.3. [More Accurate Evaluation](3.3.-More-Accurate-Evaluation)4. [Finishing Up](4.-Finishing-Up)* [Thank You!](-Thank-you!) ---  [0.](Table-of-Contents) Import Necessary Modules---Just run the next cell to prepare all the modules needed.<jupyter_code># DO NOT CHANGE THE CODE HERE
import random
from collections import deque

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import torch

from ddpg_agent import Agent
from train import ddpg

from unityagents import UnityEnvironment

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')<jupyter_output><empty_output><jupyter_text>---  [1.](Table-of-Contents) Start and Explore the Environment---In this section, you will load the environment and explore it with random actions. Exploration is optional, but loading the environment is necessary.  [1.1.](Table-of-Contents) Start the Environment---Here, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.- **Mac**: `"path/to/Reacher.app"`- **Windows** (x86): `"path/to/Reacher_Windows_x86/Reacher.exe"`- **Windows** (x86_64): `"path/to/Reacher_Windows_x86_64/Reacher.exe"`- **Linux** (x86): `"path/to/Reacher_Linux/Reacher.x86"`- **Linux** (x86_64): `"path/to/Reacher_Linux/Reacher.x86_64"`For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:```env = UnityEnvironment(file_name="Reacher.app")```<jupyter_code>env = UnityEnvironment(
    file_name="Reacher_Environment/twenty_agents/Reacher_Windows_x86_64/Reacher.exe",    # <--- File Path to the Environment file
    seed=0
)

# DO NOT EDIT THE LINES BELOW
brain_name = env.brain_names[0]
env_info = env.reset(train_mode=False)[brain_name]
num_agents = len(env_info.agents)
action_size = env.brains[brain_name].vector_action_space_size
state_size = env_info.vector_observations.shape[1]<jupyter_output>INFO:unityagents:
'Academy' started successfully!
Unity Academy name: Academy
        Number of Brains: 1
        Number of External Brains : 1
        Lesson number : 0
        Reset Parameters :
		goal_speed -> 1.0
		goal_size -> 5.0
Unity brain name: ReacherBrain
        Number of Visual Observations (per agent): 0
        Vector Observation space type: continuous
        Vector Observation space size (per agent): 33
        Number of stacked Vector Observation: 1
        Vector Action space type: continuous
        Vector Action space size (per agent): 4
        Vector Action descriptions: , , ,<jupyter_text>[1.2.](Table-of-Contents) Examine the State and Action Spaces---In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of the agent is to maintain its position at the target location for as many time steps as possible.The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.Run the code cell below to print some information about the environment.<jupyter_code># number of agents
print('Number of agents:', num_agents, end='\n\n')

# size of each action
print('Size of each action:', action_size, end='\n\n')

# examine the state space
states = env_info.vector_observations
print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))
print('The state for the first agent looks like:\n', states[0], end='\n\n')
print('\nShapes:')
print("rewards", len(env_info.rewards))
print("vector_observations", env_info.vector_observations.shape)
print("local_done", len(env_info.local_done))<jupyter_output>Number of agents: 20

Size of each action: 4

There are 20 agents. Each observes a state with length: 33
The state for the first agent looks like:
 [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00
 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00
  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00
  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08
  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00
  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00
 -1.68164849e-01]


Shapes:
rewards 20
vector_observations (20, 33)
local_done 20<jupyter_text>[1.3.](Table-of-Contents) Take Random Actions in the Environment---In the next code cell, you will watch the agent's performance when it selects an action at random with each time step. You should be able to observe the agent on the window as it moves through the environment.<jupyter_code>env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    
states = env_info.vector_observations                  # get the current state (for each agent)
scores = np.zeros(num_agents)                          # initialize the score (for each agent)
while True:
    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)
    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1
    env_info = env.step(actions)[brain_name]           # send all actions to tne environment
    next_states = env_info.vector_observations         # get next state (for each agent)
    rewards = env_info.rewards                         # get reward (for each agent)
    dones = env_info.local_done                        # see if episode finished
    scores += env_info.rewards                         # update the score (for each agent)
    states = next_states                               # roll over states to next time step
    if np.any(dones):                                  # exit loop if episode finished
        break
print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))<jupyter_output>Total score (averaged over agents) this episode: 0.07749999826774001<jupyter_text>---  [2.](Table-of-Contents) It's Your Turn!---Here, you can train your own agent to solve the environment!  [2.1.](Table-of-Contents) Tune Your Hyperparameters!---All necessary hyperparameters and training parameters for the agent are provided below. Try fiddling with the values to get a better performance out of it!<jupyter_code>hyperparams = {
    # Reproducibility
    'seed'                : 0,        # random seed for reproducible results
    
    # Agent basic parameters
    'batch_size'          : 256,      # batch size for each learning step
    'buffer_size'         : int(1e6), # up to how many recent experiences to keep
    'start_since'         : 256,      # how many experiences to collect before starting learning
    'gamma'               : 0.99,     # discount factor
    'update_every'        : 1,        # update step frequency
    'n_updates'           : 10,       # number of updates per update step
    'tau'                 : 1e-3,     # soft-update parameter [0, 1]
    
    'actor_lr'            : 5e-4,     # learning rate for the actor network
    'critic_lr'           : 5e-4,     # learning rate for the critic network
    'clip'                : 1,        # gradient clipping to prevent gradient spikes
    'weight_decay'        : 0,        # weight decay for the *critic* network

    'distributional'      : True,     # whether to use distributional learning

    # Prioritized Experience Replay Parameters
    'priority_eps'        : 1e-3,     # base priority in order to ensure nonzero priorities
    'a'                   : 1.,       # priority exponent parameter [0, 1]

    # n-step Bootstrap Target parameter
    'n_multisteps'        : 4,        # number of steps to bootstrap
    'separate_experiences': False,    # whether to store experiences with no overlap

    # Distributional Learning parameters
    'v_min'               : 0,        # minimum value for support
    'v_max'               : 10,       # maximum value for support
    'n_atoms'             : 51,       # number of atoms for distribution

    # Noisy Layer parameters
    'initial_sigma'       : 0.050,    # initial noisy parameter value
    'linear_type'         : 'noisy',  # either 'linear' or 'noisy'
    'factorized'          : False     # whether to use factorized gaussian noise or not(independent gaussian noise)
}

### -------------------------------------------------------------------------------------------------------------- ###

train_params = {
    'n_episodes'           : 200,   # number of total episodes to train
    'continue_after_solved': False, # whether to keep training even after the environment is solved
    
    # Exploration using gaussian noise
    'eps_start'            : 0.0,   # initial epsilon value
    'eps_min'              : 0.0,   # minimum value for epsilon
    'eps_decay'            : 0.0,   # epsilon decay rate

    # Importance-Sampling Weight parameter for Prioritized Experience Replay
    'beta_start'           : 1.,    # starting value 
    'beta_end'             : 1.     # end value
}<jupyter_output><empty_output><jupyter_text>[2.2.](Table-of-Contents) Instantiate Your Agent---After tuning your hyperparameters, the agent instance must be created.To do so, simply run the cell below!<jupyter_code>agent = Agent(state_size, action_size, num_agents, **hyperparams)<jupyter_output><empty_output><jupyter_text>[2.3.](Table-of-Contents) Train Your Agent---With your agent and your training settings, the next cell will run the training session for your agent!(This may take a long time! It can take hours to train a good agent, depending on your PC performance.)<jupyter_code>scores = ddpg(env, agent, **train_params)<jupyter_output>Episode 0 | Total Steps (all agents): 0 | Current Average Score:  0.000 | Last 100 Average Score:  0.000 | Epsilon: 0.0000 | A: 1.0000 | Beta: 1.0000 *** timestep 1 ***
Episode 0 | Total Steps (all agents): 0 | Current Average Score:  0.000 | Last 100 Average Score:  0.000 | Epsilon: 0.0000 | A: 1.0000 | Beta: 1.0000 *** timestep 2 ***
Episode 0 | Total Steps (all agents): 0 | Current Average Score:  0.000 | Last 100 Average Score:  0.000 | Epsilon: 0.0000 | A: 1.0000 | Beta: 1.0000 *** timestep 3 ***
Episode 0 | Total Steps (all agents): 0 | Current Average Score:  0.000 | Last 100 Average Score:  0.000 | Epsilon: 0.0000 | A: 1.0000 | Beta: 1.0000 *** timestep 4 ***
Episode 0 | Total Steps (all agents): 0 | Current Average Score:  0.000 | Last 100 Average Score:  0.000 | Epsilon: 0.0000 | A: 1.0000 | Beta: 1.0000 *** timestep 5 ***
Episode 0 | Total Steps (all agents): 0 | Current Average Score:  0.000 | Last 100 Average Score:  0.000 | Epsilon: 0.0000 | A: 1.0000 | Beta: 1.0000 *** [...]<jupyter_text>[2.4.](Table-of-Contents) Plot the Training Scores---The code cell below will plot the scores your agent received during its training.This allows you to see your agent's learning performance at a glance!<jupyter_code>fig = plt.figure(figsize=(10, 4), facecolor='lightsteelblue')
ax = fig.add_subplot(111)
ax.grid(True)
ax.set_facecolor('slategray')
ax.plot(np.arange(len(scores)), scores,
        alpha=0.7, linewidth=2, color="blue", label="Raw Train Score")
ax.plot(np.arange(len(scores)), np.array([np.mean(scores[max(0,i-100):i]) for i in range(1, len(scores)+1)]),
        alpha=0.7, linewidth=2, color="orange", label="Mean Score over up to 100 Episodes")
ax.legend()
xlim = ax.get_xlim()
ax.plot(xlim, (30, 30), color="black")
ax.set_xlim(*xlim)
ax.set_ylabel('Average Score')
ax.set_xlabel('Episode #');<jupyter_output><empty_output><jupyter_text>[2.5.](Table-of-Contents) Save the Agent---If you are satisfied with your agent, you would want to save the agent, scores, and your hyperparameter settings so that it won't be lost.**_Before running the cell below_**, change the file name parameter `f` to the file name you want.<jupyter_code>torch.save(

    f='pretrained.pth',    # <--- Name of the save

    obj={'hyperparams': hyperparams, 'train_params': train_params,
         'train_scores': scores,
         'actor_dict': agent.actor_local.state_dict(),
         'critic_dict': agent.critic_local.state_dict()}
)<jupyter_output><empty_output><jupyter_text>---  [3.](Table-of-Contents) Load & Evaluate Your/Pretrained Agent---Once the training is done and you want to test your agent, or if you want to see how well my agent performs, you can load the model and run it in this section.  [3.1.](Table-of-Contents) Load the Agent---First, you need to load the agent you wish to run, unless the agent is already up and running.**_Before running the next cell_**, change the file name parameter `f` in `torch.load()` to the name of the saved file you want to load.<jupyter_code>saved = torch.load(

            f="pretrained.pth",    # <--- Name of the file to be loaded

            map_location=device

        )

agent = Agent(state_size, action_size, num_agents, **saved['hyperparams'])
agent.actor_local.load_state_dict(saved['actor_dict'])
agent.critic_local.load_state_dict(saved['critic_dict'])<jupyter_output><empty_output><jupyter_text>[3.2.](Table-of-Contents) Watch the Agent!---Watch the trained agent performing the task!<jupyter_code>env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    
states = env_info.vector_observations                  # get the current state (for each agent)
scores = np.zeros(num_agents)                          # initialize the score (for each agent)
t = 0                                                  # initialize the time step count
while True:
    actions = agent.act(states)                        # select an action (for each agent)
    env_info = env.step(actions)[brain_name]           # send all actions to tne environment
    next_states = env_info.vector_observations         # get next state (for each agent)
    rewards = env_info.rewards                         # get reward (for each agent)
    dones = env_info.local_done                        # see if episode finished
    scores += env_info.rewards                         # update the score (for each agent)
    states = next_states                               # roll over states to next time step
    t += 1
    print("\rCurrent Average Score at timestep {}: {}".format(t, np.mean(scores)), end='')
    if np.any(dones):                                  # exit loop if episode finished
        break
print("\nFinal Average Score: {}".format(np.mean(scores)))<jupyter_output>Current Average Score at timestep 1001: 39.353499120380734
Final Average Score: 39.35349912038073<jupyter_text>[3.3.](Table-of-Contents) More Accurate Evaluation---To get a better prediction for the agent's performance, you can run multiple episodes and get the average score.Set `n_episodes` in the code cell below to the number of episodes you want to test for, and run the code!<jupyter_code>n_episodes = 100    # <--- Number of Episodes to test the agent for



score_avgs = []
report_str_format = "\rEpisode {} | Current Episode Average Score: {:.3f} | Previous Episode Average Score: {:.3f} | Overall Average Score: {:.3f}"
for i in range(n_episodes):
    env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    
    states = env_info.vector_observations                  # get the current state (for each agent)
    scores = np.zeros(num_agents)                          # initialize the score (for each agent)
    while True:
        actions = agent.act(states)                        # select an action (for each agent)
        env_info = env.step(actions)[brain_name]           # send all actions to tne environment
        next_states = env_info.vector_observations         # get next state (for each agent)
        rewards = env_info.rewards                         # get reward (for each agent)
        dones = env_info.local_done                        # see if episode finished
        scores += env_info.rewards                         # update the score (for each agent)
        states = next_states                               # roll over states to next time step
        print(report_str_format.format(i+1, np.mean(scores), score_avgs[-1] if score_avgs else 0, np.mean(score_avgs) if score_avgs else 0), end='')
        if np.any(dones):                                  # exit loop if episode finished
            break
    if (i+1) % 10 == 0:
        print()
    score_avgs.append(np.mean(scores))
print("\nFinal Average Score: {}".format(np.mean(score_avgs)))<jupyter_output>Episode 10 | Current Episode Average Score: 39.151 | Previous Episode Average Score: 39.436 | Overall Average Score: 39.310
Episode 20 | Current Episode Average Score: 39.458 | Previous Episode Average Score: 39.344 | Overall Average Score: 39.309
Episode 30 | Current Episode Average Score: 39.319 | Previous Episode Average Score: 39.357 | Overall Average Score: 39.320
Episode 40 | Current Episode Average Score: 38.740 | Previous Episode Average Score: 39.345 | Overall Average Score: 39.333
Episode 50 | Current Episode Average Score: 39.320 | Previous Episode Average Score: 39.272 | Overall Average Score: 39.319
Episode 60 | Current Episode Average Score: 39.337 | Previous Episode Average Score: 39.409 | Overall Average Score: 39.321
Episode 70 | Current Episode Average Score: 39.327 | Previous Episode Average Score: 39.174 | Overall Average Score: 39.317
Episode 80 | Current Episode Average Score: 39.328 | Previous Episode Average Score: 39.420 | Overall Average Score: 39.315
Episode [...]<jupyter_text>---  [4.](Table-of-Contents) Finishing Up---When you are done, run the cell below to close the environment.<jupyter_code>env.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10954731645835522, 'fraction_numerical': 0.05109757378426636, 'mean_word_length': 3.039669070852779, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 28, 'word_list_counts': {'cursed_substrings.json': 54, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '605072', 'license': 'MIT', 'n_tokens_mistral': 6363, 'n_tokens_neox': 5518, 'n_words': 1910, 'path': 'Continuous_Control.ipynb', 'repo_name': 'wytyang00/Udacity-DRL-Nanodegree-Continuous-Control-Project', 'size': '19042'}