<jupyter_start><jupyter_text>Navigation---In this notebook, you will learn how to use the Unity ML-Agents environment for the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893). 1. Start the EnvironmentWe begin by importing some necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/).<jupyter_code>from unityagents import UnityEnvironment
import numpy as np<jupyter_output><empty_output><jupyter_text>Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.- **Mac**: `"path/to/Banana.app"`- **Windows** (x86): `"path/to/Banana_Windows_x86/Banana.exe"`- **Windows** (x86_64): `"path/to/Banana_Windows_x86_64/Banana.exe"`- **Linux** (x86): `"path/to/Banana_Linux/Banana.x86"`- **Linux** (x86_64): `"path/to/Banana_Linux/Banana.x86_64"`- **Linux** (x86, headless): `"path/to/Banana_Linux_NoVis/Banana.x86"`- **Linux** (x86_64, headless): `"path/to/Banana_Linux_NoVis/Banana.x86_64"`For instance, if you are using a Mac, then you downloaded `Banana.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:```env = UnityEnvironment(file_name="Banana.app")```<jupyter_code>env = UnityEnvironment(file_name="...")<jupyter_output><empty_output><jupyter_text>Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python.<jupyter_code># get the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]<jupyter_output><empty_output><jupyter_text>2. Examine the State and Action SpacesThe simulation contains a single agent that navigates a large environment.  At each time step, it has four actions at its disposal:- `0` - walk forward - `1` - walk backward- `2` - turn left- `3` - turn rightThe state space has `37` dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction.  A reward of `+1` is provided for collecting a yellow banana, and a reward of `-1` is provided for collecting a blue banana. Run the code cell below to print some information about the environment.<jupyter_code># reset the environment
env_info = env.reset(train_mode=True)[brain_name]

# number of agents in the environment
print('Number of agents:', len(env_info.agents))

# number of actions
action_size = brain.vector_action_space_size
print('Number of actions:', action_size)

# examine the state space 
state = env_info.vector_observations[0]
print('States look like:', state)
state_size = len(state)
print('States have length:', state_size)<jupyter_output><empty_output><jupyter_text>3. Take Random Actions in the EnvironmentIn the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.Once this cell is executed, you will watch the agent's performance, if it selects an action (uniformly) at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!<jupyter_code>env_info = env.reset(train_mode=False)[brain_name] # reset the environment
state = env_info.vector_observations[0]            # get the current state
score = 0                                          # initialize the score
while True:
    action = np.random.randint(action_size)        # select an action
    env_info = env.step(action)[brain_name]        # send the action to the environment
    next_state = env_info.vector_observations[0]   # get the next state
    reward = env_info.rewards[0]                   # get the reward
    done = env_info.local_done[0]                  # see if episode has finished
    score += reward                                # update the score
    state = next_state                             # roll over the state to next time step
    if done:                                       # exit loop if episode finished
        break
    
print("Score: {}".format(score))<jupyter_output><empty_output><jupyter_text>When finished, you can close the environment.<jupyter_code>env.close()<jupyter_output><empty_output><jupyter_text>4. It's Your Turn!Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:```pythonenv_info = env.reset(train_mode=True)[brain_name]```RESTART KERNEL BEFORE STARTING<jupyter_code># Q Network Model 

import torch
import torch.nn as nn


class QNN(nn.Module):
    def __init__(self, state_size, action_size, seed, config):
        
        super(QNN, self).__init__()
        self.seed = torch.manual_seed(seed)

        # TODO Build model generator per configuration.  
        self.config = config 
        self.fc1_units = self.config['fc1_units']
        self.fc2_units = self.config['fc2_units']


        # Create sequential container with two hidden layers 
        self.model = nn.Sequential(nn.Linear(state_size, self.fc1_units),
                                   nn.ReLU(),
                                   nn.Linear(self.fc1_units, self.fc2_units),
                                   nn.ReLU(),
                                   nn.Linear(self.fc2_units, action_size))

    def forward(self, state):
        
        return self.model.forward(state)

# Deep Q Learning Agent

import matplotlib.pyplot as plt
import numpy as np

from dqn_agent import QLAgent
from collections import deque
from unityagents import UnityEnvironment

    
# Initilize Q-Learning Agent with the following inputs:   
# TODO rewrite network config to programatically build network (QNN in model.py) based on these settings
network_config = {  #   network_config (dict) = hidden layer network configuration
    'layers': 2,
    'fc1_units': 64,
    'fc2_units': 64,
} 
state_size = 37     # state_size (int)      = 37 [State space with `37` dimensions that contains the agent's velocity and ray-based perception of objects around agent's forward direction]
action_size = 4     # action_size (int)     = 4  [Discrete 0 (forward), 1 (back), 2 (turn left), 3 (turn right)]
seed = 0            #   seed  (int)           = 0  []
agent = QLAgent(state_size, action_size, seed, network_config)

# Initialize Unity Environment
env = UnityEnvironment(file_name="Banana_Windows_x86_64\Banana_Windows_x86_64\Banana.exe")

# get the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]

def dqn(n_episodes=1800, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):
    """Deep Q-Learning.
    
    Params
    ======
        n_episodes (int): maximum number of training episodes
        max_t (int): maximum number of timesteps per episode
        eps_start (float): starting value of epsilon, for epsilon-greedy action selection
        eps_end (float): minimum value of epsilon
        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon
    """
    scores = []                        # list containing scores from each episode
    avg_scores = []                    # List of average score per 100 episodes
    scores_window = deque(maxlen=100)  # last 100 scores
    eps = eps_start                    # initialize epsilon
    for i_episode in range(1, n_episodes+1):
        env_info = env.reset(train_mode=True)[brain_name] # reset the environment
        state = env_info.vector_observations[0]  
        score = 0
        for t in range(max_t):
            # Update variables used for next step
            action = agent.act(state, eps)
            env_info = env.step(action)[brain_name]        # send the action to the environment
            next_state = env_info.vector_observations[0]   # get the next state
            reward = env_info.rewards[0]                   # get the reward
            done = env_info.local_done[0] 

            # Step through Q Learning agent
            agent.step(state, action, reward, next_state, done)
            state = next_state

            # Update reward
            score += reward
            if done:
                break 

        
        scores_window.append(score)       # save most recent score
        scores.append(score)              # save most recent score
        eps = max(eps_end, eps_decay*eps) # decrease epsilon
        print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end="")
        if i_episode % 100 == 0:
            print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))
            avg_scores.append(np.mean(scores_window))

    torch.save(agent.qnetwork_local.state_dict(), 'optimized_weights.pth')

    return scores, avg_scores


# Initialize interation between QLearning agent and environment 
scores, avg_scores = dqn()

# plot the scores
fig = plt.figure()

# define scores axis 
scores_ax = fig.add_subplot(1,2,1)
scores_ax.plot(range(len(scores)), scores)
scores_ax.set_xlabel('Episode')
scores_ax.set_title('Episodic scores')
scores_ax.set_ylabel('Reward')


avg_scores_ax = fig.add_subplot(1,2,2)
avg_scores_x = [i*100 for i in range(len(avg_scores))]
avg_scores_ax.plot(avg_scores_x, avg_scores, '-o')
avg_scores_ax.set_xlabel('Episode')
avg_scores_ax.set_title('Average Scores Every 100 Episodes')

plt.show()

# Load weights and test agent. 

episodes = 100
weights_file_path = 'trained_data/optimized_weights_ddqn.pth'
print(f'Starting brain testing of file {weights_file_path} for {episodes} episodes')

# Initialize new agent with trained weights
network_config = {  #   network_config (dict) = hidden layer network configuration
'layers': 2,
'fc1_units': 64,
'fc2_units': 64,
} 
state_size = 37     # state_size (int)      = 37 [State space with `37` dimensions that contains the agent's velocity and ray-based perception of objects around agent's forward direction]
action_size = 4     # action_size (int)     = 4  [Discrete 0 (forward), 1 (back), 2 (turn left), 3 (turn right)]
seed = 1            #   seed  (int)           = 0  []
agent = QLAgent(state_size, action_size, seed, network_config)

agent.qnetwork_local.load_state_dict(torch.load(weights_file_path))

scores = []                        # list containing scores from each episode
scores_window = deque(maxlen=100)  # last 10 scores

for i_episode in range(episodes):
    # get the default brain
    brain_name = env.brain_names[0]
    env_info = env.reset(train_mode=False)[brain_name] # reset the environment setting train mode to Falase
    state = env_info.vector_observations[0]            # get the current state
    score = 0                                          # initialize the score
    while True:
        action = agent.act(state).item()
        env_info = env.step(action)[brain_name]
        next_state = env_info.vector_observations[0]
        reward = env_info.rewards[0]
        done = env_info.local_done[0]
        score += reward
        state = next_state
        if done:
            break

    print("\rScore: {}".format(score))

    scores_window.append(score)       # save most recent score
    scores.append(score)              # save most recent score
    print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end="")
    if i_episode % 10 == 0:
        print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))

# plot the scores
fig = plt.figure()

# define scores axis 
scores_ax = fig.add_subplot(1,1,1)
reward_data = scores_ax.plot(range(len(scores)), scores, label = 'Reward')

average_label = f'Mean {np.mean(scores)}'
average = scores_ax.plot(range(len(scores)), [np.mean(scores)]*len(scores), label=average_label, linestyle = '--')
scores_ax.set_xlabel('Episode')
legend = scores_ax.legend(loc='upper right')
scores_ax.set_title('Reward Output Using Trained Weights')
scores_ax.set_ylabel('Reward')


plt.show()
    



env.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0974285257946015, 'fraction_numerical': 0.015173295000798594, 'mean_word_length': 3.3093599449415003, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '376295', 'license': 'MIT', 'n_tokens_mistral': 3980, 'n_tokens_neox': 3662, 'n_words': 1145, 'path': 'Navigation.ipynb', 'repo_name': 'joselaral/p1-navigation-solution-drl', 'size': '12522'}