<jupyter_start><jupyter_text>Box World Navigation Project---This notebook describes the implementation of the Deep Reinforcement Learning agent used to solve the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893).This notebook also serves as live and executable documentation. Therefore, there will be a little bit of code and modules imported.Nevertheless, all the *important* stuff is implemented in the [deeprl](deeprl/) package. So, you are encouraged to take a look at that package as well. A fully trained agent should perform as in the video below (if the video doesn't show automatically, execute the cell manually).<jupyter_code>%%HTML
<iframe width="560" height="315"
 src="https://www.youtube.com/embed/B4JKTivr4qA"
    frameborder="0"
    allow="autoplay; encrypted-media" allowfullscreen>
</iframe><jupyter_output><empty_output><jupyter_text>1. Loading necessary packagesWe begin by importing some necessary packages.  If the code cell below returns an error, please revisit the [README](README.md) instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md), [NumPy](http://www.numpy.org/), and [PyTorch](http://pytorch.org).If you are running a Windows system, please pay special attention to the README, as the code **will not** be able to automatically execute the environment.<jupyter_code>%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import numpy as np
import matplotlib.pyplot as plt

from deeprl.train import dqn
from deeprl.agent import Agent
from deeprl.model import QNetwork
from deeprl.util import print_source
from deeprl.util import load_environment<jupyter_output>INFO:root:Using first CUDA device for computation. Tune visibility by setting the CUDA_VISIBLE_DEVICES env var.<jupyter_text>2. The algorithmAs aforementioned, this code uses Deep Reinforcement Learning to solve the environment. In particular, we use the Deep Q-Networks (DQN) algorithm.DQN use *experience replay* and *fixed Q-targets* to be able to "stabilize" the learning process. Overall, the algorithm is: * Take action $a_t$ according to an $\epsilon$-greedy policy * Store transition $(s_t, a_t, r_{t+1}, s_{t+1})$ in replay memory $\mathcal{D}$ * Sample random mini-batch of transitions $(s, a, r, s')$ from $\mathcal{D}$ * Compute Q-learning targets with regards to old, fixed parameters $w^-$ * Optimize Mean Squared Error between Q-network and Q-learning targets$$\mathcal{L}_i(w_i) = \mathbb{E}_{s, a, r, s' \sim \mathcal{D}_i} \left[\left(r + \gamma\max_{a'}Q(s', a'; w_i^-)-Q(s, a; w_i)\right)^2\right]$$ * Using a variant of stochastic gradient descent 2.1 The optimizerThe variant of stochastic gradient descent we use is the [Adam optimization algorithm](https://arxiv.org/abs/1412.6980). 2.2 The network architectureFor this problem, we're using a four-layer fully-connected neural network. Layer sizes are: 1. `state_size` inputs, 16 outputs 2. 16 inputs, 32 outputs 3. 32 inputs, 64 outputs 4. 64 inputs, `action_size` outputsWhere `state_size` is the size of the state in the current environment (37 for the banana environment), and `action_size` is the size of the actions in the current environment (4 for the banana environment). The PyTorch implementation can be seen in the cell below.<jupyter_code>print_source(QNetwork.__init__)
print_source(QNetwork.forward)<jupyter_output><empty_output><jupyter_text>2.3 The actual learning algorithmWith the textual description given above and having the network and optimizer defined, one can devise a learning algorithm written in PyTorch. Assuming `self.qnetwork_target` implements the network that uses weights $w^-$ and `self.qnetwork_local` implements the network that uses weights $w$. The algorithm can be implemented as below (again, the reader is encouraged to read the [full source](deeprl/agent.py)).Notice that we *do not* fully transition the local weights $w$ to the $w^-$ weights. Rather, we perform a *soft* update, controlled by hyperparameter $\tau$. This allows us to slowly transition from set of weights to the other, giving more smooth operation to the algorithm.<jupyter_code>print_source(Agent.learn)<jupyter_output><empty_output><jupyter_text>2.4 HyperparametersAs can be seen above, the algorithm has many hyperparameters. The set of hyperparameters used are: * $\gamma = 0.99$ (Discount factor) * $\tau = 1\times10^{-3}$ (Network soft-update parameter) * $\alpha = 5\times10^{-5}$ (Learning rate used in the Adam optimizer) * Batch size = 64 * $|\mathcal{D}| = 100000$ (Size of the replay buffer) * Period between target updates = 5 (every 5 episodes we perform a soft update so that $w^- = \tau w + (1 - \tau)w^-$. * Since we're using an $\epsilon$-greedy policy, we decay the $\epsilon$ parameter with each episode. For this particular agent, we start with $\epsilon=1$ and decay it by $0.995$ until it reaches $0.001$. Updates to epsilon are, therefore $\epsilon \leftarrow \epsilon * \mathrm{decay}$.  3 Training and evaluation 3.1 TrainingWith all of the above defined, we can train our agent. Training is performed by the `dqn` function, shown below.What it does, essentially, is to load the environment, configure the agent with the environment parameters, and execute the learning process by decaying $\epsilon$ and checking whether the solution criterion is met. If it is, then it finishes training and persists the neural network model.The optimization criterion we're using is achieving an average reward greater than 13 over 100 episodes.<jupyter_code>print_source(dqn)<jupyter_output><empty_output><jupyter_text>3.2 Evaluation<jupyter_code>env = load_environment()
agent, scores, episodes = dqn(env, checkpointfn='checkpoint.pth')
env.close()
fig = plt.figure()
plt.plot(np.arange(len(scores)), scores, linewidth=1.0)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.grid()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0962433510638298, 'fraction_numerical': 0.01579122340425532, 'mean_word_length': 5.456008583690987, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '581674', 'license': 'MIT', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1636, 'n_words': 713, 'path': 'Report.ipynb', 'repo_name': 'renatolfc/deep-rl-navigation', 'size': '6016'}