<jupyter_start><jupyter_text>Policy Gradient on cartpoleIn this notebook, we run the policy gradient algorithm on the cartpole example. * [You can read about the cartpole problem here.](cartpole.ipynb)* [You can read about policy gradient here.](pg_notebook.ipynb)* [You can see the pure code for policy gradient on cartpole here.](./cartpole/pg_on_cartpole.py)  Summary of the algorithmWe build a (deep) network to represent the probability density function $\pi_\theta$= `network(state).````network = keras.Sequential([            keras.layers.Dense(30, input_dim=n_s, activation='relu'),            keras.layers.Dense(30, activation='relu'),            keras.layers.Dense(n_a, activation='softmax')])network.compile(loss='categorical_crossentropy')``` Then, we iteratively improve the network. In each iteration of the algorithm, we do the following* i. We rollout the environment to collect data for PG by following these steps:    * i.a. We initialize empty histories for `states=[]`, `actions=[]`, `rewards=[]`    * i.b. We observe the `state` $s$ and sample `action` $a$ from the poliy pdf $\pi_{\theta}(s)$        `softmax_out = network(state)`        `a = np.random.choice(n_a, p=softmax_out.numpy()[0])`        * i.c. We derive the environment using $a$ and observe the `reward` $r$.    * i.d. We add $s,\:a,\:r$ to the history batch `states`, `actions`, `rewards`.    * i.e. We continue from i.b. until the episode ends.* ii. We improve the policy by following these steps    * ii.a. We calculate the reward to go and standardize it.     * ii.b. We optimize the policy.        `    target_actions = tf.keras.utils.to_categorical(np.array(actions), n_a)    loss = self.network.train_on_batch(states, target_actions, sample_weight=rewards_to_go)    `      Running on google colabIf you want to run on google colab, go ahead and run the following cell. If you want to run on your computer, skip this cell and start from Importing libraries.<jupyter_code>!git clone https://github.com/FarnazAdib/Crash_course_on_RL.git
%cd Crash_course_on_RL
!pip install .<jupyter_output><empty_output><jupyter_text>Importing libraries We start coding by importing the required libraries. If you get an error, you have possibly forgotten to change the kernel. See [Prepare a virtual environment](Preparation.ipynb).<jupyter_code>import numpy as np
import tensorflow as tf
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
from tensorflow import keras
import datetime as dt
import warnings
warnings.filterwarnings('ignore')
from cartpole.dynamics import CartPole
from cartpole.pltlib import PLTLIB<jupyter_output><empty_output><jupyter_text>Saving directoriesNext, we set up some paths to write data and possibly capture some videos for future investigation.<jupyter_code>STORE_PATH = '/tmp/cartpole_exp1/PG'
data_path = STORE_PATH + f"/data_{dt.datetime.now().strftime('%d%m%Y%H%M')}"
agent_path = STORE_PATH + f"/agent_{dt.datetime.now().strftime('%d%m%Y%H%M')}"
train_writer = tf.summary.create_file_writer(data_path)<jupyter_output><empty_output><jupyter_text>Making the environmentWe select the random seed and make the cartpole environment.<jupyter_code>Rand_Seed = 1
env_par = {
    'Rand_Seed': Rand_Seed,
    'STORE_PATH': STORE_PATH,
    'monitor': False,
    'threshold': 195.0
}
Rand_Seed = 1
CP = CartPole(env_par)<jupyter_output><empty_output><jupyter_text>Making the policy gradient agentWe define the policy gradient class. This class builds a network representing the pdf $\pi_{\theta}$ of the parametric policy. The parameter vector $\theta$ is the weights and biases in the network. The network has a structure which you can see in PG class. `hidden_size` denotes the number of nodes in the layers of the network. `GAMMA`is the discount factor $0\leq \gamma \leq 1$ quantifies how much we care about the immediate rewards and future rewards. We update the parameter vector $\theta$ by an ADAM optimizer which is a gradient algorithm. `learning_rate_adam`is the learning rate in ADAM optimizer is usually in $[0.0001,\:0.1]$. `adam_eps` is another hyperparameter to tune in ADAM algorithm.<jupyter_code>class PG:
    def __init__(self, hparams):
        self.hparams = hparams
        np.random.seed(hparams['Rand_Seed'])
        tf.random.set_seed(hparams['Rand_Seed'])

        # The policy network
        self.network = keras.Sequential([
            keras.layers.Dense(self.hparams['hidden_size'], input_dim=self.hparams['num_state'], activation='relu',
                               kernel_initializer=keras.initializers.he_normal(), dtype='float64'),
            keras.layers.Dense(self.hparams['hidden_size'], activation='relu',
                               kernel_initializer=keras.initializers.he_normal(), dtype='float64'),
            keras.layers.Dense(self.hparams['num_actions'], activation='softmax', dtype='float64')
        ])
        self.network.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.Adam(
            epsilon=self.hparams['adam_eps'], learning_rate=self.hparams['learning_rate_adam']))

    def get_action(self, state, env):
        
        # Building the pdf for the given state
        softmax_out = self.network(state.reshape((1, -1)))
        
        # Sampling an action according to the pdf
        selected_action = np.random.choice(self.hparams['num_actions'], p=softmax_out.numpy()[0])
        return selected_action

    def update_network(self, states, actions, rewards):
        reward_sum = 0
        rewards_to_go = []
        for reward in rewards[::-1]:  # reverse buffer r
            reward_sum = reward + self.hparams['GAMMA'] * reward_sum
            rewards_to_go.append(reward_sum)
        rewards_to_go.reverse()
        rewards_to_go = np.array(rewards_to_go)
        # standardise the rewards
        rewards_to_go -= np.mean(rewards_to_go)
        rewards_to_go /= np.std(rewards_to_go)
        states = np.vstack(states)
        target_actions = tf.keras.utils.to_categorical(np.array(actions), self.hparams['num_actions'])
        loss = self.network.train_on_batch(states, target_actions, sample_weight=rewards_to_go)
        return loss<jupyter_output><empty_output><jupyter_text>Now that we have defined the PG class, it is enough to build an object and to iterate. You can change the following hyper parameters if you like. For example, you can set `GAMMA`: 0.9 to see if you can solve the problem faster.* `hidden_size`: Number of nodes in the layers of the network.* `GAMMA`: The discount factor $0\leq \gamma \leq 1$ quantifies how much we care about the immediate rewards and future rewards. We have two extreme cases 1. $\gamma \rightarrow 0$: We only care about the current reward not what we'll receive in future. 2. $\gamma \rightarrow 1$: We care about rewards at all times equally. We usually select $\gamma$ near to $1$.* `num_episodes`: The maximum number of iterations to run the PG algorithm. You can increase it if the problem is not solved within the specified number of iterations* `learning_rate_adam`: We have chosen ADAM optimizer for the gradient optimization. The learning rate is usually in $[0.0001,\:0.1]$ and it is usally selected by try and error. A good point to start is the defalut value in the ADAM optimizer which is 0.01. * `adam_eps`: This is another hyperparameter to tune in ADAM algorithm. Leave it as it is. You can read about it [here](https://arxiv.org/abs/1412.6980). The epsilon in adam optimization.<jupyter_code>agent_par = {
    'num_state': CP.env.observation_space.shape[0],
    'num_actions': CP.env.action_space.n,
    'Rand_Seed': Rand_Seed,
    'hidden_size': 30,
    'GAMMA': 1.0,
    'num_episodes': 2000,
    'learning_rate_adam': 0.001,
    'adam_eps': 1e-7,
}
policy = PG(agent_par)<jupyter_output><empty_output><jupyter_text>Start learningNow, we start the learning loop. The learning loop iterates for a maximum of number `num_episodes`. In each iteration* The agent derives the environment for one episode to collect data for PG.* We update the agent by policy gradient algorithm using the recorded data.* We check if the problem is solved.* We write the data.At the end of the learning loop, we close the environment.<jupyter_code>tot_rews = []
mean_100ep = 0
for episode in range(agent_par['num_episodes']):

    # Do one rollout
    states, actions, rewards, _, _ = CP.one_rollout(policy)

    # Update the network
    loss = policy.update_network(states, actions, rewards)

    # Check if the problem is solved
    if episode > 100:
        mean_100ep = np.mean(tot_rews[-101:-1])

    tot_reward = sum(rewards)
    tot_rews.append(tot_reward)
    print(f"Episode: {episode}, Reward: {tot_reward}, Mean of 100 cons episodes: {mean_100ep}")
    if mean_100ep > env_par['threshold']:
        print(f"Problem is solved.")
        policy.network.save(agent_path)
        break

    # Save data
    with train_writer.as_default():
        tf.summary.scalar('reward', tot_reward, step=episode)

# Close the environment
CP.env.close()<jupyter_output>Episode: 0, Reward: 11.0, Mean of 100 cons episodes: 0
Episode: 1, Reward: 28.0, Mean of 100 cons episodes: 0
Episode: 2, Reward: 26.0, Mean of 100 cons episodes: 0
Episode: 3, Reward: 19.0, Mean of 100 cons episodes: 0
Episode: 4, Reward: 20.0, Mean of 100 cons episodes: 0
Episode: 5, Reward: 14.0, Mean of 100 cons episodes: 0
Episode: 6, Reward: 12.0, Mean of 100 cons episodes: 0
Episode: 7, Reward: 12.0, Mean of 100 cons episodes: 0
Episode: 8, Reward: 10.0, Mean of 100 cons episodes: 0
Episode: 9, Reward: 37.0, Mean of 100 cons episodes: 0
Episode: 10, Reward: 14.0, Mean of 100 cons episodes: 0
Episode: 11, Reward: 42.0, Mean of 100 cons episodes: 0
Episode: 12, Reward: 15.0, Mean of 100 cons episodes: 0
Episode: 13, Reward: 10.0, Mean of 100 cons episodes: 0
Episode: 14, Reward: 9.0, Mean of 100 cons episodes: 0
Episode: 15, Reward: 11.0, Mean of 100 cons episodes: 0
Episode: 16, Reward: 28.0, Mean of 100 cons episodes: 0
Episode: 17, Reward: 14.0, Mean of 100 cons episodes: 0
Epi[...]<jupyter_text>ResultsIt will get around 1-2 minutes to run the above cell. You will probably get some WARNING\ERROR. Some of these are related to incompatibility between some libraries. Don't panic. The problem will be solved after possibly after 800-1200 episodes.You can see a summary of results by running the cell below. you can also see the evaluation of the rewards vs. iteration as the iteration goes on<jupyter_code># Print the results if the problem is solved
if mean_100ep > env_par['threshold']:
    # Print the summary of the solution
    print(f"\n\nProblem is solved after {episode} Episode with the mean reward {mean_100ep} over the last 100 episodes")

    # Plot the result
    MyPlot = PLTLIB()
    MyPlot.reward_it(tot_rews)<jupyter_output>Problem is solved after 831 Episode with the mean reward 195.96 over the last 100 episodes
{'fraction_non_alphanumeric': 0.09846182186607719, 'fraction_numerical': 0.02560560007368518, 'mean_word_length': 4.033843300880853, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 34, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '82597', 'license': 'MIT', 'n_tokens_mistral': 3571, 'n_tokens_neox': 3281, 'n_words': 1201, 'path': 'pg_on_cartpole_notebook.ipynb', 'repo_name': 'FarnazAdib/Crash_course_on_RL', 'size': '10857'}