<jupyter_start><jupyter_text>Hyperparameters Used- $\gamma$: Discount Factor- Model Architecture    - 2 Fully-connected layers    - 4 states feed into a 128 node hidden-layer        - dropout = 0.6 (hyperparameter)        - RELU activation function    - Hidden layer feeds into a 4 node output-layer        - softmax activation function- Optimizer:    - Adam Optimizer        - default params        - $\alpha$: learning-rate- $\beta$: Exponentially-weighted average coefficient-Source: https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py<jupyter_code>import gym
import numpy as np
from itertools import count

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical

import matplotlib.pyplot as plt
%matplotlib notebook<jupyter_output><empty_output><jupyter_text>Some system setup<jupyter_code>gamma = 0.95 # discount factor
seed  = 543
render = False
log_interval = 10

env = gym.make('CartPole-v1')
#env = gym.make('InvertedDoublePendulum-v2')
#env.seed(seed)
#torch.manual_seed(seed)<jupyter_output><empty_output><jupyter_text>Deep Learning ModelThe following class defines the neural network used to train our agent.We use a two layer fully-connected model with droupout on the first layer, followed by a RELU activation function.The second layer uses a softmax activation function to estimate the best action to take in the current state.<jupyter_code>class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.affine1 = nn.Linear(4, 124)
        #self.dropout = nn.Dropout(p=0.6)
        self.affine2 = nn.Linear(124, 2)

        self.saved_log_probs = []
        self.rewards = []

    def forward(self, x):
        x = self.affine1(x)
        #x = self.dropout(x)
        x = F.relu(x)
        action_scores = self.affine2(x)
        return F.softmax(action_scores, dim=1)<jupyter_output><empty_output><jupyter_text>Model and optimizer setupHere we set up the model and choose to optimize it with an Adam Optimizer with learning rate $\alpha=0.01$.<jupyter_code>policy = Policy()
optimizer = optim.Adam(policy.parameters(), lr=1.2e-2)
eps = np.finfo(np.float32).eps.item()<jupyter_output><empty_output><jupyter_text>Helper Functions<jupyter_code>def select_action(state):
    state = torch.from_numpy(state).float().unsqueeze(0)
    probs = policy(state)
    m = Categorical(probs)
    action = m.sample()
    policy.saved_log_probs.append(m.log_prob(action))
    return action.item()

def sim_action(state):
    state = torch.from_numpy(state).float().unsqueeze(0)
    actions = policy(state)
    _, action = actions.max(1)
    return action.item()

def finish_episode():
    R = 0
    policy_loss = []
    returns = []
    for r in policy.rewards[::-1]:
        R = r + gamma * R
        returns.insert(0, R)
    returns = torch.tensor(returns)
    returns = (returns - returns.mean()) / (returns.std() + eps)
    for log_prob, R in zip(policy.saved_log_probs, returns):
        policy_loss.append(-log_prob * R)
    optimizer.zero_grad()
    policy_loss = torch.cat(policy_loss).sum()
    policy_loss.backward()
    optimizer.step()
    del policy.rewards[:]
    del policy.saved_log_probs[:]

def sim():
    tot_reward = 0
    state = env.reset()
    for t in range(1, 10000):
        action = sim_action(state)
        state, reward, done, _ = env.step(action)
        tot_reward += reward
        env.render()
        if done:
            print(tot_reward)
            break
duration = []
#env.spec.max_episode_steps = 100
#env.spec.max_episode_steps<jupyter_output><empty_output><jupyter_text>TrainingWe train the model in the cell below until our average reward surpasses the built in reward threshold. We use an exponentially weighted average with $\beta = 0.4$ on the running reward.<jupyter_code>running_reward = 10
for i_episode in count(1):
    state, ep_reward = env.reset(), 0
    for t in range(1, 20000):  # Don't infinite loop while learning
        action = select_action(state)
        state, reward, done, _ = env.step(action)
        if render:
            env.render()
        policy.rewards.append(reward)
        ep_reward += reward
        if done:
            duration.append(t)
            break

    running_reward = 0.04 * ep_reward + (1 - 0.04) * running_reward
    finish_episode()
    if i_episode % log_interval == 0:
        print('Episode {}\tLast reward: {:.2f}\tAverage reward: {:.2f}'.format(
              i_episode, ep_reward, running_reward))
        #print(action)
    if running_reward > env.spec.reward_threshold:
        print("Solved! Running reward is now {} and "
              "the last episode runs to {} time steps!".format(running_reward, t))
        env.close()
        break<jupyter_output>Episode 10	Last reward: 69.00	Average reward: 19.78
Episode 20	Last reward: 50.00	Average reward: 30.86
Episode 30	Last reward: 74.00	Average reward: 41.98
Episode 40	Last reward: 61.00	Average reward: 50.60
Episode 50	Last reward: 92.00	Average reward: 68.95
Episode 60	Last reward: 430.00	Average reward: 106.02
Episode 70	Last reward: 51.00	Average reward: 114.58
Episode 80	Last reward: 75.00	Average reward: 115.34
Episode 90	Last reward: 112.00	Average reward: 113.21
Episode 100	Last reward: 500.00	Average reward: 140.29
Episode 110	Last reward: 500.00	Average reward: 241.04
Episode 120	Last reward: 500.00	Average reward: 327.83
Episode 130	Last reward: 500.00	Average reward: 385.54
Episode 140	Last reward: 500.00	Average reward: 423.90
Episode 150	Last reward: 500.00	Average reward: 449.41
Episode 160	Last reward: 21.00	Average reward: 367.53
Episode 170	Last reward: 15.00	Average reward: 248.53
Episode 180	Last reward: 15.00	Average reward: 170.01
Episode 190	Last reward: 18.00	Ave[...]<jupyter_text>Training performanceThe plot below is a measure of training performance. As the model is trained, it should on average be able to run longer without "failing".<jupyter_code>plt.plot(duration)
plt.xlabel("Episode")
plt.ylabel("Time Steps")
plt.title("Training Duration")
plt.savefig('training_duration_lr_12en2.png')
from gym import wrappers
from time import time # just to have timestamps in the files<jupyter_output><empty_output><jupyter_text>Simulate Environment and Record Current Model<jupyter_code>env = gym.make('CartPole-v1')
env = wrappers.Monitor(env, './videos/' + str(time()) + '/')
sim()
#sudo apt-get install ffmpeg<jupyter_output>500.0<jupyter_text>Save the model<jupyter_code>torch.save(policy.state_dict(), "./models/model_gamma_p95.pth")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10476190476190476, 'fraction_numerical': 0.04625850340136054, 'mean_word_length': 3.6558761435608726, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '436564', 'license': 'Linux-OpenIB', 'n_tokens_mistral': 2424, 'n_tokens_neox': 2131, 'n_words': 611, 'path': 'training_test.ipynb', 'repo_name': 'ashoats/DeepSeededGeneticLearning', 'size': '6615'}