<jupyter_start><jupyter_text>REINFORCE in TensorFlowThis notebook implements a basic reinforce algorithm a.k.a. policy gradient for CartPole env.It has been deliberately written to be as simple and human-readable. The notebook assumes that you have [openai gym](https://github.com/openai/gym) installed.In case you're running on a server, [use xvfb](https://github.com/openai/gymrendering-on-a-server)<jupyter_code>import sys, os
if 'google.colab' in sys.modules:
    %tensorflow_version 1.x
    
    if not os.path.exists('.setup_complete'):
        !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/master/setup_colab.sh -O- | bash

        !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/coursera/grading.py -O ../grading.py
        !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/coursera/week5_policy_based/submit.py

        !touch .setup_complete

# This code creates a virtual display to draw game images on.
# It will have no effect if your machine has a monitor.
if type(os.environ.get("DISPLAY")) is not str or len(os.environ.get("DISPLAY")) == 0:
    !bash ../xvfb start
    os.environ['DISPLAY'] = ':1'
import gym
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

env = gym.make("CartPole-v0")

#gym compatibility: unwrap TimeLimit
if hasattr(env,'env'):
    env=env.env

env.reset()
n_actions = env.action_space.n
state_dim = env.observation_space.shape

plt.imshow(env.render("rgb_array"))
env.close()<jupyter_output><empty_output><jupyter_text>Building the policy network For REINFORCE algorithm, we'll need a model that predicts action probabilities given states.For numerical stability, please __do not include the softmax layer into your network architecture__. We'll use softmax or log-softmax where appropriate.<jupyter_code>import tensorflow as tf
sess = tf.InteractiveSession()

#create input variables. We only need <s,a,R> for REINFORCE
states = tf.placeholder('float32',(None,)+state_dim,name="states")
actions = tf.placeholder('int32',name="action_ids")
cumulative_rewards = tf.placeholder('float32', name="cumulative_returns")
import keras
from keras.layers import Dense

model = keras.models.Sequential()
model.add(Dense(128, activation='relu', input_shape=state_dim))
model.add(Dense(64, activation='relu'))
model.add(Dense(n_actions, activation='linear'))

logits = model(states)

policy = tf.nn.softmax(logits)
log_policy = tf.nn.log_softmax(logits)
# Initialize model parameters
sess.run(tf.global_variables_initializer())
#utility function to pick action in one given state
get_action_proba = lambda s: policy.eval({states:[s]})[0] 
def predict_probs(states):
    """ 
    Predict action probabilities given states.
    :param states: numpy array of shape [batch, state_shape]
    :returns: numpy array of shape [batch, n_actions]
    """
    return policy.eval({ph_states: [states]})[0]<jupyter_output><empty_output><jupyter_text>Loss function and updatesWe now need to define objective and update over policy gradient.Our objective function is$$ J \approx  { 1 \over N } \sum  _{s_i,a_i} \pi_\theta (a_i | s_i) \cdot G(s_i,a_i) $$Following the REINFORCE algorithm, we can define our objective as follows: $$ \hat J \approx { 1 \over N } \sum  _{s_i,a_i} log \pi_\theta (a_i | s_i) \cdot G(s_i,a_i) $$When you compute gradient of that function over network weights $ \theta $, it will become exactly the policy gradient.<jupyter_code>#get probabilities for parti
indices = tf.stack([tf.range(tf.shape(log_policy)[0]),actions],axis=-1)
log_policy_for_actions = tf.gather_nd(log_policy,indices)
# policy objective as in the last formula. please use mean, not sum.
# note: you need to use log_policy_for_actions to get log probabilities for actions taken

J = tf.reduce_mean(log_policy_for_actions*cumulative_rewards)

#regularize with entropy
entropy = -tf.reduce_sum(policy*log_policy, axis=1)
#all network weights
all_weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)

#weight updates. maximizing J is same as minimizing -J. Adding negative entropy.
loss = -J -0.1 * entropy

update = tf.train.AdamOptimizer().minimize(loss,var_list=all_weights)<jupyter_output><empty_output><jupyter_text>Computing cumulative rewards<jupyter_code>def get_cumulative_rewards(rewards, #rewards at each step
                           gamma = 0.99 #discount for reward
                           ):
    """
    take a list of immediate rewards r(s,a) for the whole session 
    compute cumulative rewards R(s,a) (a.k.a. G(s,a) in Sutton '16)
    R_t = r_t + gamma*r_{t+1} + gamma^2*r_{t+2} + ...
    
    The simple way to compute cumulative rewards is to iterate from last to first time tick
    and compute R_t = r_t + gamma*R_{t+1} recurrently
    
    You must return an array/list of cumulative rewards with as many elements as in the initial rewards.
    """
    
    cumulative_rewards = np.array(rewards).astype(np.float32)
    for i in range(len(rewards)-2, -1, -1):
        cumulative_rewards[i] += gamma*cumulative_rewards[i+1]
  
    return cumulative_rewards
    
    
assert len(get_cumulative_rewards(range(100))) == 100
assert np.allclose(get_cumulative_rewards([0,0,1,0,0,1,0],gamma=0.9),[1.40049, 1.5561, 1.729, 0.81, 0.9, 1.0, 0.0])
assert np.allclose(get_cumulative_rewards([0,0,1,-2,3,-4,0],gamma=0.5), [0.0625, 0.125, 0.25, -1.5, 1.0, -4.0, 0.0])
assert np.allclose(get_cumulative_rewards([0,0,1,2,3,4,0],gamma=0), [0, 0, 1, 2, 3, 4, 0])
print("looks good!")
def train_step(_states,_actions,_rewards):
    """given full session, trains agent with policy gradient"""
    _cumulative_rewards = get_cumulative_rewards(_rewards)
    update.run({states:_states,actions:_actions,cumulative_rewards:_cumulative_rewards})<jupyter_output><empty_output><jupyter_text>Playing the game<jupyter_code>def generate_session(env, t_max=1000):
    """play env with REINFORCE agent and train at the session end"""
    
    #arrays to record session
    states,actions,rewards = [],[],[]
    
    s = env.reset()
    
    for t in range(t_max):
        
        #action probabilities array aka pi(a|s)
        action_probas = get_action_proba(s)
        
        a = np.random.choice(n_actions, p=action_probas)
        
        new_s,r,done,info = env.step(a)
        
        #record session history to train later
        states.append(s)
        actions.append(a)
        rewards.append(r)
        
        s = new_s
        if done: break
            
    train_step(states,actions,rewards)
            
    return states, actions, rewards, sum(rewards)
        
s = env.reset()
s = tf.InteractiveSession()
s.run(tf.global_variables_initializer())

for i in range(100):
    
    rewards = [generate_session(env)[3] for _ in range(100)] #generate new sessions
    
    print ("mean reward:%.3f"%(np.mean(rewards)))

    if np.mean(rewards) > 300:
        print ("You Win!")
        break<jupyter_output>/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py:1750: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).
  warnings.warn('An interactive session is already active. This can '<jupyter_text>Results & video<jupyter_code>#record sessions
import gym.wrappers
env = gym.wrappers.Monitor(gym.make("CartPole-v0"),directory="videos",force=True)
sessions = [generate_session(env)[3] for _ in range(100)]
env.env.close()

#show video
from IPython.display import HTML
import os

video_names = list(filter(lambda s:s.endswith(".mp4"),os.listdir("./videos/")))

HTML("""
<video width="640" height="480" controls>
  <source src="{}" type="video/mp4">
</video>
""".format("./videos/"+video_names[-1])) #this may or may not be _last_ video. Try other indices
from submit import submit_cartpole
submit_cartpole(generate_session, '<EMAIL>', 'n7Z9M8kSMXspG44S')
# That's all, thank you for your attention!
# Not having enough? There's an actor-critic waiting for you in the honor section.
# But make sure you've seen the videos first.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1194468241341329, 'fraction_numerical': 0.020438134867213315, 'mean_word_length': 4.309941520467836, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '27038', 'license': 'MIT', 'n_tokens_mistral': 2837, 'n_tokens_neox': 2695, 'n_words': 720, 'path': 'Practical Reinforcement Learning/Week5_policy_based/z-practice_reinforce.ipynb', 'repo_name': 'Zuyuf/Advanced-Machine-Learning-Specialization', 'size': '8189'}