<jupyter_start><jupyter_text>Reinforcement Learning ExampleThis notebook will guide you through implementing a gradient policy algorithm in Neon to train an agent to play Pong. You will learn- Using the Autodiff interface to perform automatic differentiation and obtain gradients given an op-tree- Implement a neural network that adjusts its parameters and policies with gradients to get better rewards over time directly from experiences. This model samples the expected action (moving a paddle up/down) from a distribution when no gradients are available and high uncertainty is in place.    PreambleReinforcement Learning trains an agent to take actions in an unknown environment based on experiences. Unlike supervised learning, actual labels are not always available, so RL algorithms explores the environment by interacting with actions and receives corresponding feedbacks as positive or negative rewards. In this tutorial, we will train an agent to play Pong with Stochastic Policy Gradients using [OpenAI Gym](https://gym.openai.com/) as the environment and [neon](https://neon.nervanasys.com/index.html/) as our deep learning framework. ![Agent interacting with its environment](data/pong_load_trained_model.gif)As described in [<NAME>'s blog](http://karpathy.github.io/2016/05/31/rl/) Policy Gradients has shown to perform better than DQN algorithms by most people including DQN Authors (https://www.youtube.com/watch?v=M8RfOCYIL8k), so we will use it to train the agent. The underlying model is a 2-layer fully connected neural network that receives image frames as inputs, explores rewards given stochastic actions (probability of moving a paddle up/down) in an unknown environment, and updates weights by weigthing rewards with a cumulative discounted rewards. The following picture depicts the main components of this learning system.  ![RL algorithm](data/pong_architecture.jpg) SetupThis example works with Python 2.7. Your environment needs to have the following packages installed:- neon v1.9.0- OpenAI Gym for initializing Atari game environments- numpy for random initialization and aritmetic operations  Model ArchitectureWe will guide you through implementing a policy function parameterized by a neural network. We first import all the needed ingredients:<jupyter_code>import numpy as np
import gym

from neon.backends import gen_backend
from neon.backends import Autodiff
import random
import os<jupyter_output><empty_output><jupyter_text>We also set up the backend and define the class containing our network. This consists of two layers 'W1' and 'W2' with values randomly initialized, but trainable with gradient updates. We have included functions implementing the forward and backward propagation steps. The forward step generates a policy given an a visual representation of the environment stored in variable x. The back propagation function updates the layers of the network modulating the loss function values with discounted rewards.<jupyter_code>be = gen_backend('cpu', batch_size=128)
class Network:
    def __init__(self, D = 80*80, H = 200, gamma = 0.99, restore_model=False):
        '''
        D: number image pixels
        H: number of hidden units in first layer of neural network
        gamma: discount factor
        '''
        self.gamma = gamma
        self.ll = {}
        self.learning_rate = 0.00001

        if restore_model and os.path.exists('model_weights.npy'):
            self.ll['W1'] = np.load('model_weights.npy').item()['W1']            
            self.ll['W2'] = np.load('model_weights.npy').item()['W2']
        else:
            self.ll['W1'] = be.array(np.random.randn(H,D) / np.sqrt(D)) #be.zeros((H,D))
            self.ll['W2'] = be.array(np.random.randn(H,1) / np.sqrt(H)) #be.zeros((H,1))        
        self.dW1 = be.array(np.zeros((H, D)))
        self.dW2 = be.array(np.zeros((H, 1)))
        

    # forward propagation
    def policy_forward(self, x):
        # map visual input to the first hidden layer of a neural network
        # a larger number of units will increase the capacity of the network to learn different game states
        # different local minima in this context represents different strategies giving the same game output
        h = be.dot(self.ll['W1'],  be.array(x))
        h = be.sig(h)
        dlogp = be.dot(h.transpose(), self.ll['W2'])

        #probability op of moving paddle up and hidden state
        p = be.sig(dlogp)

        p_val = be.empty((1, 1))
        h_val = be.empty((200, 1))
        p_val[:] = p
        h_val[:] = h
        return p_val.get(), h_val.get(), p, h

    # backward propagation
    def policy_backward(self, losses_op, episode_dlogps, episode_rewards):
        discounted_rewards = self.discount_rewards(episode_rewards)
        # to reduce the variance of the gradient estimator and avoid potential vanishing problems
        # when computing gradients: http://www.scholarpedia.org/article/Policy_gradient_methods
        discounted_rewards -= np.mean(discounted_rewards)
        discounted_rewards /= np.std(discounted_rewards)

        episode_dlogps *= discounted_rewards  # to modulate gradients with discount factors
        
        # compute gradients using neon backend
        # Note: gradients are vectors that point in the direction of the greatest
        # rate of increase of the loss function. We use them to negatively update
        # weight values since we are interested in decreasing loss (error) values
        for i in range(len(losses_op)):
            ad = Autodiff(op_tree=losses_op[i]*be.array(episode_dlogps[i]), be=be, next_error=None)
            # compute gradients and assign them to self.dW2 and self.dW1        
            ad.back_prop_grad([self.ll['W2'], self.ll['W1']], [self.dW2, self.dW1])                        
            # weights update:
            self.ll['W2'][:] = self.ll['W2'].get() - self.learning_rate * self.dW2.get()/len(losses_op)
            self.ll['W1'][:] = self.ll['W1'].get() - self.learning_rate * self.dW1.get()/len(losses_op)
        return

    def sigmoid(self, x): 
        return 1.0 / (1.0 + np.exp(-x)) # sigmoid "squashing" function to interval [0,1]

    def get_loss(self, y_fake, up_probability):
        loss = y_fake-up_probability
        return loss
    
    # which game_actions lead to winning a game?: given this episodic environment, we assign rewards per time-step assuming
    # there is a stationary distribution for a policy within the episode.
    #   reward < 0, if agent missed the ball and hence lose the game
    #   reward > 0, if agent won the game
    #   reward == zero, game in progress
    # the agent receives rewards generated by the game and implements discounted reward backwards with exponential
    # moving average. More weight is given to earlier rewards. Reset to zero when game ends.
    def discount_rewards(self, r):
      discounted_r = np.zeros_like(r)
      running_add = 0
      for t in reversed(range(0, r.size)):
        # if reward at index t is nonzero, then there is a positive/negative reward. This also marks a game boundary
        # for the sequence of game_actions produced by the agent
        if r[t] != 0.0: running_add = 0.0 
        # moving average given discount factor gamma, it assigns more weight to recent game actions
        running_add = running_add * self.gamma + r[t]
        discounted_r[t] = running_add
      return discounted_r

    # takes a single game frame as input and preprocesses before feeding into model
    def prepro(self, I):
      """ prepro 210x160x3 uint8 frame into 6400 (80x80) 1D float vector """
      I = I[35:195] # crop
      I = I[::2,::2,0]              # downsample by factor of 2
      I[I == 144] = 0               # erase background (background type 1)
      I[I == 109] = 0               # erase background (background type 2)
      I[I != 0] = 1                 # everything else (paddles, ball) just set to 1
      return I.astype(np.float).ravel() #flattens

    # stochastic process to choose an action (moving up) proportional to its
    # predicted probability. If the probability of going up is higher than tossing 
    # a coin, then it's more likely to sample an up action from this stochastic process.
    # Probablity of choosing the opposite action is (1-probability_up)
    #    action_ == 2, moving up
    #    action_ == 3, moving down
    def sample_action(self, up_probability):
        stochastic_value = np.random.uniform()
        action = 2 if stochastic_value < up_probability else 3
        return action<jupyter_output><empty_output><jupyter_text>Part of the magic behind reinforcement learning is about the possibility of optimizing objective functions under uncertainty or non-differentiable scenarios considering a stochastic process. In our case, we sample an action from the probability returned by the last layer of a neural network and consider that as a temporary (fake) label. This logic is implemented in the function __sample_action__().<jupyter_code># stochastic process to choose an action (moving up) proportional to its
# predicted probability. If such value is high, it's more
# likely to sample an up action from this stochastic process.
# Probablity of choosing the opposite action is (1-probability_up)
#    action_ == 2, moving up
#    action_ == 3, moving down
def sample_action(up_probability):
    stochastic_value = np.random.uniform()
    action = 2 if stochastic_value < up_probability else 3
    return action<jupyter_output><empty_output><jupyter_text>Initialization of variables.<jupyter_code>render = False               # to visualize agent 
restore_model = True        # to load a trained model when available

random.seed(2017)

D = 80 * 80                 # number of pixels in input
H = 200                     # number of hidden layer neurons
# Game environment
env = gym.make("Pong-v0")
network = Network(D=D, H=H, restore_model=restore_model)

# Each time step, the agent chooses an action, and the environment returns an observation and a reward.
# The process gets started by calling reset, which returns an initial observation
observation = env.reset()
prev_x = None

# hidden state, gradient ops, gradient values, rewards
hs, losses_op, dlogps, rewards = [],[],[], []
running_reward = None       # current reward
reward_sum = 0.0            # sum rewards
episode_number = 0

game_actions = []
game_rewards = []
game_gradients = []<jupyter_output><empty_output><jupyter_text>Let's train!Our goal is to train an agent to win Pong against its opponent. An action consists of moving a paddle UP/DOWN, this eventually generates a positive reward (+1) if the trained agent wins a game or negative one (-1) if agent misses the ball.Before knowing the result of a game, the model gets a fake label via the stochastic process explained before. This is like tossing a coin to decide to accept the log probabilities of a neural network. An optimal set of actions will maximize the sum of rewards along the game. An import event is when the agent wins/losses a game. But what caused this outcome?. The algorithm decided to modulate the loss functions of the network with the positive or negative rewards obtained from the environment and assign more weight to earlier actions using a moving average scheme. This logic is implement in function policy_backward() of the Network class.<jupyter_code>while True:
    cur_x = network.prepro(observation)
    x = cur_x - prev_x if prev_x is not None else np.zeros(D)
    prev_x = cur_x

    up_probability, h_value, p, h = network.policy_forward(x)
    action = network.sample_action(up_probability)                              

    # assign a fake label, this decreases uncertainty and
    # this is one of the beauties of Reinforcement Learning
    y_fake = 1 if action == 2 else 0     
    
    # loss function gets closer to assigned label, the smaller difference
    # between probabilities the better
    # store gradients: derivative(log(p(x|theta)))       
    dlogp = np.abs(y_fake - up_probability)    
    # loss value
    dlogps.append(dlogp) 
    # loss op
    losses_op.append(be.absolute(y_fake - p))
    
    if render:
        env.render()
    
    #action: 
    #    0: no movement
    #    1: no movement
    #    2: up
    #    3: down
    #    4: up
    #    5: down
    observation, reward, done, info = env.step(action)
    
    # modifying rewards to favor longer games and thus to increase number of
    # positive rewards.
    reward = 0.0 if reward == 0.0 else reward
    reward = 1.0*len(game_rewards) if reward!=0.0 and len(game_rewards)>80 else reward
    reward = -1.0*len(game_rewards) if reward!=0.0 and len(game_rewards)<=50 else reward

    rewards.append(reward)
    reward_sum += reward
    
    game_actions.append(action)
    game_rewards.append(reward)
    game_gradients.append(dlogp[0][0])

    # end of a game
    # Pong has either +1 or -1 as reward when game ends.
    if reward != 0:  
        message = "Episode %d: game finished." % (episode_number)
        if reward < 0:
            message += "\x1b[0;31;40m  (RL loses)\x1b[0m"
        elif reward > 0:
            message += "\x1b[0;32;40m  (RL wins)\x1b[0m"
        print(message)
        print('Game duration: %d steps | Sum rewards: %f | Sum errors: %f' %(len(game_actions), np.sum(game_rewards), np.sum(game_gradients)))
        print('------------------------------------')
        game_actions = []
        game_rewards = []
        game_gradients = []
        
    # to save model
    if (episode_number+1)%10==0:
        np.save('model_weights.npy', network.ll)
        
    # end of an episode (minibatch of games)
    if done:
        episode_number +=1
        dlogps = np.vstack(dlogps)
        rewards = np.vstack(rewards)
        
        network.policy_backward(losses_op, dlogps, rewards)
        mean_loss = np.sum([x * x for x in dlogps])
        running_reward = reward_sum if running_reward is None else running_reward * 0.99 + reward_sum * 0.01
        print('-----------------------------------------------')
        print('Episode %d has finished, time to backpropagate.' % (episode_number - 1))
        print('Total reward was %f Running_reward: %f Mean_loss: %f' % (reward_sum, running_reward, mean_loss))
        print('-----------------------------------------------')

        # reset game environment
        observation = env.reset()  
        reward_sum = 0
        prev_x = None        
        dlogps, rewards = [], []
        losses_op = []<jupyter_output>Episode 0: game finished.[0;32;40m  (RL wins)[0m
Game duration: 89 steps | Sum rewards: 88.000000 | Sum errors: 42.775948
------------------------------------
Episode 0: game finished.[0;31;40m  (RL loses)[0m
Game duration: 47 steps | Sum rewards: -46.000000 | Sum errors: 21.791306
------------------------------------
Episode 0: game finished.[0;31;40m  (RL loses)[0m
Game duration: 48 steps | Sum rewards: -47.000000 | Sum errors: 22.625721
------------------------------------
Episode 0: game finished.[0;31;40m  (RL loses)[0m
Game duration: 47 steps | Sum rewards: -46.000000 | Sum errors: 22.683254
------------------------------------
Episode 0: game finished.[0;31;40m  (RL loses)[0m
Game duration: 45 steps | Sum rewards: -44.000000 | Sum errors: 21.699604
------------------------------------
Episode 0: game finished.[0;31;40m  (RL loses)[0m
Game duration: 47 steps | Sum rewards: -46.000000 | Sum errors: 22.708187
------------------------------------
Episode 0: game finished[...]
{'fraction_non_alphanumeric': 0.10443527591542032, 'fraction_numerical': 0.02320783909231563, 'mean_word_length': 2.858955223880597, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '60899', 'license': 'MIT', 'n_tokens_mistral': 4608, 'n_tokens_neox': 4266, 'n_words': 1685, 'path': "An-Introduction-to-Practical-Deep-Learning/Week-5-Hot-Research-and-Intel's-Roadmap/Exercise+8.ipynb", 'repo_name': 'fengjings/Coursera', 'size': '15521'}