<jupyter_start><jupyter_text>Assignment 4 - Reinforcement Learning <NAME>   INDEXI. OverviewII. Problems&emsp;&emsp;II.I. 2D Marble Control&emsp;&emsp;&emsp;&emsp;II.I.I. STEPS for 2D Marble&emsp;&emsp;&emsp;&emsp;II.I.II. Explanation of the environment&emsp;&emsp;&emsp;&emsp;II.I.III. Explanation of the RLAgent1&emsp;&emsp;&emsp;&emsp;II.I.IV. Result Discussion&emsp;&emsp;II.II. Mountain Car&emsp;&emsp;&emsp;&emsp;II.II.I. STEPS for Mountain Car&emsp;&emsp;&emsp;&emsp;II.II.II. Description of Classic Mountain Car Problem&emsp;&emsp;&emsp;&emsp;II.II.III. Explanation of the environment&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;II.II.III.I. States&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;II.II.III.II. Actions&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;II.II.III.III. Reward&emsp;&emsp;&emsp;&emsp;II.II.IV. Explanation of the RLAgent2&emsp;&emsp;&emsp;&emsp;II.II.V. Result DiscussionIII. Methods&emsp;&emsp;III.I. Control Problems and Q function&emsp;&emsp;III.II. Observations and Result AnalysisIV. ResultsV. ConclusionsVI. Extra CreditVII. References  I. OverviewIn this assignment I implemented `Reinforcement Learning` agent for two classicial controll problem (`2D-Marble` and `Mountain Car`). For the `2D-Marble` problem, I implemented the environment as well as the the RL-Agent. For the `Mountain Car` problem, I used the `gym` environment and implemented the RL-Agent. For both of the problem, I used the deep neural network as the function approximator.  II. Problems  II.I. 2D Marble Control![image.png](attachment:image.png) II.I.I. STEPS for 2D MarbleAgenda:1. [II Problems] First, build 2D marble class by extending the Marble class from the class note. (&check;)2. [II Problems] Explain the environment. (&check;)3. [III Methods] Build your own RLAgent class to solve the problem. (&check;)4. [III Methods] Explain your RLAgent (with a neural network function approximator) (&check;)5. [IV Results]  Discuss the results (&check;) II.I.II. Explanation of the environmentFor this problem, I build my own 2D marble environment class (i.e. `Marble`). I considered 4 states (x and y position of the marble and the velocity in x and y direction). The marble can do 5 actions (go left, right, up, down and/or do-nothing). The `next` method takes and action and move the marble to the next state. I considered the cases where, marble goes outside of the board and I fixed the marble position on the boundary. The `get_reward` method return 1 if the marble goes within a certain region (i.e., +/- goal_width, goal_hight) of the goal defined by `Goal_x` and `Goal_y` respectively. There are some other auxiliary methods that are quite straight forward to understand by the method name. II.I.III. Explanation of the RLAgent1To solve the 2D-marble control problem, I implemented a RLAgent class (i.e., `MarbleRLAgent`). In the `train()` function I implemented the on-policy control td-learning algorithm (i.e., SARSA). I followed the reading note on `Reinforcement Learning with Neural Network Function Approximation` to implement the `MarbleRLAgent` class. Here, I used the deep neural network as the function approximator. I passed the environment as the parameter while building the `MarbleRLAgent` class. In each step, I used the `epsilon greedy` algorithm to select the next action for the `MarbleRLAgent`. In the training, I started with the fixed starting state for each epoch.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import IPython.display as ipd  # for display and clear_output
import collections
from copy import deepcopy as copy

from util import Standardizer
class Marble():
    """ 2d marble problem
        
        states: x, y, dx, dy
        action: action [[0, -1], [0, 1], [0, 0], [1, 0], [-1, 0]]


        |            ___                     |
        |___________|///|____G_______________|
                    <- ->
    """

    def __init__(self, goal_x=5, goal_y=5, **params):
        """ Initializes the marble game board
        
            goal_x (float): x position of goal where marble must within self.goal_width 
                of this goal to complete the task.
            
            goal_y (float): y position of goal where marble must within self.goal_hight 
                of this goal to complete the task.

            bound (list): Contains bounds for marble game.
                [
                    x lower boundary, x upper boundary,
                    y lower boundary, y upper boundary, 
                    min x negative velocity, max x positive velocity,
                    min y negative velocity, max y positive velocity
                ]
        """
        # Number of state features
        self.n_state = 4
        
        # Number of actions. This seems to never be used
        self.n_action = 5
        
        # Location of goal given by its x position
        self.Goal_x = goal_x
        self.Goal_y = goal_y
        
        # Sets boundaries for positions [x, y] and velocity [dx, dy] 
        self.bound = params.pop('bound', [0, 10, 0, 10, -5, 5, -5, 5])
        
        # Set defaults for velocity if the input does not contains it.
        # This actually takes first 4 index value from bound (boundaries for positions)
        # and set the rest 4 index value that will represent boundary for velocity.
        # todo: Commenting this section right now. Changed the bound layout to build the _st_range in easier way.
        #       As this seems unnecessary, may be comeback later to think about this.
        # if len(self.bound) != 8:
        #     self.bound = self.bound[:4] + [-5, -5, 5, 5]

        # Creates an array where the 1st row contains min values and the 2nd row
        # contains max vaules. Not sure if min and max action matter, this
        # seems useful only for continuous actions only???
        # [[ x lower boundary, y lower boundary, min x negative velocity, min y negative velocity, min action x, min action y]
        #  [ x upper boundary, y upper boundary, max x positive velocity, max y positive velocity, max action x, max action y]]
        self._st_range = np.array([self.bound[:2], self.bound[2:4], self.bound[4:6], self.bound[6:], [-1, 1], [-1, 1]]).T
        
        # print("self._st_range: {}".format(self._st_range))
        
        # Never used in the rest of the code; don't care!
        self.nnNI = self.n_state + 1
        
        # Used check if marble is within a certain distance of the goal
        self.goal_width = 1
        self.goal_hight = 1
        
        # Initialize start state in case self.init isn't called
        # s[0] = x position
        # s[1] = y position
        # s[2] = dx or velocity in the x direction
        # s[3] = dy or velocity in the y direction
        self._s = [0, 0, 0, 0]

    def init(self, start=None):
        """Initializes starting state for board      
           
           If you want to set a specific starting state or
           a random one call this funtion!
           
           Args:
               start (list): 
                   start[0] should contain the starting x position of the marble
                   start[1] should contain the starting y position of the marble
                   start[2] should contain the starting dx velocity
                   start[3] should contain the starting dy velocity
        """
        if start is not None:
            # Set start state to passed x, y, dx, and dy
            self._s = start
        else: 
            # Randomly select start position between x-bounds and y-bounds
            self._s = [np.random.randint(self.bound[0], self.bound[1]), np.random.randint(self.bound[2], self.bound[3]), 0., 0.]
        return self._s
       
    def get_random_action(self):
        """ Randomly select a discrete action INDEX
            WARNING: Hardcoded for 5 actions; this is the wrong way of doing this.
            Todo: Never used in the process; consider remove this finally.
        """
        return [float(np.random.randint(3) -1), float(np.random.randint(3) -1)] # discrete action

    def get_bound_act(self, a):
        """Todo: Never used! Don't bother to change this; consider remove this finally."""
        if a[0] > 1:
            return 1
        elif a[0] < -1:
            return -1
        else:
            return a[0]

    def next(self, a):
        """ Given an action moves state s to next state s1
        
            Args:
                a (numeric array): Action taken whose value can be
                    (0, -1) for moving left, (0, 1) for moving right
                    (0, 0) for no movement, or
                    (1, 0) for moving up, (-1, 0) for moving down
        
            Returns:
                Reward given marble is within goal location bounds
        """
        # print("next->a: {}".format(a))
        # print("s: {}".format(self._s))
        
        # Store curent state
        s = self._s
        
        # Not sure why this is here but if object is iterable
        # then it selects the first index.
        # Todo: Consider removing this later
        # if isinstance(a, collections.Iterable):
        #     a = a[0]
        
        # Copies current state values into next state so they can be updated.
        s1 = copy(s)
        
        # Set velocity variable for increasing velocity over time
        dT = 0.1
        
        if a[0] != 0:
            # Update x position
            s1[0] += dT * s[2]
            # Update dx velocity (i.e., velocity of x)
            s1[2] += dT * ( 2*a[0] - 0.2 * s[2] )
        
        elif a[1] != 0:
            # Update y position
            s1[1] += dT * s[3]
            # Update dx velocity (i.e., velocity of x)
            s1[3] += dT * ( 2*a[1] - 0.2 * s[3] )

        # adjust velocity when outside of the track
        # Adjust x velocity when hitting the bounds
        # or the sides of the case/track. Set veloctiy 
        # to 0 and set x position equal to the x bound.
        if s1[0] < self.bound[0]:
            s1[0] = self.bound[0]
            #s1[2] = 0
            #s1[3] = 0
            
        elif s1[0] > self.bound[1]:
            s1[0] = self.bound[1]
            #s1[2] = 0
            #s1[3] = 0
        
        if s1[1] < self.bound[2]:
            s1[1] = self.bound[2]
            #s1[2] = 0
            #s1[3] = 0
            
        elif s1[1] > self.bound[3]:
            s1[1] = self.bound[3]
            #s1[2] = 0
            #s1[3] = 0
        
        # Clips velocity in postive and negative x direction
        s1[2] = np.clip(s1[2], self._st_range[0, 2], self._st_range[1, 2])
        s1[3] = np.clip(s1[3], self._st_range[0, 3], self._st_range[1, 3])

        # Set next state equal to current state
        self._s =  s1
        return self.get_reward(s, s1, a)

    def get_cur_state(self):
        """Returns the current state features"""
        return self._s

    def get_reward(self,s,s1,a):
        """ Returns current reward
            
            If marble's next state s1 is within a certain distance
            of the goal then a positive reward of 1 is given,
            else 0 reward is given. The distance to the goal in order 
            to recieve a positive reward is determined by self.goal_width and self.goal_hight.
        """
        return 1 if (abs(s1[0] - self.Goal_x) < self.goal_width and abs(s1[1] - self.Goal_y) < self.goal_hight) else 0

    def get_state_range(self):
        """ Returns state min and max values
        
            This is basically the same as self.boundaries 
            but given as an NumPy array.
        """
        return self._st_range

    def get_actions(self):
        """ Return all possible actions
            
            Notes:
                move left = (-1, 0)
                move right = (1, 0)
                don't move = (0, 0)
                move up = (0, 1)
                move down = (0, -1)
        """
        return np.array([[-1, 0], [1, 0], [0, 0], [0, 1], [0, -1]])

    def get_action_index(self, action):
        """ Returns index of a selected action
        
            Notes: Be aware all actions are hardcoded! 
        """
        if action[0] == -1 and action[1] == 0:
            return 0
        if action[0] == 1 and action[1] == 0:
            return 1
        if action[0] == 0 and action[1] == 0:
            return 2
        if action[0] == 0 and action[1] == 1:
            return 3
        if action[0] == 0 and action[1] == -1:
            return 4
        return -1

    def draw_trajectory(self, smplX):
        """ todo: Never used; consider removing this in future"""
        
        if smplX.shape[1] == 1: return
        plt.plot(smplX[:,0],smplX[:,1])
        plt.axis([self.bound[0], self.bound[1],-5,5])
        plt.plot(smplX[0,0],smplX[0,1],'go')
        plt.plot(self.Goal,0,'ro')
        # draw a goal region
        plt.fill_between([self.Goal-self.goal_width, self.Goal+self.goal_width],
                         [-5,-5], [5,5],
                         color="red", alpha=0.3)
        plt.xlabel("s") 
        plt.ylabel("s dot")
## Parameters
class MarbleRLAgent():
    def __init__(self, env_):
        self.env = env_
        self.K = 300
        self.n_actions = 5
        self.max_steps = 1000
        self.gamma = 0.95
        self.verbose = False

        self.epsilon = 1.
        self.final_epsilon = 0.8
        self.epsilon_decay =  np.exp(np.log(self.final_epsilon) / self.K)

        ####################
        self.nHiddens = 50
        self.nSamples = 1
        self.nOutputs = 1
        self.nInputs = 6
        self.max_iter = 10

        self.rhoh = 0.01
        self.rhoo = 0.01

        self.rh = self.rhoh / (self.nSamples * self.nOutputs)
        self.ro = self.rhoo / (self.nSamples * self.nOutputs)

        # Initialize weights to uniformly distributed values between small normally-distributed between -0.1 and 0.1
        self.V = 0.1*2*(np.random.uniform(size=(self.nInputs+1,self.nHiddens))-0.5)
        self.W = 0.1*2*(np.random.uniform(size=(1+self.nHiddens,self.nOutputs))-0.5)
        
        ####################
        # Normalize 
        self.stdX = Standardizer(self.env.get_state_range())

    # Add constant column of 1's
    def addOnes(self, A):
        return np.insert(A, 0, 1, axis=len(np.array(A).shape)-1)

    def forward(self, X):
        X = self.stdX.standardize(X)
        # Forward pass on training data
        X1 = self.addOnes(X)
        Z = np.tanh(X1 @ self.V)
        Z1 = self.addOnes(Z)
        Y = Z1 @ self.W
        return Y, Z

    def as_array(self, A):
        A = np.array(A)
        if len(A.shape) == 1:
            return A.reshape((1, -1))
        return A

    def backward(self, error, Z, X):
        ### make sure the array shapes
        X = self.as_array(X)
        Z = self.as_array(Z)
        E = self.as_array(error)

        Z1 = self.addOnes(Z)
        X1 = self.addOnes(X)

        # Backward pass - the backpropagation and weight update steps
        dV = self.rh * X1.T @ ( ( E @ self.W[1:,:].T) * (1-Z**2))
        dW = self.ro * Z1.T @ E
        return dV, dW


    def epsilon_greedy(self, e, s, env):
        actions = self.env.get_actions()
        if np.random.rand() < e:
            return np.ravel(actions[np.random.randint(self.n_actions)]).tolist()
        else:
            Q, _ = self.forward(np.hstack((np.tile(s, (5,1)), env.get_actions())))
            max_as = np.where(Q == np.max(Q))[0] # index to action value
            #return np.random.choice(max_as)
            return np.ravel(self.env.get_actions()[max_as]).tolist()


    def train(self):
        fig = plt.figure(figsize=(8, 8))

        # rewards and step trace
        rtrace = []
        etrace = [self.epsilon]
        for j in range(self.K):

            if self.verbose: print("\tepisode #", j, "   ",end="")
            self.env.init([3, 0, 0, 0]) #[float(np.random.randint(-5, 5, 1)), 0])
            s = self.env.get_cur_state()
            # selection an action
            a = self.epsilon_greedy(self.epsilon, s, self.env)

            # print("a: {}".format(a));

            rewards = []
            trace = np.array(s)
            for step in range(self.max_steps):
                if self.verbose: print("\tstep #", step, "   ",end="")
                # move
                r1 = self.env.next(a)
                s1 = self.env.get_cur_state()
                a1 = self.epsilon_greedy(self.epsilon, s1, self.env)

                rewards.append(r1)
                trace = np.vstack((trace, s1))

                # update neural networks
                Q1, _ = self.forward(np.hstack((s1, a1)))  # output of neural network is Q for next state
                Q, Z = self.forward(np.hstack((s, a)))  # output of neural network is Q for next state
                error = r1 + self.gamma * Q1 - Q  # use action value as index by adding one
                dV, dW = self.backward(error, Z, np.hstack((s, a)))
                self.V += dV
                self.W += dW

                s = s1
                a = a1

            self.epsilon *= self.epsilon_decay
            etrace.append(self.epsilon)


            if self.verbose: print("Done (", np.sum(rewards), ")", step)

            rtrace.append(np.sum(rewards))

            last_plot = (j == self.K-1)

            if j % 10 == 0 or last_plot:
                plt.clf()
                fig.add_subplot(221)
                plt.plot(rtrace, "b-")
                plt.ylabel("sum of rewards")

                fig.add_subplot(222)
                plt.plot(etrace, "-")
                plt.ylabel("p(random action), $\epsilon$")

                # plot traces
                fig.add_subplot(223)
                plt.plot(trace[:, 0], trace[:, 1], "k-")
        #         plt.fill_between([env.Goal-1, env.Goal+1], [-5, -5],[5, 5], color='red', alpha=0.3)
                plt.title("trace")
                plt.xlim([0, 10])
                plt.ylim([0, 10])

                plt.suptitle(''.join(["Episode ",str(j)]))
                plt.tight_layout(rect=[0, 0.03, 1, 0.95])
                plt.draw()

                ipd.clear_output(wait=True)
                ipd.display(fig)
        ipd.clear_output(wait=True)
K = 3000
n_actions = 5
max_steps = 1000
gamma = 0.95
verbose = False

epsilon = 1.
final_epsilon = 0.1

nHiddens = 30
nSamples = 1
nOutputs = 1
nInputs = 6
max_iter = 10

####################
# Marbel model environment 
env = Marble()

agent = MarbleRLAgent(env)
agent.train()<jupyter_output><empty_output><jupyter_text>II.I.IV. Result DiscussionFor 2D-marble problem, I plotted the `sum of rewards` in different epochs, `random action Vs. epsilon` and `traces`. I run the experiment for $300$ epochs and each epoch goes for maximum $1000$ steps. The value of the epsilon is initialized by $1$ and then decayed exponentially. From the plotting it is easily observable that the solution didn't converge and agter $180$ epochs it actually didn't able to get any further rewards. I tried with different parameters and tried to tune my deep neural network. But those experimentation didn't improve the performance. I will list my parameters and the values later.  II.II. Mountain Car II.II.I. STEPS for Mountain CarAgenda:1. [II Problems] Learn the classic mountain car problem and explain it (&check;)2. [II Problems] Learn how to use the OpenAI Gym "MountainCarContinuous-v0" environment ([OpenAI Gym MountainCarContinuous-v0](https://gym.openai.com/envs/MountainCarContinuous-v0/)) (&check;)&emsp;&emsp;2.a. Following the link, by clicking the document tab, you can learn how to install and use openai.gym3. [III Methods] Build your own RLAgent class to solve the problem. (&check;)4. [III Methods] Explain your RLAgent (with a neural network function approximator) (&check;)5. [IV Results]  Discuss the results (&check;) II.II.II. Description of Classic Mountain Car Problem [[1]](https://gym.openai.com/envs/MountainCarContinuous-v0/)A car is on a one-dimensional track, positioned between two "mountains". The goal is to drive up the mountain on the right; however, the car's engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum. Here, the reward is greater if you spend less energy to reach the goal. II.II.III. Explanation of the environmentFor this problem, I used the environment from `gym` (i.e., `MountainCar-v0`). Here is the details about the gym-environment: II.II.III.I. States| Num	| Observation	| Min	| Max |--|:--|:--|:--| 0	| position	| -1.2	| 0.6 || 1	| velocity	| -0.07	| 0.07 | II.II.III.II. Actions| Num	| Action |--|:--| 0	| push left| 1	| no push| 2	| push right II.II.III.III. Reward$-1$ for each time step, until the goal position of $0.5$ is reached. There is no penalty for climbing the left hill, which upon reached acts as a wall. I further boost the reward value during the training of the RLAgent. I will explain it in the corresponding section. II.II.IV. Explanation of the RLAgent2To solve the MountainCar control problem [[1]](https://gym.openai.com/envs/MountainCarContinuous-v0/), I implemented a RLAgent class (i.e., `MountainCarRLAgent`). In the `train()` function I implemented the on-policy control td-learning algorithm (i.e., SARSA). Again, in doing this, I followed the reading note on `Reinforcement Learning with Neural Network Function Approximation` to implement the `MountainCarRLAgent` class. Here, I used the deep neural network as the function approximator. I passed the environment as the parameter while building the `MountainCarRLAgent` class. In each step, I used the `epsilon greedy` algorithm to select the next action for the `MountainCarRLAgent`. In the training, I reset the environment by making `env.reset()` function call and start with a random state provided by the environment library.<jupyter_code>import random
import gym
import numpy as np
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import IPython.display as ipd  # for display and clear_output
import collections
from copy import deepcopy as copy

from util import Standardizer
## Parameters
class MountainCarRLAgent():
    def __init__(self, env_):
        self.env = env_
        self.K = 300
        self.n_actions = 3
        self.state_size = env.observation_space.shape[0]
        self.max_steps = 1000
        self.gamma = 0.95
        self.verbose = False

        self.epsilon = 1.
        self.final_epsilon = 0.1
        self.epsilon_decay =  np.exp(np.log(self.final_epsilon) / self.K)

        self.nHiddens = 50
        self.nSamples = 1
        self.nOutputs = 1
        self.nInputs = 3
        self.max_iter = 10

        self.rhoh = 0.01
        self.rhoo = 0.01

        self.rh = self.rhoh / (self.nSamples * self.nOutputs)
        self.ro = self.rhoo / (self.nSamples * self.nOutputs)

        # Initialize weights to uniformly distributed values between small normally-distributed between -0.1 and 0.1
        self.V = 0.1*2*(np.random.uniform(size=(self.nInputs+1,self.nHiddens))-0.5)
        self.W = 0.1*2*(np.random.uniform(size=(1+self.nHiddens,self.nOutputs))-0.5)
        
    # Add constant column of 1's
    def addOnes(self, A):
        return np.insert(A, 0, 1, axis=len(np.array(A).shape)-1)

    def forward(self, X):
        # Forward pass on training data
        X1 = self.addOnes(X)
        Z = np.tanh(X1 @ self.V)
        Z1 = self.addOnes(Z)
        Y = Z1 @ self.W
        return Y, Z

    def as_array(self, A):
        A = np.array(A)
        if len(A.shape) == 1:
            return A.reshape((1, -1))
        return A

    def backward(self, error, Z, X):
        ### make sure the array shapes
        X = self.as_array(X)
        Z = self.as_array(Z)
        E = self.as_array(error)

        Z1 = self.addOnes(Z)
        X1 = self.addOnes(X)

        # Backward pass - the backpropagation and weight update steps
        dV = self.rh * X1.T @ ( ( E @ self.W[1:,:].T) * (1-Z**2))
        dW = self.ro * Z1.T @ E
        return dV, dW
    
    
    def get_actions(self):
        return np.array([0, 1, 2])


    def epsilon_greedy(self, e, s, env):
        if np.random.rand() < e:
            return random.randrange(self.n_actions)
        else:
            Q, _ = self.forward(np.hstack((np.tile(s, (3,1)), self.get_actions().reshape(-1, 1))))
            return np.argmax(Q[0])
        

    def train(self):
        fig = plt.figure(figsize=(8, 8))

        # rewards and step trace
        rtrace = []
        etrace = [self.epsilon]
        for j in range(self.K):

            if self.verbose: print("\tepisode #", j, "   ",end="")
            s = env.reset()
            
            # selection an action
            a = self.epsilon_greedy(self.epsilon, s, self.env)

            rewards = []
            trace = np.array(s)
            for step in range(self.max_steps):
                if self.verbose: print("\tstep #", step, "   ",end="")
                # move
                s1, r1, done, info = env.step(a)
                a1 = self.epsilon_greedy(self.epsilon, s1, self.env)
                
                # print("s: {}".format(s))
                # print("s.shape: {}".format(s.shape))
                # print("s1: {}".format(s1))
                # print("s1.shape: {}".format(s1.shape))
                
                if s1[1] > s[1] and s1[1]>0 and s[1]>0:
                    r1 += 15
                
                elif s1[1] < s[1] and s1[1]<=0 and s[1]<=0:
                    r1 += 15

                # give more reward if the cart reaches the flag in max_steps
                if done:
                    r1 += 10000
                else:
                    # put a penalty if the no of time steps is more
                    r1 -= 10

                rewards.append(r1)
                trace = np.vstack((trace, s1))

                # update neural networks
                Q1, _ = self.forward(np.hstack((s1, a1)))  # output of neural network is Q for next state
                Q, Z = self.forward(np.hstack((s, a)))  # output of neural network is Q for next state
                error = r1 + self.gamma * Q1 - Q  # use action value as index by adding one
                dV, dW = self.backward(error, Z, np.hstack((s, a)))
                self.V += dV
                self.W += dW

                s = s1
                a = a1

            self.epsilon *= self.epsilon_decay
            etrace.append(self.epsilon)


            if self.verbose: print("Done (", np.sum(rewards), ")", step)

            rtrace.append(np.sum(rewards))

            last_plot = (j == self.K-1)

            if j % 10 == 0 or last_plot:
                plt.clf()
                fig.add_subplot(221)
                plt.plot(rtrace, "b-")
                plt.ylabel("sum of rewards")

                fig.add_subplot(222)
                plt.plot(etrace, "-")
                plt.ylabel("p(random action), $\epsilon$")


                plt.suptitle(''.join(["Episode ",str(j)]))
                plt.tight_layout(rect=[0, 0.03, 1, 0.95])
                plt.draw()

                ipd.clear_output(wait=True)
                ipd.display(fig)
        ipd.clear_output(wait=True)
env = gym.make('MountainCar-v0')

state_size = env.observation_space.shape[0]
action_size = env.action_space.n

print("states: {}".format(env.observation_space))
print("actions: {}".format(env.action_space))

print('state size: {}'.format(state_size))
print('action size: {}'.format(action_size))

agent = MountainCarRLAgent(env)
agent.train()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11521455494346187, 'fraction_numerical': 0.022144099739054797, 'mean_word_length': 2.0044642857142856, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 18, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '628864', 'license': 'MIT', 'n_tokens_mistral': 8917, 'n_tokens_neox': 8374, 'n_words': 2776, 'path': 'programming_assignments/4_Reinforcement-Learning/Assign4.ipynb', 'repo_name': 'biqar/Fall-2020-ITCS-8156-MachineLearning', 'size': '27611'}