<jupyter_start><jupyter_text>______Copyright by <NAME> Inc.For more information, visit us at www.pieriandata.com  Q Learning Exercise  **We'll be reviewing and testing your skills with Q-Learning on a continuous space! Please feel free to reference the lecture notebooks you are definitely not expected to be able to fill out all this code from memory, just the ability to understand the core concepts and apply it to a different situation.**-------------------- Complete the tasks in bold below. In this exercise we take a look at the MountainCar-v0 (https://gym.openai.com/envs/MountainCar-v0/) game again. That is the game from our original discussion of OpenAi gym environments for which we created an agent manually.Remember, that the goal is to reach the top of the mountain within some time limit----- **TASK: Import any relevant libraries you think you might need.**  **TASK: Create the gym mountain car environment**  **TASK: Write a function to create a numpy array holding the bins for the observations of the car (position and velocity).** Feel free to explore different bins per observation spacings.The function should take one argument which acts as the bins per observation Hint: You can find the observations here: https://github.com/openai/gym/blob/master/gym/envs/classic_control/mountain_car.py Hint: You will probably need around 25 bins for good results, but feel free to use less to reduce training time.<jupyter_code>def create_bins(num_bins_per_observation):
    # CODE HERE
    return bins<jupyter_output><empty_output><jupyter_text>**TASK: Here you should write the code which creates the bins and defines the NUM_BINS attribute** **TASK: Create a function that will take in observations from the environment and the bins array and return the discretized version of the observation.** Now we need the code to discretize the observations. We can use the same code as used in the last notebook<jupyter_code>def discretize_observation(observations, bins):
    # CODE HERE
    return tuple(binned_observations) # Important for later indexing<jupyter_output><empty_output><jupyter_text>**Let's check to make sure your previous two function calls work with a quick task! Otherwise it may be hard to debug later on.** **TASK: Confirm that your *create_bins()* function works with *discretize_observation() by running the following cell***<jupyter_code>test_bins = create_bins(5)
np.testing.assert_almost_equal(test_bins[0], [-1.2 , -0.75, -0.3 ,  0.15,  0.6])
np.testing.assert_almost_equal(test_bins[1], [-0.07 , -0.035,  0.   ,  0.035,  0.07 ])

test_observation = np.array([-0.9, 0.03])
discretized_test_bins = discretize_observation(test_observation, test_bins)
assert discretized_test_bins == (1, 3)<jupyter_output><empty_output><jupyter_text>**TASK: Create the Q-Table** Remember the shape that the Q-Table needs to have.<jupyter_code># CREATE THE Q TABLE<jupyter_output><empty_output><jupyter_text>**TASK: Fill out the Epislon Greedy Action Selection function:**<jupyter_code>def epsilon_greedy_action_selection(epsilon, q_table, discrete_state):
    #CODE HERE
    return action<jupyter_output><empty_output><jupyter_text>**TASK: Fill out the function to compute the next Q value.**<jupyter_code>def compute_next_q_value(old_q_value, reward, next_optimal_q_value):
    
    return # CODE HERE<jupyter_output><empty_output><jupyter_text>**TASK: Create a function to reduce epsilon, feel free to choose any reduction method you want. We'll use a reduction with BURN_IN and EPSILON_END limits in the solution. We'll also show a way to reduce epsilon based on the number of epochs. Feel free to experiment here.**<jupyter_code>def reduce_epsilon(epsilon, epoch):
    # CODE HERE
    return epsilon<jupyter_output><empty_output><jupyter_text>**TASK: Define your hyperparameters. Note, we'll show our solution hyperparameters here, but depending on your *reduce_epsilon* function, your epsilon hyperparameters may be different.** Here are the solution initial hyperparameters, your will probably be different!<jupyter_code># Feel free to change!

EPOCHS = 30000
BURN_IN = 100
epsilon = 1

EPSILON_END= 10000
EPSILON_REDUCE = 0.0001 

ALPHA = 0.8
GAMMA = 0.9<jupyter_output><empty_output><jupyter_text>**TASK: Create the training loop for the reinforcement learning agent and run the loop. We've gone ahead and created the basic structure of the loopwith some comments. We also pre-filled the visualization portion.** Note: Use the lecture notebook as a guide and reference<jupyter_code>####### VISUALIZATION CODE FOR YOU. TOTALLY OPTIONAL. ##########################
########## FEEL FREE TO REMOVE OR ADD YOUR OWN VISUAL CODE. #################

log_interval = 100  # How often do we update the plot? (Just for performance reasons)
### Here we set up the routine for the live plotting of the achieved points ######
fig = plt.figure()
ax = fig.add_subplot(111)
plt.ion()
fig.canvas.draw()

max_position_log = []  # to store all achieved points
mean_positions_log = []  # to store a running mean of the last 30 results
epochs = []  # store the epoch for plotting


#############################################################################
################################## TRAINING TASKS ##########################
###########################################################################
for epoch in range(EPOCHS):
    ################################# TODO ######################################
    
    # TODO: Get initial observation and discretize them. Set done to False
    #########################################
    #
    #
    #            CODE NEEDED HERE!!!
    #
    #
    ##########################################
    
    #############################
    # These lines are for plotting.
    max_position = -np.inf  
    epochs.append(epoch)
    #############################
    
    # TASK TO DO: As long as current run is alive (i.e not done) perform the following steps:
    while not done:  # Perform current run as long as done is False (as long as there is still time to reach the top)


        # TASK TO DO: Select action according to epsilon-greedy strategy
        #########################################
        #
        #
        #            CODE NEEDED HERE!!!
        #
        #
        ##########################################
        # TASK TO DO: Perform selected action and get next state. Do not forget to discretize it
        #########################################
        #
        #
        #            CODE NEEDED HERE!!!
        #
        #
        ##########################################
        # TASK TO DO: Get old Q-value from Q-Table and get next optimal Q-Value
        #########################################
        #
        #
        #            CODE NEEDED HERE!!!
        #
        #
        ##########################################
        # TASK TO DO: Compute next Q-Value and insert it into the table
        #########################################
        #
        #
        #            CODE NEEDED HERE!!!
        #
        #
        ##########################################
        # TASK TO DO: Update the old state with the new one
        #########################################
        #
        #
        #            CODE NEEDED HERE!!!
        #
        #
        ##########################################
        
        
        ##############################
        ##  Only for plotting the results - store the highest point the car is able to reach
        if position > max_position:  
            max_position = position

    # TASK TO DO: Reduce epsilon
    #########################################
    #
    #
    #            CODE NEEDED HERE!!!
    #
    #
    ##########################################
    ##############################################################################

    max_position_log.append(max_position)  # log the highest position the car was able to reach
    running_mean = round(np.mean(max_position_log[-30:]), 2)  # Compute running mean of position over the last 30 epochs
    mean_positions_log.append(running_mean)  # and log it
    
    ################ Plot the points and running mean ##################
    if epoch % log_interval == 0:
        ax.clear()
        ax.scatter(epochs, max_position_log)
        ax.plot(epochs, max_position_log)
        ax.plot(epochs, mean_positions_log, label=f"Running Mean: {running_mean}")
        plt.legend()
        fig.canvas.draw()
  ######################################################################

env.close()<jupyter_output><empty_output><jupyter_text>**TASK: Use your Q-Table to test your agent and render its performance.**<jupyter_code># CODE HERE<jupyter_output>done<jupyter_text>**OPTIONAL: Play with our Q-Table with 40 bins per observation.**<jupyter_code># We saved our matrix q_table for you (with 40 bins)
our_q_table = np.load('40bin_qtable_mountaincar.npy')
our_q_table.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2012965239745166, 'fraction_numerical': 0.009388621884430536, 'mean_word_length': 3.844612885760693, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '494450', 'license': 'MIT', 'n_tokens_mistral': 2534, 'n_tokens_neox': 2343, 'n_words': 864, 'path': 'practical_ai/archive/06-Classical-Q-Learning/02-Q-Learning-Exercise.ipynb', 'repo_name': 'Anshul-GH/machine_learning_advanced', 'size': '8953'}