<jupyter_start><jupyter_text>Applied Deep Learning Tutorial contact: <EMAIL> Deep Reinforcement Learning with Deep-Q-Network (DQN) IntroductionIn this tutorial, you will attempt to implement a Deep-Q-Network that is able to do a classic control. The approaches are build upon the paper by DeepMind: Playing Atari with Deep Reinforcement Learning [paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf), which first introduces the notion of a Deep Q-Network. Fig. 1: Breakout environment of the Atari game  Core ideaAs you probably remember from the lecture, during trial and error we can learn a policy for our Atari game, and model it within our Q-matrix. This is done with a deep neural network. After training, this Q-matrix gives us an estimate of the expected reward when taking action a in state s: Q(s, a).Playing the action with the maximum Q-value in any given state is the same as playing optimal, or following a full exploitation strategy. OpenAI Gym[OpenAI Gym](https://gym.openai.com/docs/) is a library that can simulate a large number of reinforcement learning environments, including Atari games (these need to be installed additionaly). You will need Python 3.5+>pip install gym  Taking our cart pole on a first rideNow that you have gym installed you can load the 'Pendulum-v0' environment of Atari.<jupyter_code># Import the gym module
import gym

import gym
env = gym.make('CartPole-v0')
env.reset()
for _ in range(1000):
    env.render()
    env.step(env.action_space.sample()) # take a random action
env.close()

# Load the environment
env = gym.make('Pendulum-v0')

# Reset, it returns the starting frame
frame = env.reset()

for _ in range(100000):
    # Perform a random action, returns the new frame, reward and whether the game is over
    
    '''
    Implement to sample a random action from the action space within the loaded environment
    action = environment.action_space.sample()
    '''
    action = env.action_space.sample()
    observation, reward, is_done, info = env.step(action)
    print('observation: ', observation, 'reward: ', reward)
    print('info :', info)
    if is_done: break
    env.render()

env.close()

env = gym.make('Pendulum-v0')

# Reset, it returns the starting frame
frame = env.reset()
#print('Actions:')
#print(env.action_space)
#print(env.action_space.high)
#print(env.action_space.low)
#print('Observations:')
#print(env.observation_space)
#print(env.observation_space.high)
#print(env.observation_space.low)

env.close()<jupyter_output><empty_output><jupyter_text>This already looks nice, yet the actions are random and thus it is time to better understand our environment. And to implement our Deep-Q-Network<jupyter_code># import the necessary libraries
import gym
import gym.spaces
import gym.wrappers
import numpy as np
import random
import pickle
from collections import deque
from keras.layers import Flatten, Dense, Input
from keras import backend as K
from keras.models import Sequential, Model, load_model
from keras import optimizers<jupyter_output>Using TensorFlow backend.<jupyter_text>ObservationThe observation is made up of cos(theta), sin(theta) and theta dot. Theta is normalized between -pi and pi. ActionJoint effort -2.0 to +2.0Write a function to discretize the continuous action space of the joint effort.<jupyter_code># define the action space
def create_action_bins(num_action_bins):
    '''
    Using linspace of numpy implement the action bins for the pendulum, when given the number of the action bins as argument
    actionbins = 
    numpy.linspace(start, stop, num = 50, endpoint = True, retstep = False, dtype = None) : 
    Returns number spaces evenly w.r.t interval. Similar to arange but instead of step it uses sample number.
    '''
    actionbins = np.linspace(-2.0, 2.0, num_action_bins)
    return actionbins

# depending on the action, find the according actionbin 
# discretization of the continuous action space
def find_actionbin(action, actionbins):
    idx = (np.abs(actionbins - action)).argmin()

    return idx<jupyter_output><empty_output><jupyter_text>RewardThe reward is defined as> -(theta^2 + 0.1 x theta_dt^2 + 0.001 x action^2)What is the lowest expected cost? And what is the highest cost?-(pi^2 + 0.1 x 8^2 + 0.001 x 2^2) = -16.2736044 - lowest reward-(0^2 + 0.1 x 0^2 + 0.001 x 0^2) = 0 - highest rewardFrom this reward function, what is the goal of the agent?In essence, the goal is to remain at zero angle (vertical), with the least rotational velocity, and the least effort.For a hint have a look at the [wiki](https://github.com/openai/gym/wiki).  train_model() FunctionHier wird das Modell auf einem Batch in Memory gespeicherter Erfahrungen trainiert<jupyter_code>def train_model(memory, gamma=0.9):
    for state, action, reward, state_new in memory:
        
        # flatten state to make it compatible to our neural network
        flat_state_new = np.reshape(state_new, [1, 3])
        flat_state = np.reshape(state, [1, 3])

        # determine estimated reward given state s' after action a, 
        # combination of observed and predicted exploited reward.
        '''
        Implement the Q-function for the flat_state_new
        target = 
        Target value is r + gamma * max[Qs in the new state] 
        '''
        target = reward + gamma * np.amax(model.predict(flat_state_new))
        # determine current expected agent rewards
        targetfull = model.predict(flat_state)
        
        # update current expected rewards with the emulated prediced reward
        targetfull[0][action] = target
        
        # Fit model based on emulation and prediction
        model.fit(flat_state, targetfull, epochs=1, verbose=0)<jupyter_output><empty_output><jupyter_text>Deep Q ModelAs a reminder, this is our Q function.> Q(s, a) = r + gamma max_a'(Q(s, a'))The input of our neural network, our generalizable Q-matrix, will be the observation or the state of the pendulum. and the output will be the estimate of the reward taking the action a'. Gamma is the discount factor of the predicted reward in our next state. r is the reward  For our first network we will implement a DQN with keras:- Layer with 128 ReLU units- Layer with 64 ReLU units- 3 inputs and one output per action bin with linear activation function- Adam optimizer with learning rate 0.0002, beta_1 0.9 and beta_2 0.999- Loss mean squared error<jupyter_code># Define the Deep-Q-Network in keras

def build_model(num_output_nodes): 
    model = Sequential()  

    model.add(Dense(128, input_shape=(3,), activation='relu'))
    model.add(Dense(64, activation='relu'))
    model.add(Dense(num_output_nodes, activation='linear'))
    
    model.compile(loss='mse', optimizer=optimizers.Adam(lr=0.0002, beta_1=0.9, beta_2=0.999))

    return model<jupyter_output><empty_output><jupyter_text>run_episodes()-FunctionHier wird das Prinzip der "Memory Replay" umgesetzt:                Das Gradientenabstiegsverfahren wird nicht auf den Erfahrungen nach einer einzelner Aktion ausgeführt, sondern auf einem Minibatch von Erfahrungen (st, at, rt, st+1). Dazu wird erst eine Aktion getätigt und deren Belohnung r und Folgezustand st+1 im Spiel beobachtet. Diese werden werden zusammen mit dem Ausgangszustand s in einer Replay-Memory gespeichert (250 Erfahrungen pro Sequence). An dieser Stelle wird nun kein Gradientenabstiegsverfahren mit der aktuellsten Erfahrung angestoßen (model.fit()). Stattdessen wird das Gradientenabstiegsverfahren auf einem Batch von 250 Erfahrungen, die in Memory gespeichert waren, durchgeführt (siehe train_model()).<jupyter_code>def run_episodes(epsilon, gamma, training_iterations, sequence_iterations):
    
    # These are hyperparameters to play around with, after your first run.
    epsilon_decay = 0.9999
    epsilon_min = 0.02
    steps_per_sequence = 250

    for epoch in range(0, training_iterations // sequence_iterations):
        for sequence_id in range(0, sequence_iterations):
            state = env.reset()
            memory = deque()
            
            total_reward = 0
            
            # Easy implementation of decaying exploration

            if epsilon > epsilon_min:
                epsilon = epsilon * epsilon_decay
                
            for i in range(0, steps_per_sequence):
                
                # explore
                if np.random.uniform() < epsilon:
                    action = env.action_space.sample()
                # exploit
                else:
                    flat_state = np.reshape(state, [1, 3])
                    action = np.amax(model.predict(flat_state))

                # determine action
                actionbin = find_actionbin(action, actionbinslist)
                action = actionbinslist[actionbin]
                action = np.array([action])

                # emulate the action in the simulation and observe the transition 
                # as well as the reward
                observation, reward, done, _ = env.step(action)
                total_reward += reward

                state_new = observation

                '''
                save transitions into memory
                Hint: The memory is used as an argument for the train_model function.
                
                memory.append((_, _, _, _))
                
                '''
                memory.append((state, actionbin, reward, state_new))
                state = state_new
                
            # train model on the samples memories
            train_model(memory, gamma)
            
            print(epoch , ' epoch', sequence_id, ' sequence. Average reward = ', total_reward / steps_per_sequence, '. epsilon = ', epsilon)<jupyter_output><empty_output><jupyter_text>Function for running the policy of our DQN after loading or training<jupyter_code>def play_game(rounds):
    state = env.reset()
    totalreward = 0

    for _ in range(0, rounds):
        # Rendering for visualization
        env.render()

        flat_state = np.reshape(state, [1, 3])
        actionbin = np.argmax(model.predict(flat_state))

        action = actionbinslist[actionbin]
        #print(action)
        action = np.array([action])
        
        observation, reward, done, _ = env.step(action)

        totalreward += reward

        state_new = observation
        state = state_new
        
    return totalreward<jupyter_output><empty_output><jupyter_text>Train the DQN<jupyter_code>env = gym.make('Pendulum-v0')

# These are hyperparameters to play around with

# iterations
training_iterations = 1000
sequence_iterations = 25

# epsilon (setting exploitation vs exploration)
epsilon = 1

# gamma (importance of predicted estimated reward)
gamma = 0.9

# Discretization settings for the action space
num_action_bins = 40
actionbinslist = create_action_bins(num_action_bins)



# Build model
model = build_model(num_action_bins)

run_episodes(epsilon, gamma, training_iterations, sequence_iterations)

'''
training takes super long, this is not efficient at all, how can we bypass this?
Hint: See cells in run pretrained model.

'''<jupyter_output>0  epoch 0  sequence. Average reward =  -6.977471208280163 . epsilon =  0.9999
0  epoch 1  sequence. Average reward =  -5.653621976237476 . epsilon =  0.9998000100000001
0  epoch 2  sequence. Average reward =  -4.84072767854339 . epsilon =  0.9997000299990001
0  epoch 3  sequence. Average reward =  -4.824327027086468 . epsilon =  0.9996000599960002
0  epoch 4  sequence. Average reward =  -8.60759402263013 . epsilon =  0.9995000999900007
0  epoch 5  sequence. Average reward =  -6.100504849267558 . epsilon =  0.9994001499800017
0  epoch 6  sequence. Average reward =  -6.082940472179801 . epsilon =  0.9993002099650037
0  epoch 7  sequence. Average reward =  -4.8540300082823915 . epsilon =  0.9992002799440072
0  epoch 8  sequence. Average reward =  -8.200375797151672 . epsilon =  0.9991003599160128
0  epoch 9  sequence. Average reward =  -4.032207818563496 . epsilon =  0.9990004498800211
0  epoch 10  sequence. Average reward =  -7.99281440524208 . epsilon =  0.9989005498350332
0  epoch 11 [...]<jupyter_text>Exploration vs. ExploitationNach 10000 Iterationen wurde epsilon im Laufe des Training von 0.9999 (reine Exploration) auf ca 0.3679 minimiert. Am Anfang eines Trainings ist der Zustandsraum unbekannt und bei hohem epsilon werden viele sehr unterschiedliche Zustände "erkundet". Mit dem fortgeschrittenen Training werden mehr und mehr Zustände bekannt und bewertet. Das Nezt wird "erwachsen" und greift mehr und mehr auf die eigene Erfahrungen - Exploitation.<jupyter_code># Save model weights

print('saving model')
model.save('pendulum_model_40_ac_jm-201414_' + str(training_iterations) + '.h5')
print('model saved')

# Evaluate performance on 10 test runs with 100 steps each
trarray = []
rounds = 100
for i in range(10):
    trarray.append(play_game(rounds))
    print(i, ' sequence. Average test reward = ', np.average(trarray)/rounds, 'Average test reward = ', trarray[-1]/rounds)<jupyter_output>0  sequence. Average test reward =  -1.3120282218178363 Average test reward =  -1.3120282218178363
1  sequence. Average test reward =  -1.2969119063628551 Average test reward =  -1.2817955909078735
2  sequence. Average test reward =  -1.2886135875950477 Average test reward =  -1.2720169500594327
3  sequence. Average test reward =  -2.1157535554307088 Average test reward =  -4.597173458937691
4  sequence. Average test reward =  -1.9530118143206436 Average test reward =  -1.3020448498803838
5  sequence. Average test reward =  -1.8360624174484612 Average test reward =  -1.25131543308755
6  sequence. Average test reward =  -1.5746747970625505 Average test reward =  -0.006349074747084901
7  sequence. Average test reward =  -1.5392880288045985 Average test reward =  -1.2915806509989332
8  sequence. Average test reward =  -1.506911758676764 Average test reward =  -1.2479015976540895
9  sequence. Average test reward =  -1.599464729611906 Average test reward =  -2.4324414680281823<jupyter_text>Run pretrained modelIn case you already trained a model or want to load the pretrained model for sanity checking use the following script (make sure you executed the necessary cells starting with the imports).- How does the performance change with the amount of trained iterations?- How can we measure performance to begin with?- Is it sufficient to start the play_game function a single time? - How can we make sure, that the evaluation is meaningful?<jupyter_code>env = gym.make('Pendulum-v0')
trarray = []
rounds = 250
actionbinslist = create_action_bins(40)

# 'pendulum_model_[iterationstrained].h5' 
# iterationstrained: 100, 1000, 10000
#model = load_model('pendulum_model_100.h5')
#model = load_model('pendulum_model_jm-201414_1000.h5')
#model = load_model('pendulum_model_jm-201414_10000.h5')

model = load_model('pendulum_model_40_ac_jm-201414_1000.h5')
#model = load_model('pendulum_model_10_ac_jm-201414_1000.h5')
'''
Is the next line meaningful for evaluation, if not, what can we do?

play_game(rounds=250)
'''    
for i in range(20):
    trarray.append(play_game(rounds))
    print(i, ' sequence. Average reward for games played = ', np.average(trarray)/rounds, '\n       Average test reward for the last game = ', trarray[-1]/rounds, '\n')
    
env.close()<jupyter_output>0  sequence. Average reward for games played =  -0.531276722599101 
       Average test reward for the last game =  -0.531276722599101 

1  sequence. Average reward for games played =  -3.2393817859879497 
       Average test reward for the last game =  -5.947486849376798 

2  sequence. Average reward for games played =  -2.334881269330553 
       Average test reward for the last game =  -0.5258802360157605 

3  sequence. Average reward for games played =  -3.3468485314415357 
       Average test reward for the last game =  -6.382750317774482 

4  sequence. Average reward for games played =  -3.8831413506094528 
       Average test reward for the last game =  -6.028312627281122 

5  sequence. Average reward for games played =  -3.3236877321097817 
       Average test reward for the last game =  -0.5264196396114282 

6  sequence. Average reward for games played =  -2.850572216190284 
       Average test reward for the last game =  -0.011879120673297076 

7  sequence. Average reward for [...]
{'fraction_non_alphanumeric': 0.0784434078075129, 'fraction_numerical': 0.07568131598330469, 'mean_word_length': 3.4528559715769336, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 42, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '380530', 'license': 'MIT', 'n_tokens_mistral': 5642, 'n_tokens_neox': 4841, 'n_words': 1702, 'path': 'ADL_JupyterNotebooks/ADL_08_Tutorial/DQN-TUT_Julia_Mueller.ipynb', 'repo_name': 'jm-201414/DeepLearningLecture_RWU', 'size': '16306'}