<jupyter_start><jupyter_text>Deep Q-Learning:In deep Q-learning, we use a neural network to approximate the Q-value function. The state is given as the input and the Q-value of all possible actions is generated as the output.![image.png](attachment:image.png)Steps:1. All the past experience is stored by the user in memory2. The next action is determined by the maximum output of the Q-network3. The loss function here is mean squared error of the predicted Q-value and the target Q-value – Q*.     This is basically a regression problem. However, we do not know the target or actual value here     as we are dealing with a reinforcement learning problem. 4. Going back to the Q-value update equation derived fromthe Bellman equation.  Import Libraries:<jupyter_code>import time
from time import sleep
import random
from itertools import count
import matplotlib.pyplot as plt

import gym_super_mario_bros
from gym_super_mario_bros.actions import SIMPLE_MOVEMENT, RIGHT_ONLY, COMPLEX_MOVEMENT
from nes_py.wrappers import JoypadSpace

import numpy as np

from DeepQ_Network import DQNetwork
from replay_from_memory import ReplayMemory
from wrappers import wrapper<jupyter_output>Using TensorFlow backend.<jupyter_text>Setting Hyperparameters:<jupyter_code>FRAME_DIM = (84, 84, 4)
ACTION_SPACE = RIGHT_ONLY   # We can also use other action space COMPLEX_MOVEMENT( for left, right, up and down actions)
REPLAY_MEMORY_CAPACITY = 200_000
NUM_EPISODES = 1001
MAX_STEPS_PER_GAME = 1000
GAMMA = 0.99
RENDER_ENVIRONMENT = False

BATCH_SIZE = 128
TRAIN_FREQUENCE = 4  # number of total steps after which the policy model is trained
TARGET_NETWORK_UPDATE_FREQUENCE = 40_000  # number of total steps after which the weights of the policy model are copied to the target model

EXPLORATION_MAX = .8
EXPLORATION_MIN = 0.05
EXPLORATION_STEPS = 1_000_000
EXPLORATION_DECAY = (EXPLORATION_MAX - EXPLORATION_MIN) / EXPLORATION_STEPS<jupyter_output><empty_output><jupyter_text>Tensorflow Video Emedding<jupyter_code>def embed_mp4(filename):
    """Embeds an mp4 file in the notebook."""
    video = open(filename,'rb').read()
    b64 = base64.b64encode(video)
    tag = '''
    <video width="640" height="480" controls>
    <source src="data:video/mp4;base64,{0}" type="video/mp4">
    Your browser does not support the video tag.
    </video>'''.format(b64.decode())

    return IPython.display.HTML(tag)<jupyter_output><empty_output><jupyter_text>Creating Video<jupyter_code>def create_policy_eval_video(policy, filename, num_episodes=5, fps=30):
    filename = filename + ".mp4"
    with imageio.get_writer(filename, fps=fps) as video:
        for _ in range(num_episodes):
            time_step = eval_env.reset()
            video.append_data(eval_py_env.render())
        while not time_step.is_last():
            action_step = policy.action(time_step)
            time_step = eval_env.step(action_step.action)
            video.append_data(eval_py_env.render())
    return embed_mp4(filename)

# create_policy_eval_video(agent.policy, "trained-agent")<jupyter_output><empty_output><jupyter_text>select next action:Based of exploration rate select next action<jupyter_code>def get_next_action(state, action_space, current_exploration):
    """Returns the next action."""
    if np.random.rand() < current_exploration:
        return random.randrange(action_space)
    else:
        predicted_q_values = policy_net.model.predict(np.expand_dims(np.asarray(state).astype(np.float64), axis=0),
                                                      batch_size=1)
        return np.argmax(predicted_q_values[0])<jupyter_output><empty_output><jupyter_text>Modeling: Trains the policy net on a batch from the replay memory.<jupyter_code>def train_policy_model(replay_memory, policy_model, target_model):

    # if there are not enough transitions yet don't train
    if len(replay_memory) < BATCH_SIZE:
        return

    batch = replay_memory.sample(BATCH_SIZE)

    # get the target q values of the target model
    current_states = []
    target_predictions = []
    for state, action, reward, next_state, done in batch:
        # convert the states
        current_state = np.expand_dims(np.asarray(state).astype(np.float64), axis=0)
        next_state = np.expand_dims(np.asarray(next_state).astype(np.float64), axis=0)

        # get the prediction of the target network and the current policy network
        target_prediction = target_model.predict(next_state)
        target_q_value = np.max(target_prediction)
        current_prediction = policy_model.predict(current_state)[0]

        # calculate the new q values
        if done:
            current_prediction[action] = reward
        else:
            current_prediction[action] = reward + GAMMA * target_q_value

        current_states.append(current_state)
        target_predictions.append(target_prediction)

    # train the policy model based on the predictions of the target model
    policy_model.fit(np.asarray(current_states).squeeze(), np.asarray(target_predictions).squeeze(),
                     batch_size=BATCH_SIZE, verbose=0)<jupyter_output><empty_output><jupyter_text>Check Available GPU:<jupyter_code>import tensorflow as tf
from tensorflow import keras
from tensorflow.python.client import device_lib

print(tf.__version__)

def check_GPU():
    local_devices = device_lib.list_local_devices()
    print("local_devices: ", local_devices)
    print("Available GPUs: ", tf.config.experimental.list_physical_devices('GPU'))
    print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
    gpus = tf.config.experimental.list_physical_devices('GPU')
    if gpus:
        try:
            # Currently, memory growth needs to be the same across GPUs
            for gpu in gpus:
                tf.config.experimental.set_memory_growth(gpu, True)
            logical_gpus = tf.config.experimental.list_logical_devices('GPU')
            print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
        except RuntimeError as e:
            # Memory growth must be set before GPUs have been initialized
            print(e)


check_GPU()<jupyter_output>1.15.0
local_devices:  [name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 12006859598251084274
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 10456358168899386685
physical_device_desc: "device: XLA_CPU device"
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 11897235047
locality {
  bus_id: 1
  links {
  }
}
incarnation: 10724935243077615978
physical_device_desc: "device: 0, name: Tesla P100-PCIE-12GB, pci bus id: 0000:04:00.0, compute capability: 6.0"
, name: "/device:XLA_GPU:0"
device_type: "XLA_GPU"
memory_limit: 17179869184
locality {
}
incarnation: 13690112305508995353
physical_device_desc: "device: XLA_GPU device"
]
Available GPUs:  [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Num GPUs Available:  1
1 Physical GPUs, 1 Logical GPUs<jupyter_text>Train Mario<jupyter_code>env = gym_super_mario_bros.make("SuperMarioBros-v0")
env = JoypadSpace(env, ACTION_SPACE)    # An environment wrapper to convert binary to discrete action space.
# apply the wrapper
env = wrapper(env, FRAME_DIM)

# create the network
policy_net = DQNetwork(stacked_frame_dim=FRAME_DIM, num_actions=env.action_space.n)
target_net = DQNetwork(stacked_frame_dim=FRAME_DIM, num_actions=env.action_space.n)

# create the replay memory
replay_memory = ReplayMemory(REPLAY_MEMORY_CAPACITY)

# play the episodes
current_exploration = EXPLORATION_MAX
total_steps = 0
reward_history = []
mean_reward_history = []
for episode in range(NUM_EPISODES):
    state = env.reset()

    # play one game
    current_reward = 0
    for steps in count(MAX_STEPS_PER_GAME):
        # render the environment
        if RENDER_ENVIRONMENT:
            env.render()

        # get the next action
        action = get_next_action(state, env.action_space.n, current_exploration)

        # perform the action
        next_state, reward, done, info = env.step(action)

        # if we are done end the loop
        if done or info["life"] < 2:
            print("Episode: ", episode, "reward: ", current_reward)
            reward_history.append(current_reward)
            mean_reward_history.append(np.mean(reward_history))
            break

        # add the transition to the replay memory
        replay_memory.push(state, action, reward, next_state, done)

        # increase the current reward and the total steps
        current_reward += reward
        total_steps += 1

        # train the policy network
        if total_steps % TRAIN_FREQUENCE == 0:
            train_policy_model(replay_memory, policy_net.model, target_net.model)

        # update the weights of the target model if necessary
        if total_steps % TARGET_NETWORK_UPDATE_FREQUENCE == 0:
            print("Updated the target Model")
            target_net.model.set_weights(policy_net.model.get_weights())

        # update the exploration rate
        current_exploration -= EXPLORATION_DECAY
        current_exploration = max(EXPLORATION_MIN, current_exploration)

    # plot the reward history
    if (episode + 1) % 100 == 0:
        plt.figure(figsize=(12,6))
        plt.plot(reward_history, "#00b4f5", label = "Reward History")
        plt.plot(mean_reward_history, "#c8eb02", label = "Mean Reward History")
        plt.ylabel("Rewards", fontsize = 18)
        plt.xlabel("Episodes",fontsize = 18)
        plt.title("Reward Per Episode", fontsize = 18)
        plt.legend()
        plt.show()<jupyter_output>WARNING:tensorflow:From /home/nadkar.k/.conda/envs/RL_GPU_TF15/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /home/nadkar.k/.conda/envs/RL_GPU_TF15/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/nadkar.k/.conda/envs/RL_GPU_TF15/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /home/nadkar.k/.conda/envs/RL_GPU_TF15/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /home/nadkar.k/.conda/envs/RL_GPU_TF15/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: [...]
{'fraction_non_alphanumeric': 0.09049050930276265, 'fraction_numerical': 0.028472091712084195, 'mean_word_length': 3.994368840919756, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '434242', 'license': 'MIT', 'n_tokens_mistral': 3616, 'n_tokens_neox': 3379, 'n_words': 821, 'path': 'DoubleDQN/train_DDQN_mario.ipynb', 'repo_name': 'KailashDN/Deep_Reinforcement_Learning_Gym', 'size': '10642'}