<jupyter_start><jupyter_text>Practical Session: Q-Learning In this practical session, you will implement the famous [Q-Learning](https://link.springer.com/content/pdf/10.1007/BF00992698.pdf) algorithm and test it in various environments.  Taxi V3The taxi problem was first introduced in [Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition](https://arxiv.org/abs/cs/9905014).  In this environment, the agent controls a taxi whose job is to pick up a passenger at one location and drop him off in his target location.  Dropping of the passenger to its destination leads to a +20 reward.  To encourage the agent to be as fast as possible, it receives a deceptive reward of -1 at each step.  It will also receive a -10 penalty reward if it takes an illegal 'pick-up' or drop-off' action.  Let's instantiate a Taxi-V3 environment:<jupyter_code>import gym
env = gym.make("Taxi-v3")
env.reset()
env.render()<jupyter_output><empty_output><jupyter_text>The taxi is represented in yellow when empty and in green when full.  The passenger is represented in blue and his destination in magenta. **Observations**: There are 500 discrete states since there are 25 taxi positions, five possible passenger locations (including the case when the passenger is in the taxi), and four destination locations.<jupyter_code>print(f"State Space {env.observation_space}")<jupyter_output><empty_output><jupyter_text>The agent controling the *taxi* has six discrete and deterministic possibles actions:  *  0: *move south**  1: *move north**  2: *move east**  3: *move west**  4: *pickup passenger**  5: *drop off passenger*<jupyter_code>print(f"Action Space {env.action_space}")<jupyter_output><empty_output><jupyter_text>Here is a little method to vizualize our taxi's trajectory:<jupyter_code>from IPython.display import clear_output
from time import sleep

def display_trajectory(frames):
    for i, frame in enumerate(frames):
        clear_output(wait=True)
        print(frame['frame'])
        print(f"Timestep: {i + 1}")
        print(f"State: {frame['state']}")
        print(f"Reward: {frame['reward']}")
        sleep(.2)<jupyter_output><empty_output><jupyter_text>Random agent The following code shows how to run an episode with an agent taking random actions.<jupyter_code>frames = [] # for animation

env.reset()
while True:
    # draw a random action from the action space
    action = env.action_space.sample()
    # the step method takes an action as input and returns 4 variables described in the OpenAI section
    state, reward, done, info = env.step(action)
    frames.append({
        'frame': env.render(mode='ansi'),
        'state': state,
        'reward': reward
        })
    #if done is True then the episode is over
    if done == True:
        break
        
display_trajectory(frames)<jupyter_output><empty_output><jupyter_text>Human policy<jupyter_code>env.reset()
env.render()<jupyter_output><empty_output><jupyter_text>We just saw a random policy. Let's now check on your policy.  Using the possible actions, make a little script to take the client to his destination and visualize it with the `display_frames` method.<jupyter_code>frames = [] 
actions = [...]
for a in actions:
    ...

display_trajectory(frames)<jupyter_output><empty_output><jupyter_text>Q-learningLet's now try to solve the TaxiV3 problem using the Q-learing algorithm.![](images/q-learning.png) Fill the `q_learning` method.<jupyter_code>import random
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd 
import seaborn as sns

def print_running_mean(training_rewards, i):
    clear_output(wait=True)
    plt.figure(figsize=(15,3))
    plt.plot(pd.Series(training_rewards[:i]).rolling(100).mean())
    plt.title("Rewards running mean on last 100 episodes")
    plt.show()

    
def q_learning(env, alpha, gamma, epsilon, nb_episodes):
    nb_states = env.observation_space.n
    nb_actions = env.action_space.n
    q_table = np.zeros([nb_states, nb_actions])
    training_rewards = np.zeros(nb_episodes)
    for i in range(nb_episodes):
        ...
        while True:
            ...

            training_rewards[i] += reward
            if done:
                break

        if i % 100 == 0:
            print_running_mean(training_rewards, i)

    return q_table<jupyter_output><empty_output><jupyter_text>Train your q-table.<jupyter_code>nb_episodes = 5000
alpha = 0.1
gamma = 0.99
epsilon = 0.1

q_table = ...<jupyter_output><empty_output><jupyter_text>Let us have a look at the learned policy.<jupyter_code>env.reset()
frames = [] # for animation
timesteps = 0
while True:
    ...
        
display_trajectory(frames)<jupyter_output><empty_output><jupyter_text>Frozen lakeNow, try your algoithm on frozen lake.<jupyter_code>import gym
env = gym.make("FrozenLake8x8-v0")
env.render()
nb_episodes = 20000
alpha = 0.1
gamma = 0.99
epsilon = 0.1

q_table = ...<jupyter_output><empty_output><jupyter_text>Your agent has probably not reached a satisfying policy.  This is due to the lack of exploration induced by the $\epsilon$-greedy policy.Fixing $\epsilon$ to 0.1 is not sufficient in Frozen Lake. The agent does not explore enougth and thus never receives a positive feedback.  A simple solution would be to use a decaying epsilon from 1 to 0.1.<jupyter_code>decay_rates = [0.9, 0.99, 0.9995, 0.9999]
eps_min = 0.1
plt.figure(figsize=(10,6))

for decay_rate in decay_rates:
    eps_list = []
    test_eps = 1
    for _ in range(nb_episodes):
        test_eps = max(test_eps * decay_rate, eps_min)
        eps_list.append(test_eps)          
    
    plt.plot(eps_list, label='decay rate: {}'.format(decay_rate))

plt.title('Effect of various decay rates')
plt.legend(loc='best')
plt.xlabel('# of episodes')
plt.ylabel('epsilon')
plt.show()<jupyter_output><empty_output><jupyter_text>Modify your `q_learning` function to decay the learning rate after each episode (don't forget to clip so it does not go below 0.1).Try your function with different values for `epsilon_decay`<jupyter_code>def q_learning(env, alpha, gamma, epsilon_decay, nb_episodes):
    epsilon = 1
    min_epsilon = 0.1
    nb_states = env.observation_space.n
    nb_actions = env.action_space.n
    q_table = np.zeros([nb_states, nb_actions])
    training_rewards = np.zeros(nb_episodes)
    for i in range(nb_episodes):
        ...
        ...
        ...
        if i % 100 == 0:
            print_running_mean(training_rewards, i)
            
    return q_table
nb_episodes = 20000
alpha = 0.1
gamma = 0.99
epsilon_decay = 0.9999

q_table = q_learning(env, alpha, gamma, epsilon_decay, nb_episodes)<jupyter_output><empty_output><jupyter_text>Let us have a look at the learned policy.<jupyter_code>env.reset()
frames = [] # for animation
timesteps = 0
state = env.reset()
while True:
    ...
        
display_trajectory(frames)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09978150036416605, 'fraction_numerical': 0.019664967225054626, 'mean_word_length': 4.026354319180088, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '324368', 'license': 'MIT', 'n_tokens_mistral': 2327, 'n_tokens_neox': 2187, 'n_words': 671, 'path': 'code/reinforcement_learning/Taxi.ipynb', 'repo_name': 'DavidBert/N7-techno-IA', 'size': '6865'}