<jupyter_start><jupyter_text>Q-learning on a discrete mazeThis example demonstrates how to use the discrete Temporal DifferenceReinforcement Learning algorithms (SARSA, Q, Q(lambda)) in a classicalfully observable MDP maze task. The goal point is in the lower right corner.<jupyter_code># prepare plotting
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams

from matplotlib import animation, rc
from IPython.display import HTML
# rcParams['animation.writer'] = 'ffmpeg'

from pybrain.rl.environments.mazes import Maze, MDPMazeTask
from pybrain.rl.learners.valuebased import ActionValueTable
from pybrain.rl.agents import LearningAgent
from pybrain.rl.learners import Q, QLambda, SARSA #@UnusedImport
from pybrain.rl.explorers import BoltzmannExplorer #@UnusedImport
from pybrain.rl.experiments import Experiment
__author__ = '<NAME>, <EMAIL>, <NAME> <EMAIL>'<jupyter_output><empty_output><jupyter_text>The RL Q-learner consists of an agent that incorporates a learning algorithm and an environment with the maze to be solved. The experiment combines the agent and environment.<jupyter_code># create the maze with walls (0 - open square, 1 - blocked)
envmatrix = np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1],
                   [1, 0, 0, 1, 0, 0, 0, 0, 1],
                   [1, 0, 0, 1, 0, 0, 1, 0, 1],
                   [1, 0, 0, 1, 0, 0, 1, 0, 1],
                   [1, 0, 0, 1, 0, 1, 1, 0, 1],
                   [1, 0, 0, 0, 0, 0, 1, 0, 1],
                   [1, 1, 1, 1, 1, 1, 1, 0, 1],
                   [1, 0, 0, 0, 0, 0, 0, 0, 1],
                   [1, 1, 1, 1, 1, 1, 1, 1, 1]])


env = Maze(envmatrix, (7, 7))

# create task
task = MDPMazeTask(env)

# create value table and initialize with ones
table = ActionValueTable(81, 4)
table.initialize(1.)

# create agent with controller and learner - use SARSA(), Q() or QLambda() here
learner = SARSA()

# standard exploration is e-greedy, but a different type can be chosen as well
# learner.explorer = BoltzmannExplorer()

# create agent
agent = LearningAgent(table, learner)

# create experiment
experiment = Experiment(task, agent)


def tb_mean(tbl):
    tb = tbl.params.reshape(81,4).max(1).reshape(9,9)
    return np.mean(tb.flatten())
tb_mean(table)<jupyter_output><empty_output><jupyter_text>The grid shows a dark blue connected channel with a goal square in the lower right.<jupyter_code># Initialize the animation canvas
fig, ax = plt.subplots(figsize= (10,10))
im = plt.imshow(envmatrix, vmin=0, vmax=1.5, animated=True)

def im_init():
    table.initialize(1.0)
    im.set_data(table.params.reshape(81,4).max(1).reshape(9,9))
    return [im]
def im_animate(i):
    # interact with the environment (here in batch mode)
    experiment.doInteractions(100)
    agent.learn()
    agent.reset()

    y = table.params.reshape(81,4).max(1).reshape(9,9)
    # Highlight the goal.  
    y[7,7]  = max(y.flatten())
    # plot
    im.set_array(y)
    # Record how we are doing. 
    value_traj.append(tb_mean(table))
    return [im]
# call the animator. blit=True means only re-draw the parts that have changed.
# See http://louistiao.me/posts/notebooks/embedding-matplotlib-animations-in-jupyter-notebooks/
im_anim = animation.FuncAnimation(fig, im_animate, init_func=im_init,
                               frames=220, interval=60, blit=True)<jupyter_output><empty_output><jupyter_text>We run the experiment, displaying an animated plot of the maze as the value improves. (This takes about 1 minute.)<jupyter_code>value_traj = []
HTML(im_anim.to_html5_video())
# Save the video animation in mp4 format, playable by
# ffplay maze_200frame_1.mp4
#
im_anim.save("maze_200frame_1.mp4")<jupyter_output><empty_output><jupyter_text>Learned value function for the mazeThe value function shows roughly increasing values shown by lighter shades, leading to the goal square.<jupyter_code>y = table.params.reshape(81,4).max(1).reshape(9,9)
y[7,7]  = max(y.flatten())
fig = plt.figure(figsize= (10,10)) 
plt.imshow(y)
fig = plt.figure(figsize= (10,10)) 
plt.plot(value_traj[1:], 'r')
plt.axis('tight')
plt.title("Mean Value Training Trajectory")
plt.xlabel("Episode")
plt.ylabel("Value")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12158632153882688, 'fraction_numerical': 0.037045832343861315, 'mean_word_length': 3.978723404255319, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '380020', 'license': 'MIT', 'n_tokens_mistral': 1597, 'n_tokens_neox': 1439, 'n_words': 436, 'path': 'Rlearning/maze.ipynb', 'repo_name': 'Bhaskers-Blu-Org2/ICSE2019', 'size': '4267'}