<jupyter_start><jupyter_text>TD controlThis notebook provides examples of different temporal difference-based approaches to control.  SetupLoad the required modules as well as some other standard ones that might be needed.<jupyter_code># Setup environment and import standard modules
import sys
import os
import datetime
import numpy as np
import pandas as pd

# Import numerical libraries (skip these for now)
#import sklearn as skl
#from sklearn import linear_model, metrics
#import tensorflow as tf
#from tensorflow import keras
#print('Tensorflow version: {}'.format(tf.__version__))

# Set up plotting (inline)
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

# Import custom modules
#sys.path.append(os.path.join(os.getcwd(), 'modules'))
#import plottools as pt

# Import OpenAI Gym
import gym
print('Open AI Gym version: {}'.format(gym.__version__))

# Autoreload (see https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html)
%load_ext autoreload
%autoreload 2<jupyter_output>Open AI Gym version: 0.15.4<jupyter_text>On-policy TD control (Sarsa) A few observations:- This environment is a bit tricky to use.  Rewards are very scarce, which means that learning is difficulty unless there is a lot of exploration.- Having a high epsilon is probably good for a while in order to encourage exploration. Once a sufficient number of rewards have been encountered, it might make sense to start to taper this towards 0.- Turning off randomness in the environment makes it very difficult to find a reward with a small epsilon- The maximum reward rate when the randomness is turned off tends to plateau at a little less than $1-\epsilon$. When environmental randomness is on, the final reward rate is ~0.45 for $\epsilon = 0.05$.<jupyter_code>import discrete

episodes = 100000
block_size = int(episodes/20)
eps = 0.5
is_slippery = False

# Test classes
env = gym.make('FrozenLake-v0', is_slippery=is_slippery)
TDBase_Agent = discrete.TDControl(env)
Sarsa_Agent = discrete.Sarsa(env)
Q_Agent = discrete.QLearning(env)
ESarsa_Agent = discrete.ExpectedSarsa(env)

TDBase_Agent.info()
Sarsa_Agent.info()
Q_Agent.info()
ESarsa_Agent.info()
# Run Agent

episodes = 20000
block_size = int(episodes/20)
eps = 0.2
is_slippery = True

# Create environment and initialize agent
env = gym.make('FrozenLake-v0', is_slippery=is_slippery)
Agent = discrete.ExpectedSarsa(env, init_mode='random')
steps, rewards = Agent.run(episodes, block_size=block_size, eps=eps)
env.close()
# Plot results

# Calculate reward rate as the running average over a the number of trials specified by update_block
reward_rate = np.convolve(rewards, np.ones(block_size), mode='valid') / block_size
episode_number = np.arange(block_size, episodes+1)

# Plot reward rate vs episodes
plt.plot(episode_number, reward_rate, color='k')
plt.ylabel('Reward rate')
plt.xlabel('Episodes')
plt.xlim(0, episodes)
plt.ylim(0, np.max(reward_rate))
plt.show()

Agent.Q
x = 'abc'
y = ['zeros', 'random', 'optimistic']
x = 'zeros' if x not in y<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09093830334190231, 'fraction_numerical': 0.011889460154241645, 'mean_word_length': 5.623404255319149, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '210744', 'license': 'MIT', 'n_tokens_mistral': 1020, 'n_tokens_neox': 986, 'n_words': 347, 'path': 'notebooks/TDControl.ipynb', 'repo_name': 'alandegenhart/ReinforcementLearning', 'size': '3112'}