<jupyter_start><jupyter_text>Ray RLlib - Multi-Armed Bandits - Exercise Solutions© 2019-2020, Anyscale. All Rights Reserved![Anyscale Academy](../../../images/AnyscaleAcademy_Logo_clearbanner_141x100.png) Let's explore a very simple contextual bandit example with 3 arms. We'll run trials using RLlib and [Tune](http://tune.io), Ray's hyperparameter tuning library.<jupyter_code>import gym
from gym.spaces import Discrete, Box
import numpy as np
import pandas as pd
import os, time, random
import ray
from ray.tune.progress_reporter import JupyterNotebookReporter
!../../../tools/start-ray.sh --check --verbose
ray.init(address='auto', ignore_reinit_error=True)<jupyter_output><empty_output><jupyter_text>03: Simple Multi-Armed Bandits - Exercise 1  Exercise 1Change the the `step` method to randomly change the `current_context` on each invocation:```pythondef step(self, action):    result = super().step(action)    self.current_context = random.choice([-1.,1.])    return (np.array([-self.current_context, self.current_context]), reward, True,            {                "regret": 10 - reward            })```Repeat the training and analysis. Does the training behavior change in any appreciable way? Why or why not?See the [solutions notebook](solutions/Multi-Armed-Bandits-Solutions.ipynb) for discussion of this and the following exercises.<jupyter_code>class SimpleContextualBandit2 (gym.Env):
    def __init__ (self, config=None):
        self.action_space = Discrete(3)     # 3 arms
        self.observation_space = Box(low=-1., high=1., shape=(2, ), dtype=np.float64)  # Random (x,y), where x,y from -1 to 1
        self.current_context = None
        self.rewards_for_context = {
            -1.: [-10, 0, 10],
            1.: [10, 0, -10],
        }

    def reset (self):
        self.current_context = random.choice([-1., 1.])
        return np.array([-self.current_context, self.current_context])

    def step (self, action):
        reward = self.rewards_for_context[self.current_context][action]
        self.current_context = random.choice([-1.,1.])
        return (np.array([-self.current_context, self.current_context]), reward, True,
                {
                    "regret": 10 - reward
                })

    def __repr__(self):
        return f'SimpleContextualBandit2(action_space={self.action_space}, observation_space={self.observation_space}, current_context={self.current_context}, rewards per context={self.rewards_for_context})'
    
bandit = SimpleContextualBandit2()
observation = bandit.reset()
f'Initial observation = {observation}, bandit = {repr(bandit)}'
stop = {
    "training_iteration": 200,
    "timesteps_total": 100000,
    "episode_reward_mean": 10.0,
}

config = {
    "env": SimpleContextualBandit2,
}
analysis = ray.tune.run("contrib/LinUCB", config=config, stop=stop, 
    progress_reporter=JupyterNotebookReporter(overwrite=False),  # This is the default, actually.
    verbose=1)
stats = analysis.stats()
secs = stats["timestamp"] - stats["start_time"]
print(f'{secs:7.2f} seconds, {secs/60.0:7.2f} minutes')
df = analysis.dataframe()
df<jupyter_output><empty_output><jupyter_text>It trains just as easily as the original implementation that didn't switch contexts between steps. Is this surprising? Probably not, because the relationship between the reward and the context remains linear, so what LinUCB learns for one context is correct for the second context, too. Also, _Tune_ runs many episodes, so it studies both contexts.  03: Simple Multi-Armed Bandits - Exercise 2Recall the `rewards_for_context` we used:```pythonself.rewards_for_context = {    -1.: [-10, 0, 10],    1.: [10, 0, -10],}```We said that Linear Upper Confidence Bound assumes a linear dependency between the expected reward of an action and its context. It models the representation space using a set of linear predictors.Change the values for the rewards as follows, so they no longer have the same simple linear relationship:```pythonself.rewards_for_context = {    -1.: [-10, 10, 0],    1.: [0, 10, -10],}```Also remove the change made for exercise 1, the line `self.current_context = random.choice([-1.,1.])` in the `step` method.Run the training again and look at the results for the reward mean in TensorBoard. How successful was the training? How smooth is the plot for `episode_reward_mean`? How many steps were taken in the training?<jupyter_code>class SimpleContextualBanditNonlinear (gym.Env):
    def __init__ (self, config=None):
        self.action_space = Discrete(3)     # 3 arms
        self.observation_space = Box(low=-1., high=1., shape=(2, ), dtype=np.float64)  # Random (x,y), where x,y from -1 to 1
        self.current_context = None
        self.rewards_for_context = {   # Changed here:
            -1.: [-10, 10, 0],
            1.: [0, 10, -10],
        }

    def reset (self):
        self.current_context = random.choice([-1., 1.])
        return np.array([-self.current_context, self.current_context])

    def step (self, action):
        reward = self.rewards_for_context[self.current_context][action]
        return (np.array([-self.current_context, self.current_context]), reward, True,
                {
                    "regret": 10 - reward
                })

    def __repr__(self):
        return f'SimpleContextualBanditNonlinear(action_space={self.action_space}, observation_space={self.observation_space}, current_context={self.current_context}, rewards per context={self.rewards_for_context})'
bandit = SimpleContextualBanditNonlinear()
observation = bandit.reset()
f'Initial observation = {observation}, bandit = {repr(bandit)}'
print(f'current_context = {bandit.current_context}')
for i in range(10):
    action = bandit.action_space.sample()
    observation, reward, done, info = bandit.step(action)
    print(f'observation = {observation}, action = {action}, reward = {reward:4d}, done = {str(done):5s}, info = {info}')
# `stop` defined above is unchanged.

config = {
    "env": SimpleContextualBanditNonlinear,
}
analysis = ray.tune.run("contrib/LinUCB", config=config, stop=stop, 
    progress_reporter=JupyterNotebookReporter(overwrite=False),  # This is the default, actually.
    verbose=1)
stats = analysis.stats()
secs = stats["timestamp"] - stats["start_time"]
print(f'{secs:7.2f} seconds, {secs/60.0:7.2f} minutes')
df = analysis.dataframe()
df<jupyter_output><empty_output><jupyter_text>It ran the maximum of 20,000 steps and the best it does falls between 4.8 to 5.2 (for different runs), not 10.0. the `episode_reward_mean` is chaotic:![Nonlinear model with LinUCB](../../../images/rllib/TensorBoard2.png).Because LinUCB expcts a linear relationship between the context and each reward, it's not surprising that it fails to converge to the desired reward mean.  03: Simple Multi-Armed Bandits - Exercise 3We briefly discussed another algorithm for selecting the next action, _Thompson Sampling_, in the [previous lesson](../02-Exploration-vs-Exploitation-Strategies.ipynb). Repeat exercises 1 and 2 using linear version, called _Linear Thompson Sampling_ ([RLlib documentation](https://docs.ray.io/en/latest/rllib-algorithms.html?highlight=greedylinear-thompson-sampling-contrib-lints)). To make this change, look at this code we used above:```pythonanalysis = tune.run("contrib/LinUCB", config=config, stop=stop,                     progress_reporter=JupyterNotebookReporter(overwrite=False),   This is the default, actually.                    verbose=2)   Change to 0 or 1 to reduce the output.```Change `contrib/LinUCB` to `contrib/LinTS`.<jupyter_code>bandit = SimpleContextualBandit2()
observation = bandit.reset()

# `stop` defined above is unchanged.

config = {
    "env": SimpleContextualBandit2,
}

analysis = ray.tune.run("contrib/LinTS", config=config, stop=stop, 
    progress_reporter=JupyterNotebookReporter(overwrite=False),  # This is the default, actually.
    verbose=1)

stats = analysis.stats()
secs = stats["timestamp"] - stats["start_time"]
print(f'{secs:7.2f} seconds, {secs/60.0:7.2f} minutes')
df = analysis.dataframe()
df<jupyter_output><empty_output><jupyter_text>As before, the training only takes 200 steps and converge to the desired reward mean of `10.0`.Now let's try the nonlinear bandit:<jupyter_code>bandit = SimpleContextualBanditNonlinear()
observation = bandit.reset()

# `stop` defined above is unchanged.

config = {
    "env": SimpleContextualBanditNonlinear,
}

start_time = time.time()

analysis = ray.tune.run("contrib/LinTS", config=config, stop=stop, 
                        progress_reporter=JupyterNotebookReporter(overwrite=False),  # This is the default, actually.
                        verbose=1)

print("The trials took", time.time() - start_time, "seconds\n")
df = analysis.dataframe()
df<jupyter_output><empty_output><jupyter_text>This run with Thompson sampling yields similar results with the reward mean between 4.5 and 5.0, with somewhat chaotic results over the 20000 steps, if you look at the `episode_reward_mean` graph in TensorBoard.  04: Linear Upper Confidence Bound - Exercise 1Change the `training_iterations` from 20 to 40. Does the characteristic behavior of cumulative regret change at higher steps?<jupyter_code>from ray.rllib.contrib.bandits.agents.lin_ucb import UCB_CONFIG
from ray.rllib.contrib.bandits.envs import ParametricItemRecoEnv
UCB_CONFIG["env"] = ParametricItemRecoEnv

# Actual training_iterations will be 40 * timesteps_per_iteration (100 by default) = 4,000
training_iterations = 40

print("Running training for %s time steps" % training_iterations)
analysis = ray.tune.run(
    "contrib/LinUCB",
    config=UCB_CONFIG,
    stop={"training_iteration": training_iterations},
    num_samples=5,
    checkpoint_at_end=False,
    verbose=1
)

stats = analysis.stats()
secs = stats["timestamp"] - stats["start_time"]
print(f'{secs:7.2f} seconds, {secs/60.0:7.2f} minutes')
frame = pd.DataFrame()

for key, df in analysis.trial_dataframes.items():
    frame = frame.append(df, ignore_index=True)

df = frame.groupby("info/num_steps_trained")[
    "info/learner/default_policy/cumulative_regret"].aggregate(["mean", "max", "min", "std"])
df
import sys
sys.path.append('../../..') # ... and line_plot functions from "util"
from util.line_plots import plot_line, plot_line_with_stddev, plot_line_with_min_max

sys.path.append("..")          # So we can load the bokeh_util from the parent directory...
from bokeh_util import plot_cumulative_regret, plot_wheel_bandit_model_weights
# The next two lines prevent Bokeh from opening the graph in a new window.
import bokeh
bokeh.io.reset_output()
bokeh.io.output_notebook()
plot_cumulative_regret(df)<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/LinUCB-cumulative-regret2.png))The slope appears to stop flattening, suggesting that the previous number of steps, 2000, was sufficient to get the optimal behavior. Beyond that, regret continues to accumulate, but it's linear in the number of steps, neither getting better or worse.    05: Linear Thompson Sampling - Exercise 1Experiment with different $\delta$ values, for example 0.7 and 0.9. What do the cumulative regret and weights graphs look like? You can set the $\delta$ value like this:```pythonTS_CONFIG["delta"] = 0.7```<jupyter_code>from ray.rllib.contrib.bandits.agents import LinTSTrainer
from ray.rllib.contrib.bandits.agents.lin_ts import TS_CONFIG
from ray.rllib.contrib.bandits.envs import WheelBanditEnv
TS_CONFIG["env"] = WheelBanditEnv

training_iterations = 20
print("Running training for %s time steps" % training_iterations)
def run_ts(delta):
    TS_CONFIG["delta"] = delta

    start_time = time.time()

    analysis = ray.tune.run(
        LinTSTrainer,
        config=TS_CONFIG,
        stop={"training_iteration": training_iterations},
        num_samples=2,
        checkpoint_at_end=True,
        verbose=1)

    print("The trials took", time.time() - start_time, "seconds\n")

    df = pd.DataFrame()

    for key, df_trial in analysis.trial_dataframes.items():
        df = df.append(df_trial, ignore_index=True)

    return df, analysis
def process_df(df, analysis):
    ts_regrets = df \
        .groupby("info/num_steps_trained")["info/learner/default_policy/cumulative_regret"] \
        .aggregate(["mean", "max", "min", "std"])
    
    trial = analysis.trials[0]
    trainer = LinTSTrainer(config=TS_CONFIG)
    trainer.restore(trial.checkpoint.value)
    
    model = trainer.get_policy().model
    means = [model.arms[i].theta.numpy() for i in range(5)]
    covs = [model.arms[i].covariance.numpy() for i in range(5)]

    return ts_regrets, model, means, covs
delta = 0.7
ts_df7, analysis7 = run_ts(delta)
ts_regrets7, model7, means7, covs7 = process_df(ts_df7, analysis7)
plot_cumulative_regret(ts_regrets7)<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/LinTS-Cumulative-Regret-07.png))The cumulative regret values are much higher than for $\delta = 0.5$ in the lesson, and the standard deviation is ... well crazy. We mentioned in the lesson that the problem becomes harder for higher $\delta$, which fits this result.<jupyter_code>plot_wheel_bandit_model_weights(means7, covs7)<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/LinTS-Weight-Distribution-of-Arms-07.png))Compare to the separation of the clusters compared to $\delta = 0.5$:![image](../../../images/rllib/LinTS-Weight-Distribution-of-Arms-05.png)<jupyter_code>delta = 0.9
ts_df9, analysis9 = run_ts(delta)
ts_regrets9, model9, means9, covs9 = process_df(ts_df9, analysis9)
plot_cumulative_regret(ts_regrets9)<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/LinTS-Cumulative-Regret-09.png))Qualitatively the same as for $\delta = 0.7$, but the size of the cumulative regret values are even higher.<jupyter_code>plot_wheel_bandit_model_weights(means9, covs9)<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/LinTS-Weight-Distribution-of-Arms-09.png))  06 Market Example - Exercise 1Try using a `LinUCBTrainer`-based trainer. How does the annualized return compare?<jupyter_code># Some properties we'll need:
DEFAULT_MAX_INFLATION = 100.0
DEFAULT_TICKERS = ["sp500", "t.bill", "t.bond", "corp"]
DEFAULT_DATA_FILE = os.path.abspath(os.path.curdir) + '/../market.tsv'  # full path

def load_market_data (file_name):
    with open(file_name, "r") as f:
        return pd.read_table(f)
df = load_market_data(DEFAULT_DATA_FILE)
df
n_years = len(df)
from ray.rllib.agents.trainer import with_base_config, with_common_config
from ray.rllib.contrib.bandits.agents.lin_ucb import UCB_CONFIG
from ray.rllib.contrib.bandits.agents.lin_ucb import LinUCBTrainer
import ray
class MarketBandit (gym.Env):

    def __init__ (self, config={}):
        self.max_inflation = config.get('max-inflation', DEFAULT_MAX_INFLATION)
        self.tickers = config.get('tickers', DEFAULT_TICKERS)
        self.data_file = config.get('data-file', DEFAULT_DATA_FILE)
        print(f"MarketBandit: max_inflation: {self.max_inflation}, tickers: {self.tickers}, data file: {self.data_file} (config: {config})")

        self.action_space = Discrete(4)
        self.observation_space = Box(
            low  = -self.max_inflation,
            high =  self.max_inflation,
            shape=(1, )
        )
        self.df = load_market_data(self.data_file)
        self.cur_context = None


    def reset (self):
        self.year = self.df["year"].min()
        self.cur_context = self.df.loc[self.df["year"] == self.year]["inflation"][0]
        self.done = False
        self.info = {}

        return [self.cur_context]


    def step (self, action):
        if self.done:
            reward = 0.
            regret = 0.
        else:
            row = self.df.loc[self.df["year"] == self.year]

            # calculate reward
            ticker = self.tickers[action]
            reward = float(row[ticker])

            # calculate regret
            max_reward = max(map(lambda t: float(row[t]), self.tickers))
            regret = round(max_reward - reward)

            # update the context
            self.cur_context = float(row["inflation"])

            # increment the year
            self.year += 1

            if self.year >= self.df["year"].max():
                self.done = True

        context = [self.cur_context]
        #context = self.observation_space.sample()

        self.info = {
            "regret": regret,
            "year": self.year
        }

        return [context, reward, self.done, self.info]


    def seed (self, seed=None):
        """Sets the seed for this env's random number generator(s).
        Note:
            Some environments use multiple pseudorandom number generators.
            We want to capture all such seeds used in order to ensure that
            there aren't accidental correlations between multiple generators.
        Returns:
            list<bigint>: Returns the list of seeds used in this env's random
              number generators. The first value in the list should be the
              "main" seed, or the value which a reproducer should pass to
              'seed'. Often, the main seed equals the provided 'seed', but
              this won't be true if seed=None, for example.
        """
        self.np_random, seed = seeding.np_random(seed)
        return [seed]
market_config = with_base_config(UCB_CONFIG, {
    "env":           MarketBandit,
    'max-inflation': DEFAULT_MAX_INFLATION,
    'tickers':       DEFAULT_TICKERS,
    'data-file':     DEFAULT_DATA_FILE
})

stop = {
    "training_iteration": 100
}
MarketLinUCBTrainer = LinUCBTrainer.with_updates(
    name="MarketLinUCBTrainer",
    default_config=market_config,      # Will be merged with Trainer.COMMON_CONFIG (rllib/agent/trainer.py)
    #default_policy=[somePolicyClass]  # If we had a policy...
)
analysis = ray.tune.run(
    MarketLinUCBTrainer,
    config=market_config,
    stop=stop,
    num_samples=3,    
    checkpoint_at_end=True,
    verbose=1
)
stats = analysis.stats()
secs = stats["timestamp"] - stats["start_time"]
print(f'{secs:7.2f} seconds, {secs/60.0:7.2f} minutes')
df_ts = pd.DataFrame()

for key, df_trial in analysis.trial_dataframes.items():
    df_ts = df_ts.append(df_trial, ignore_index=True)
    
df_ts.head()
rewards = df_ts \
    .groupby("info/num_steps_trained")["episode_reward_mean"] \
    .aggregate(["mean", "max", "min", "std"])

rewards
regrets = df_ts \
    .groupby("info/num_steps_trained")["info/learner/default_policy/cumulative_regret"] \
    .aggregate(["mean", "max", "min", "std"])

regrets<jupyter_output><empty_output><jupyter_text>The results for _LinTS_ were ~570 for reward mean and the regret stayed under 10000. So, training with _LinUCB_ isn't as successful.<jupyter_code>plot_line_with_stddev(rewards, x_col='info/num_steps_trained', y_col='mean', stddev_col='std', 
                      title='Rewards vs. Steps', x_axis_label='step', y_axis_label='reward')<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/Market-Bandit-Rewards-vs-Steps-LinUCB.png))<jupyter_code>plot_cumulative_regret(regrets)<jupyter_output><empty_output><jupyter_text>([image](../../../images/rllib/Market-Bandit-Cumulative-Regret-LinUCB.png))What's the annualized return?<jupyter_code>print("{:5.2f}% optimized return annualized".format(max(rewards["mean"]) / n_years))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12093239131555876, 'fraction_numerical': 0.016399360528080036, 'mean_word_length': 3.8528528528528527, 'pattern_counts': {'":': 16, '<': 58, '<?xml version=': 0, '>': 59, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '140265', 'license': 'Apache-2.0', 'n_tokens_mistral': 6433, 'n_tokens_neox': 6093, 'n_words': 1550, 'path': 'ray-rllib/multi-armed-bandits/solutions/Multi-Armed-Bandits-Solutions.ipynb', 'repo_name': 'mathematicsofpaul/academy', 'size': '19391'}