<jupyter_start><jupyter_text>Ray RLlib - RecSys: Recommender System© 2019-2020, Anyscale. All Rights Reserved![Anyscale Academy](../../images/AnyscaleAcademyLogo.png)This section explores one approach for using *reinforcement learning* with [Ray RLlib](https://rllib.io/) to build a [*recommender system*](https://en.wikipedia.org/wiki/Recommender_system). For reference, the GitHub public repo for this code is available at  and full source code for this example recommender system is also in the `recsys.py` script. You can run that with default settings to exercise the code:```shellpython recsys.py```To see the available command line options use:```shellpython recsys.py --help```A full run takes about 5-10 minutes on a recent model MacBook Pro laptop.  Load the Training DataThe approach given here for building a recommender system could be applied to just about any dataset where *users* are *rating* a set of *items*. It's bounded in terms of memory requirements as the number of users grows. It could be re-engineered to handle a very large set of items.We'll be using the [Jester collaborative filtering dataset](https://goldberg.berkeley.edu/jester-data/) – which is known for having a high *density*, i.e., where many users have rated many of the items: https://goldberg.berkeley.edu/jester-data/.Jester was an online joke recommender hosted a UC Berkeley which collected data from April 1999 through May 2003. See the discussion of "universal queries" in:> "Eigentaste: A Constant Time Collaborative Filtering Algorithm"  <NAME>, <NAME>, <NAME>, <NAME>  *Information Retrieval*, 4(2), 133-151 (July 2001)  The data is split into three downloadable files, and the first file contains anonymous ratings from 24,983 users who have rated 36 or more jokes.Ratings data is organized as a matrix with dimensions `24983 X 101`  * one row per user  * first column gives the number of jokes rated by that user  * the next 100 columns give the ratings for jokes `01` through `100`  * ratings are real values ranging from `-10.00` to `+10.00`  * the value `"99"` corresponds to `None` = "not rated"  Here's a function to load the dataset:<jupyter_code>import csv

NO_RATING = "99"
MAX_RATING = 10.0

def load_data (data_path):
    rows = []

    with open(data_path, newline="") as csvfile:
        csvreader = csv.reader(csvfile, delimiter=",")

        for row in csvreader:
            conv = [None] * (len(row) - 1)

            for i in range(1, len(row)):
                if row[i] != NO_RATING:
                    rating = float(row[i]) / MAX_RATING
                    conv[i - 1] = rating

            rows.append(conv)

    return rows<jupyter_output><empty_output><jupyter_text>Data DiscoveryLet's load the dataset into a `pandas` dataframe, then take a look at it:<jupyter_code>from pathlib import Path
import os
import pandas as pd

DATA_PATH = Path(os.getcwd()) / Path("jester-data-1.csv")
sample = load_data(DATA_PATH)

df = pd.DataFrame(sample)
df.head()<jupyter_output><empty_output><jupyter_text>Notice above how there are many `NaN` values in the dataset.Those are *missing values*.Before we analyze this data any further, we'll need to [*impute*](https://en.wikipedia.org/wiki/Imputation_(statistics)) those missing values.In other words, substitute each missing value with some appropriate estimate, such as a column mean.The `SimpleImputer` class in `scikit-learn` provides a good way to handle that requirement:.We'll save the imputed sample data in the `X` matrix:<jupyter_code>import numpy as np
from sklearn.impute import SimpleImputer

imp = SimpleImputer(missing_values=np.nan, strategy="mean")
imp.fit(df.values)

X = imp.transform(df.values).T

print(X)<jupyter_output>[[-0.782       0.408       0.09045733 ...  0.09045733  0.09045733
   0.243     ]
 [ 0.879      -0.029       0.02084596 ...  0.02084596  0.02084596
   0.267     ]
 [-0.966       0.636       0.03165743 ...  0.03165743  0.03165743
  -0.398     ]
 ...
 [ 0.07676917  0.034       0.07676917 ...  0.07676917  0.07676917
   0.07676917]
 [-0.00312438 -0.432      -0.00312438 ... -0.00312438 -0.00312438
  -0.00312438]
 [ 0.13547109  0.107       0.13547109 ...  0.13547109  0.13547109
   0.13547109]]<jupyter_text>Next, we'll use *K-means clustering* as an unsupervised learning approach to understand more about how the dataset is structured. In other words, let's re-arrange the dataset into `k` clusters of items, based on how many different users have rated them.Since `k` is a *hyperparameter* we'll need some way to select an appropriate value for it. One approach that's popular for K-means is to evaluate the *inertia* of decreasing error in the clustering models as `k` grows from zero to some limit. *  *  We really don't want to work with more than a few clusters – since many small clusters would lead to less *predictive power* in the _recsys_. Given that there are 100 items, let's measure inertia up to `k=20` clusters.<jupyter_code>from sklearn.cluster import KMeans

distortions = []

for i in range(1, 20):
    km = KMeans(
        n_clusters=i,
        init="random",
        n_init=10,
        max_iter=300,
        tol=1e-04,
        random_state=0
    )

    km.fit(X)
    distortions.append(km.inertia_)<jupyter_output><empty_output><jupyter_text>Let's visualize the curve of inertia, using `matplotlib`:<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt

plt.plot(range(1, 20), distortions, marker="o")
plt.xlabel("Number of clusters")
plt.ylabel("Distortion")
plt.show()<jupyter_output><empty_output><jupyter_text>This kind of cluster analysis has stochastic aspects, so results may differ on different runs. Generally, the plot shows a "knee" in the curve near `k=7` as the decrease in error begins to level out. That's a reasonable number of clusters, such that each cluster will tend to have ~14% of the items. That choice has an inherent trade-off:  * too few clusters → poor predictions (less precision)  * too many clusters → poor predictive power (less recall)Now we can run K-means in `scikit-learn` with that hyperparameter `k=7` to get the clusters that we'll use in our RL environment:<jupyter_code>K_CLUSTERS = 7

km = KMeans(n_clusters=K_CLUSTERS)
km.fit(X)

y_km = km.fit_predict(X)<jupyter_output><empty_output><jupyter_text>From this analysis we'll need the *labels* for each item, i.e., the number of the cluster to which each item gets assigned:<jupyter_code>from collections import defaultdict

labels = km.labels_
clusters = defaultdict(set)

for i in range(len(labels)):
    clusters[labels[i]].add(i)

CLUSTERS = dict(clusters)<jupyter_output><empty_output><jupyter_text>We'll also use the cluster *centers*, which provide the vector positions (the per-dimension "means") for each cluster.<jupyter_code>centers = km.cluster_centers_
CENTERS = centers.tolist()<jupyter_output><empty_output><jupyter_text>BTW, let's take a look at the K clusters from this analysis…<jupyter_code>plt.scatter(
    X[y_km == 0, 0], X[y_km == 0, 1],
    s=50, c="lightgreen",
    marker="s", edgecolor="black",
    label="cluster 0"
)

plt.scatter(
    X[y_km == 1, 0], X[y_km == 1, 1],
    s=50, c="orange",
    marker="o", edgecolor="black",
    label="cluster 1"
)

plt.scatter(
    X[y_km == 2, 0], X[y_km == 2, 1],
    s=50, c="lightblue",
    marker="v", edgecolor="black",
    label="cluster 2"
)

plt.scatter(
    X[y_km == 3, 0], X[y_km == 3, 1],
    s=50, c="blue",
    marker="^", edgecolor="black",
    label="cluster 3"
)

plt.scatter(
    X[y_km == 4, 0], X[y_km == 4, 1],
    s=50, c="yellow",
    marker="<", edgecolor="black",
    label="cluster 4"
)

plt.scatter(
    X[y_km == 5, 0], X[y_km == 5, 1],
    s=50, c="purple",
    marker=">", edgecolor="black",
    label="cluster 5"
)

plt.scatter(
    X[y_km == 6, 0], X[y_km == 6, 1],
    s=50, c="brown",
    marker="X", edgecolor="black",
    label="cluster 6"
)

# plot the centroids
plt.scatter(
    km.cluster_centers_[:, 0], km.cluster_centers_[:, 1],
    s=250, marker="*",
    c="red", edgecolor="black",
    label="centers"
)

plt.legend(scatterpoints=1)
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>Not bad, based on the centers these clusters show some separation – at least when we plot in 2 dimensions.  Thesis for Building an RL Recommender SystemWe're using a thesis here that people tend to like similar "kinds" of items, in this case jokes. While we could go to extra effort to build a classifier model for jokes, it's much simpler to measure ratings for them across a large sample of users' likes. In a classic [*collaborative filtering*](https://en.wikipedia.org/wiki/Collaborative_filtering) approach, we could base our recommendations on other users who have similar patterns of rating items. On the one hand, that may be expensive to scale as the number of users grows. Instead, an [*item-based collaborative filter*](https://en.wikipedia.org/wiki/Item-item_collaborative_filtering) approach which was invented by [Amazon](https://ieeexplore.ieee.org/document/1167344) for "people who bought x also bought y". This scales to massive datasets by clustering the items based on user ratings.On the other hand, people change their interests over time but a class CF model doesn't account for that.  For a thorough study of these effects, see ["From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews"](https://arxiv.org/abs/1303.4402) by <NAME> and <NAME> (2013-03-18).Let's use reinforcement learning to build an item-based recommender system that can "evolve" its recommendations for each individual over time. We'll leverage *exploration* (gather info that leads toward better recommendations in the future) versus *exploitation* (use the best decision given the info known so far). We have clustered the items (jokes), so now we can create a custom environment where:  * *actions* taken by an agent – recommend items from specific clusters  * *observation space* – a vector distance from the user's rating history to the cluster centers  * *rewards* – user ratings for recommended items, with minor penalties for unrated items  Each *episode* for an agent will simulate one user's ratings vs. which jokes get recommdened based on our *policy* which is trained using RL. One other point: a *dense sub-matrix* – in which almost all users have rated the jokes – is defined by the following columns:<jupyter_code>DENSE_SUBMATRIX = [ 5, 7, 8, 13, 15, 16, 17, 18, 19, 20 ]<jupyter_output><empty_output><jupyter_text>One well-known issue encountered in production use of recommenders is the [*cold start*](https://en.wikipedia.org/wiki/Cold_start_(recommender_systems)) problem. When we first begin to make recommendations for a given user, we have zero information about them. So it's not possible to provide personalized recommendations. However, we have the dense sub-matrix above, which almost all users have rated. We can sample from this subset – as "warm" items – to provide to each user at the start of an *episode*. That will give the recommender some initial information to leverage as a starting point.  A Custom Gym EnvironmentNext we'll build a custom environment for training a policy with RLlib.  We'll use OpenAI [Gym](https://gym.openai.com/) as the basis for it. For details about how to build custom Gym environments for working with RLlib, see ["Anatomy of a custom environment for RLlib"](https://medium.com/distributed-computing-with-ray/anatomy-of-a-custom-environment-for-rllib-327157f269e5) and its accompanying source code in the  Git repo. Some details are also covered in the _Explore RLlib_ lesson, [Custom Environments and Reward Shaping](../explore-rllib/03-Custom-Environments-Reward-Shaping.ipynb) First, we'll define a few globals… `MAX_STEPS` is the length of an episode, based on how many jokes can be recommended before repeating. Reward values are either the user's rating for the recommended item, or minor penalties for recommending either a "depleted" cluster or an item which the user never rated.<jupyter_code>ROW_LENGTH = 100
MAX_STEPS = ROW_LENGTH - len(DENSE_SUBMATRIX)

REWARD_DEPLETED = -0.1	# item recommended from a depleted cluster (no-op)
REWARD_UNRATED = -0.05	# item was never rated by this user<jupyter_output><empty_output><jupyter_text>Now for the Gym environment. This is one class and should be defined within one cell – albeit this is a long cell to scroll through…<jupyter_code>import gym
from gym import spaces
import random

class JokeRec (gym.Env):
    def __init__ (self, config):
        # NB: here we're passing strings via config; RLlib use of JSON
        # parser was throwing exceptions due to config values
        self.dense = eval(config["dense"])
        self.centers = eval(config["centers"])
        self.clusters = eval(config["clusters"])

        lo = np.array([np.float64(-1.0)] * K_CLUSTERS)
        hi = np.array([np.float64(1.0)] * K_CLUSTERS)

        self.observation_space = spaces.Box(lo, hi, shape=(K_CLUSTERS,), dtype=np.float64)
        self.action_space = spaces.Discrete(K_CLUSTERS)

        # load the dataset
        self.dataset = load_data(config["dataset"])


    def _warm_start (self):
        """
        attempt a warm start, sampling half the dense submatrix of most-rated items
        """
        sample_size = round(len(self.dense) / 2.0)

        for action, items in self.clusters.items():
            for item in random.sample(self.dense, sample_size):
                if item in items:
                    state, reward, done, info = self.step(action)


    def _get_state (self):
        """
        calculate root-mean-square (i.e., normalized vector distance) for the agent's current 
        "distance" measure from each cluster center as the observation
        """
        n = float(len(self.used))

        if n > 0.0:
            state = [ np.sqrt(x / n) for x in self.coords ]
        else:
            state = self.coords

        return state


    def reset (self):
        """
        reset the item recommendation history, select a new user to simulate from among the 
        dataset rows, then run an initial 'warm-start' sequence of steps before handing step
        control back to the agent
        """
        self.count = 0
        self.used = []
        self.depleted = 0
        self.coords = [np.float64(0.0)] * K_CLUSTERS

        # select a random user to simulate
        self.data_row = random.choice(self.dataset)

        # attempt a warm start
        self._warm_start()

        return self._get_state()


    def step (self, action):
        """
        recommend one item, which may result in a no-op -- 
        in production, skip any repeated items per user
        """
        assert action in self.action_space, action
        assert_info = "c[item] {}, rating {}, scaled_diff {}"

        # enumerate items from the cluster selected by the action that haven't been recommended
        # previously to the simulated user
        items = set(self.clusters[action]).difference(set(self.used))

        if len(items) < 1:
            # oops! items from the selected cluster have been depleted, i.e. all have been 
            # recommended previously to the simulated user; hopefully the agent will learn
            # to switch to exploring among the other clusters
            self.depleted += 1
            item = None
            reward = REWARD_DEPLETED
        else:
            # chose an item at random from the selected cluster
            item = random.choice(list(items))
            rating = self.data_row[item]

            if not rating:
                # no-op! this action resulted in an unrated item
                reward = REWARD_UNRATED

            else:
                # success! this action resulted in an item rated by the simulated user
                reward = rating
                self.used.append(item)

                # update the coords history: agent observes its distance to each cluster "evolve"
                for i in range(len(self.coords)):
                    c = self.centers[i]
                    scaled_diff = abs(c[item] - rating) / 2.0
                    self.coords[i] += scaled_diff ** 2.0

        self.count += 1
        done = self.count >= MAX_STEPS
        info = { "item": item, "count": self.count, "depleted": self.depleted }

        return self._get_state(), reward, done, info


    def render (self, mode="human"):
        last_used = self.used[-10:]
        last_used.reverse()
        print(">> used:", last_used)
        print(">> dist:", [round(x, 2) for x in self._get_state()])
        print(">> depl:", self.depleted)<jupyter_output><empty_output><jupyter_text>The `JokeRec.step()` method contains the heart of the simulation logic here. It determines the item to be recommend based on the input `action`, then determines the `reward` and the agent's updated vector distance to the cluster centers as the `observation space`.Overall, this approach is relatively well-behaved and bounded for its memory use as the number of items grows. We're keeping most of the data in memory, but could readily use a distributed key/value store for selecting items, maintaining user history, etc.  ConfigurationNow we'll use the results of K-means clustering on the data sample to prepare a configuration for our custom environment.We'll use [*proximal policy optimization*](https://docs.ray.io/en/latest/rllib-algorithms.html?highlight=ppoproximal-policy-optimization-ppo) (PPO) for training a policy in RLlib, taking the default PPO configuration as the foundation here.<jupyter_code>import ray
import ray.rllib.agents.ppo as ppo

CONFIG = ppo.DEFAULT_CONFIG.copy()

CONFIG["log_level"] = "WARN"
CONFIG["num_workers"] = 3    # set to `0` for debug

CONFIG["env_config"] = {
    "dataset": DATA_PATH,
    "dense": str(DENSE_SUBMATRIX),
    "clusters": repr(CLUSTERS),
    "centers": repr(CENTERS),
}<jupyter_output>WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term<jupyter_text>Now let's create an instance of our custom Gym environment and call the `reset()` method, which will initialize an episode (simulating one user's ratings), run a "warm start", then return the initial observation:<jupyter_code>env = JokeRec(CONFIG["env_config"])
env.reset()<jupyter_output><empty_output><jupyter_text>We can randomly select an action, i.e., the label of an item cluster to recommend:<jupyter_code>action = env.action_space.sample()
print("action:", action)<jupyter_output>action: 1<jupyter_text>Let's use that action to take one step:<jupyter_code>state, reward, done, info = env.step(action)
print("obs:", state)
print("reward:", reward)<jupyter_output>obs: [0.382587135262065, 0.26891405414838127, 0.3588139074637544, 0.32777714287592735, 0.2773204181940893, 0.5425337547102485, 0.30016170467946984]
reward: -0.359<jupyter_text>We can call the `render()` method to describe more about the agent's state within its environment:<jupyter_code>env.render()<jupyter_output>>> used: [15, 17, 14, 55, 7, 10]
>> dist: [0.38, 0.27, 0.36, 0.33, 0.28, 0.54, 0.3]
>> depl: 0<jupyter_text>Notice how there are already some other "used" items, due to the warm start.Next, we'll define a function that runs through one entire episode:<jupyter_code>def run_one_episode (env, naive=False, verbose=False):
    """
    step through one episode, using either a naive strategy or random actions
    """
    env.reset()
    sum_reward = 0

    action = None
    avoid_actions = set([])
    depleted = 0

    for i in range(MAX_STEPS):
        if not naive or not action:
            action = env.action_space.sample()

        state, reward, done, info = env.step(action)

        if verbose:
            print("action:", action)
            print("obs:", i, state, reward, done, info)

        # naive strategy: select items from the nearest non-depleted cluster
        if naive:
            if info["depleted"] > depleted:
                depleted = info["depleted"]
                avoid_actions.add(action)

            obs = []

            for a in range(len(state)):
                if a not in avoid_actions:
                    dist = round(state[a], 2)
                    obs.append([dist, a])

            action = min(obs)[1]

        sum_reward += reward

        if done:
            if verbose:
                print("DONE @ step {}".format(i))

            break

    if verbose:
        print("CUMULATIVE REWARD: ", round(sum_reward, 3))

    return sum_reward<jupyter_output><empty_output><jupyter_text>Next, another function which uses `run_one_episode()` to measure the baseline performance of a "naïve" strategy, i.e., without use of reinforcement learning to train a policy:<jupyter_code>from tqdm import tqdm

def measure_baseline (env, n_iter=1, naive=False, verbose=False):
    history = []

    for episode in tqdm(range(n_iter), ascii=True, desc="measure baseline"):
        sum_reward = run_one_episode(env, naive=naive, verbose=verbose)
        history.append(sum_reward)

    baseline = sum(history) / len(history)
    return baseline<jupyter_output><empty_output><jupyter_text>Now let's use this to measure how well our recommender system runs without leveraging RLlib:<jupyter_code>baseline = measure_baseline(env, n_iter=1000, naive=True)
print("BASELINE CUMULATIVE REWARD", round(baseline, 3))<jupyter_output>measure baseline: 100%|##########| 1000/1000 [00:05<00:00, 188.55it/s]<jupyter_text>Running Ray and RLlibAt this point we're ready to train a policy with RLlib. First we'll initialize the directory in which to save *checkpoints*, and the directory in which to log results…<jupyter_code>import os
import shutil

CHECKPOINT_ROOT = "tmp/rec"
shutil.rmtree(CHECKPOINT_ROOT, ignore_errors=True, onerror=None)

ray_results = "{}/ray_results/".format(os.getenv("HOME"))
shutil.rmtree(ray_results, ignore_errors=True, onerror=None)<jupyter_output><empty_output><jupyter_text>Now start Ray, register our custom environment, and create an agent. BTW, if you see lots of "deprecation" warnings from [Tensorflow](https://www.tensorflow.org/) just ignore those…<jupyter_code>info = ray.init(ignore_reinit_error=True)<jupyter_output>2020-10-28 16:44:23,055	INFO services.py:1166 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8265[39m[22m<jupyter_text>Register our environment so we can reference it by name. Then create a `PPOTrainer`.<jupyter_code>from ray.tune.registry import register_env

env_key = "JokeRec-v0"
register_env(env_key, lambda config_env: JokeRec(config_env))
AGENT = ppo.PPOTrainer(CONFIG, env=env_key)<jupyter_output>2020-10-28 16:44:23,661	INFO trainer.py:591 -- Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution
2020-10-28 16:44:23,662	INFO trainer.py:618 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
[2m[36m(pid=91945)[0m WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=91945)[0m Instructions for updating:
[2m[36m(pid=91945)[0m non-resource variables are not supported in the long term
[2m[36m(pid=91941)[0m WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[2m[36m(pid=91941)[0m Instructions for updating:
[2m[3[...]<jupyter_text>Let's train a policy using the PPO optimizer in RLlib. As the following code steps through each training iteration, watch how the measured improvements in the min, mean, and max rewards per episode.<jupyter_code>TRAIN_ITER = 20

df = pd.DataFrame(columns=[ "min_reward", "avg_reward", "max_reward", "steps", "checkpoint"])
status = "reward {:6.2f} {:6.2f} {:6.2f}  len {:4.2f}  saved {}"

for i in range(TRAIN_ITER):
    result = AGENT.train()
    checkpoint_file = AGENT.save(CHECKPOINT_ROOT)

    row = [
        result["episode_reward_min"],
        result["episode_reward_mean"],
        result["episode_reward_max"],
        result["episode_len_mean"],
        checkpoint_file,
        ]

    df.loc[len(df)] = row
    print(status.format(*row))
    
BEST_CHECKPOINT = checkpoint_file<jupyter_output>WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/tf_policy.py:874: Variable.load (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Variable.assign which has equivalent behavior in 2.X.<jupyter_text>Rollout to Emulate a Use Case DeploymentNow let's define a function to run a *rollout* using a checkpointed policy.Each rollout iteration will emuluate a deployed use of our recommender system for one user, and we'll measure the average rewards across many iterations:<jupyter_code>def run_rollout (agent, env, n_iter=1, verbose=False):
    """
    iterate through `n_iter` episodes in a rollout to emulate deployment in a production use case
    """
    for episode in range(n_iter):
        state = env.reset()
        sum_reward = 0

        for step in range(MAX_STEPS):
            try:
                action = agent.compute_action(state)
                state, reward, done, info = env.step(action)
                sum_reward += reward

                if verbose:
                    print("reward {:6.3f}  sum {:6.3f}".format(reward, sum_reward))
                    env.render()
            except Exception:
                traceback.print_exc()

            if done:
                # report at the end of each episode
                print("CUMULATIVE REWARD:", round(sum_reward, 3), "\n")
                yield sum_reward
                break<jupyter_output><empty_output><jupyter_text>Now we can apply the best trained policy in a rollout:<jupyter_code>AGENT.restore(BEST_CHECKPOINT)
history = []

for episode_reward in run_rollout(AGENT, env, n_iter=500, verbose=False):
    history.append(episode_reward)
    
print("average reward:", round(sum(history) / len(history), 3))<jupyter_output>2020-10-28 16:46:42,107	INFO trainable.py:482 -- Restored on 192.168.1.65 from checkpoint: tmp/rec/checkpoint_20/checkpoint-20
2020-10-28 16:46:42,109	INFO trainable.py:489 -- Current state after restoring: {'_iteration': 20, '_timesteps_total': None, '_time_total': 124.7182719707489, '_episodes_total': 986}<jupyter_text>How does the reported *average reward* from many rollouts (using RLlib to train a policy) compare with the *baseline cumulative reward* above based on a naïve strategy and no learning?  How does it compare with the predicted *mean reward per episode* from training? The baseline reward from a naïve strategy should be much lower (worse user ratings) than the other two measures.These measures are an estimate for how a user would rate their recommended items.Of course, not all users will like the jokes, so there will be some rollouts with negative rewards.Overall we want the average reward to be *positive*, with `MAX_STEPS` as an upper bounds.  Examine the trained policyUse the following code to examine the trained policy that was optimized using PPO:<jupyter_code>policy = AGENT.get_policy()
model = policy.model

print("\n", model.base_model.summary())<jupyter_output>Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
observations (InputLayer)       [(None, 7)]          0                                            
__________________________________________________________________________________________________
fc_1 (Dense)                    (None, 256)          2048        observations[0][0]               
__________________________________________________________________________________________________
fc_value_1 (Dense)              (None, 256)          2048        observations[0][0]               
__________________________________________________________________________________________________
fc_2 (Dense)                    (None, 256)          65792       fc_1[0][0]            [...]<jupyter_text>Evaluate learning with TensorBoardYou also can run [TensorBoard](https://www.tensorflow.org/tensorboard) to visualize the RL training metrics from the log files. The results during training were written to a directory under `$HOME/ray_results`If you are viewing this lesson on the Anyscale hosted platform, use the provided link to open TensorBoard.If you are viewing this lesson on a laptop, open a terminal and run the following command, then open the URL shown in the output. (You can open a terminal using the `+` in the upper left-hand corner of Jupyter Lab.)```shelltensorboard --logdir=~/ray_results```Open the URL printed to view the TensorBoard GUI. ---  Exercise 1Compare use of the other datasets `"jester-data-2.csv"` and `"jester-data-3.csv"` by substituting them during the rollout.How do the mean cumulative reward differ from the metrics in the lesson?  Exercise 2Compare the effect of using a larger `K` value for the number of clusters.Show the difference, if any, by comparing:  * baseline with random actions   * baseline with the naïve strategy  * predicted average reward from training  * stats from the rollout  Discussion Questions  1. In what ways could the "warm start" be improved?  2. How could this code be modified to scale to millions of users?  Or to thousands of items?  Clean upFinally, let's shutdown Ray gracefully:<jupyter_code>ray.shutdown()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1000997672098437, 'fraction_numerical': 0.03292317924842035, 'mean_word_length': 3.659281066005578, 'pattern_counts': {'":': 7, '<': 111, '<?xml version=': 0, '>': 125, 'https://': 15, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '69256', 'license': 'Apache-2.0', 'n_tokens_mistral': 9737, 'n_tokens_neox': 8732, 'n_words': 3178, 'path': 'ray-rllib/recsys/01-Recsys.ipynb', 'repo_name': 'aksakalli/academy', 'size': '30111'}