<jupyter_start><jupyter_text>Ray RLlib Tutorial - Exercise Solutions© 2019-2020, Anyscale. All Rights Reserved![Anyscale Academy](../../images/AnyscaleAcademy_Logo_clearbanner_141x100.png)This notebook contains the solutions for all the exercises in the RLlib tutorial.First, we have to setup everything needed from the other notebooks.<jupyter_code>import gym
import numpy as np
import pandas as pd
import json, sys, os<jupyter_output><empty_output><jupyter_text>01 Introduction to Reinforcement Learning Exercise 1Finish implementing the `rollout_policy` function below, which should take an environment *and* a policy. Recall that the *policy* is a function that takes in a *state* and returns an *action*. The main difference is that instead of choosing a **random action**, like we just did (with poor results), the action should be chosen **with the policy** (as a function of the state).<jupyter_code>env = gym.make('CartPole-v1')
print('Created env:', env)
def rollout_policy(env, policy):
    state = env.reset()
    
    done = False
    cumulative_reward = 0

    # Keep looping as long as the simulation has not finished.
    while not done:
        # Choose a random action (either 0 or 1).
        action = policy(state)
        
        # Take the action in the environment.
        state, reward, done, _ = env.step(action)
        
        # Update the cumulative reward.
        cumulative_reward += reward
        
    # Return the cumulative reward.
    return cumulative_reward

def sample_policy1(state):
    return 0 if state[0] < 0 else 1

def sample_policy2(state):
    return 1 if state[0] < 0 else 0

reward1 = np.mean([rollout_policy(env, sample_policy1) for _ in range(100)])
reward2 = np.mean([rollout_policy(env, sample_policy2) for _ in range(100)])

print('The first sample policy got an average reward of {}.'.format(reward1))
print('The second sample policy got an average reward of {}.'.format(reward2))

assert 5 < reward1 < 15, ('Make sure that rollout_policy computes the action '
                          'by applying the policy to the state.')
assert 25 < reward2 < 35, ('Make sure that rollout_policy computes the action '
                           'by applying the policy to the state.')<jupyter_output><empty_output><jupyter_text>Exercise 2The current network and training configuration are too large and heavy-duty for a simple problem like `CartPole`. Modify the configuration to use a smaller network and to speed up the optimization of the surrogate objective. (Fewer SGD iterations and a larger batch size should help.)<jupyter_code>import ray
from ray.rllib.agents.ppo import PPOTrainer, DEFAULT_CONFIG
from ray.tune.logger import pretty_print
!../../tools/start-ray.sh --check --verbose
ray.init(address='auto', ignore_reinit_error=True, log_to_driver=False)<jupyter_output><empty_output><jupyter_text>Here's one possible set. It takes longer for the max reward to reach 200, so I increased the number of episodes `N` to 10.<jupyter_code>config = DEFAULT_CONFIG.copy()
config['num_workers'] = 3
config['num_sgd_iter'] = 10                       # was 30
config['sgd_minibatch_size'] = 256                # was 128
config['model']['fcnet_hiddens'] = [20, 20]       # was [100, 100]
config['num_cpus_per_worker'] = 0
agent = PPOTrainer(config, 'CartPole-v1')
N=20                # was 10
results = []
episode_data = []
episode_json = []
for n in range(N):
    result = agent.train()
    results.append(result)
    episode = {'n': n, 
               'episode_reward_min': result['episode_reward_min'],  
               'episode_reward_mean': result['episode_reward_mean'], 
               'episode_reward_max': result['episode_reward_max'],  
               'episode_len_mean': result['episode_len_mean']}    
    episode_data.append(episode)
    episode_json.append(json.dumps(episode))
    print(f'{n:3d}: Min/Mean/Max reward: {result["episode_reward_min"]:8.4f}/{result["episode_reward_mean"]:8.4f}/{result["episode_reward_max"]:8.4f}')
df = pd.DataFrame(data=episode_data)
df
import sys
sys.path.append("../..")
from util.line_plots import plot_line, plot_line_with_min_max

import bokeh.io
# The next two lines prevent Bokeh from opening the graph in a new window.
bokeh.io.reset_output()
bokeh.io.output_notebook()
plot_line_with_min_max(df, x_col='n', y_col='episode_reward_mean', min_col='episode_reward_min', max_col='episode_reward_max',
                      title='Episode Rewards', x_axis_label='n', y_axis_label='reward')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10276854928017719, 'fraction_numerical': 0.020376522702104098, 'mean_word_length': 3.93551912568306, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '218829', 'license': 'Apache-2.0', 'n_tokens_mistral': 1492, 'n_tokens_neox': 1382, 'n_words': 431, 'path': 'ray-rllib/solutions/Ray-RLlib-Solutions.ipynb', 'repo_name': 'mathematicsofpaul/academy', 'size': '4515'}