<jupyter_start><jupyter_text>Trying out basic openai gym and stable baselines stuff  Let's start with some code based off the snippet in the stable baselines readme: Create an environment for CartPole-v1<jupyter_code>import gym

from stable_baselines.common.policies import MlpPolicy
from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines.ppo2 import PPO2

env_cartpole = gym.make('CartPole-v1')
env_cartpole = DummyVecEnv([lambda: env_cartpole])  #` The algorithms require a vectorized environment to run<jupyter_output><empty_output><jupyter_text>Use PPO2 (just because it was in the example) to train an agent, and keep track of how many steps each run lasts. There's a lot of output, so we'll set verbosity to 0.<jupyter_code>model_ppo2 = PPO2(MlpPolicy, env_cartpole, verbose=0)
env_cartpole.reset()
model_ppo2.learn(total_timesteps=25000)<jupyter_output>WARNING: Logging before flag parsing goes to stderr.
W0814 21:31:29.220499 140328486889280 deprecation_wrapper.py:119] From /home/rishabh/anaconda3/envs/rl-env/lib/python3.7/site-packages/stable_baselines/common/tf_util.py:98: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0814 21:31:29.222148 140328486889280 deprecation_wrapper.py:119] From /home/rishabh/anaconda3/envs/rl-env/lib/python3.7/site-packages/stable_baselines/common/tf_util.py:107: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W0814 21:31:29.512462 140328486889280 deprecation_wrapper.py:119] From /home/rishabh/anaconda3/envs/rl-env/lib/python3.7/site-packages/stable_baselines/common/policies.py:115: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0814 21:31:29.513362 140328486889280 deprecation_wrapper.py:119] From /home/rishabh/anaconda3/envs/rl-env/lib/python3.7/site-packages/stable_baselines/common/input.py:2[...]<jupyter_text>Time to evaluate! Cart-Pole-v1 is considered solved when avg reward over 100 episdoes is at least 195.0, so we'll use a sliding window over the episodes (and some fun renders) to get an idea of how well the agent performs.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
def evaluate_model(model, env, num_eval_steps=25000, num_eval_episodes=100, render=True, plot=True):
    obs = env.reset()
    episode_durations = []
    current_duration = 0
    for i in range(num_eval_steps):
        if len(episode_durations) == num_eval_episodes:
                break
        action, _states = model.predict(obs)
        obs, reward, done, _ = env.step(action)
        current_duration += reward[0]
        if done[0] == 1:
            episode_durations.append(current_duration)
            current_duration = 0
            obs = env.reset()
        if render:
            env.render()
        
    return episode_durations
returns_ppo2 = evaluate_model(model_ppo2, env_cartpole, render=False)
print("Mean returns:", np.mean(returns_ppo2))<jupyter_output>Mean returns: 201.9<jupyter_text>Looks like that went pretty well, so let's try seeing how reducing the number of steps trained affects the agent.<jupyter_code>num_steps_to_train_ppo2 = [0, 10, 100, 500, 1000, 5000, 10_000, 25_000]
mean_rewards_ppo2 = []
for steps in num_steps_to_train_ppo2:
    curr_model_ppo2 = PPO2(MlpPolicy, env_cartpole, verbose=0)
    curr_model_ppo2.learn(total_timesteps=steps)
    mean_rewards_ppo2.append(np.mean(evaluate_model(curr_model_ppo2, env_cartpole, render=False)))
plt.plot(num_steps_to_train_ppo2, mean_rewards_ppo2)
plt.show()<jupyter_output><empty_output><jupyter_text>Seems we can get away with 10,000 of training using PPO2 for CarPole-V1. I'm interested to see how it compares to DQN.<jupyter_code>from stable_baselines import DQN
from stable_baselines.deepq import MlpPolicy as MlpPolicyDQN
num_steps_to_train_dqn = [0, 10, 100, 500, 1000, 5000, 10_000, 25_000]
mean_rewards_dqn = []
for steps in num_steps_to_train_dqn:
    curr_model_dqn = DQN(MlpPolicyDQN, env_cartpole, verbose=0)
    curr_model_dqn.learn(total_timesteps=steps)
    mean_rewards_dqn.append(np.mean(evaluate_model(curr_model_dqn, env_cartpole, render=False)))
plt.plot(num_steps_to_train_dqn, mean_rewards_dqn)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08955576559546313, 'fraction_numerical': 0.06521739130434782, 'mean_word_length': 5.299107142857143, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '555327', 'license': 'MIT', 'n_tokens_mistral': 1690, 'n_tokens_neox': 1467, 'n_words': 345, 'path': 'baselines-stuff/hello_woRLd.ipynb', 'repo_name': 'iarhbahsir/rl-scratch', 'size': '4232'}