<jupyter_start><jupyter_text>En este taller, aprenderemos los conceptos básicos para poder generar sistemas de RL con las Stable Baselines 3. Para ello, crearemos un modelo de RL, lo entrenaremos y lo evaluaremos. Como todos los algoritmos comparten la misma interfaz, veremos lo sencillo que es cambiar de un algoritmo a otro.  Setup<jupyter_code># Instalación de Stable Baselines 3 y dependencias
!apt-get install ffmpeg freeglut3-dev xvfb  # Para visualización
!pip install stable-baselines3[extra]<jupyter_output>Reading package lists... Done
Building dependency tree       
Reading state information... Done
freeglut3-dev is already the newest version (2.8.1-3).
freeglut3-dev set to manually installed.
ffmpeg is already the newest version (7:3.4.8-0ubuntu0.2).
The following NEW packages will be installed:
  xvfb
0 upgraded, 1 newly installed, 0 to remove and 40 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,270 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.9 [784 kB]
Fetched 784 kB in 2s (413 kB/s)
Selecting previously unselected package xvfb.
(Reading database ... 160837 files and directories currently installed.)
Preparing to unpack .../xvfb_2%3a1.19.6-1ubuntu4.9_amd64.deb ...
Unpacking xvfb (2:1.19.6-1ubuntu4.9) ...
Setting up xvfb (2:1.19.6-1ubuntu4.9) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Collecting stable-baselines3[extra]
  Downloading stable_baselines3-[...]<jupyter_text>Stable-Baselines3 funciona en entornos que se ciñen a la interfaz del gimnasio (https://stable-baselines3.readthedocs.io/en/master/guide/custom_env.html). En el siguiente enlace se puede consultar la lista de entornos disponibles: https://gym.openai.com/envs/classic_controlTambién se recomienda consultar el código fuente para conocer más sobre el espacio de observación y acción de cada env, ya que gym no cuenta con la documentación adecuada. No todos los algoritmos pueden funcionar con todos los espacios de acción, puede encontrar más en esta tabla de resumen<jupyter_code>import gym
import numpy as np
import os
import base64
from pathlib import Path
from IPython import display as ipythondisplay

# Modelo de RL preentrenado
from stable_baselines3 import PPO

# Clase de política que se utilizará para crear las redes (para las funciones de política / valor). 
# Este paso es opcional, ya que se pueden usar cadenas directamente en el constructor: 
# PPO('MlpPolicy', env) en lugar de PPO(MlpPolicy, env)
from stable_baselines3.ppo.policies import MlpPolicy

from stable_baselines3.common.evaluation import evaluate_policy<jupyter_output><empty_output><jupyter_text>Creación del entorno Gym y de una instancia del agentePara este ejemplo, usaremos el entorno CartPole, un problema de control clásico."Un poste está sujeto por una articulación no accionada a un carro, que se mueve a lo largo de una pista sin fricción. El sistema se controla aplicando una fuerza de +1 o -1 al carro. El péndulo comienza en posición vertical y el objetivo es evitar que se caiga. Se proporciona una recompensa de +1 por cada paso de tiempo que el poste permanece en posición vertical".Entorno de cartpole: https://gym.openai.com/envs/CartPole-v1/<jupyter_code>env = gym.make('CartPole-v1')

model = PPO(MlpPolicy, env, verbose=0)<jupyter_output><empty_output><jupyter_text>Creamos una función auxiliar para evaluar al agente:<jupyter_code>def evaluate(model, num_episodes=100):
    """
    Evaluate a RL agent
    :param model: (BaseRLModel object) the RL Agent
    :param num_episodes: (int) number of episodes to evaluate it
    :return: (float) Mean reward for the last num_episodes
    """
    # This function will only work for a single Environment
    env = model.get_env()
    all_episode_rewards = []
    for i in range(num_episodes):
        episode_rewards = []
        done = False
        obs = env.reset()
        while not done:
            # _states are only useful when using LSTM policies
            action, _states = model.predict(obs)
            # here, action, rewards and dones are arrays
            # because we are using vectorized env
            obs, reward, done, info = env.step(action)
            episode_rewards.append(reward)

        all_episode_rewards.append(sum(episode_rewards))

    mean_episode_reward = np.mean(all_episode_rewards)
    print("Mean reward:", mean_episode_reward, "Num episodes:", num_episodes)
    return mean_episode_reward<jupyter_output><empty_output><jupyter_text>Evaluemos al agente no entrenado, que de momento simplemente debería ser un agente aleatorio:*texto en cursiva*<jupyter_code># Random Agent, before training
mean_reward_before_train = evaluate(model, num_episodes=100)
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=100)

print(f"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}")

# Train the agent for 10000 steps
model.learn(total_timesteps=10000)
# Evaluate the trained agent
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=100)

print(f"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}")<jupyter_output>/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/evaluation.py:69: UserWarning: Evaluation environment is not wrapped with a ``Monitor`` wrapper. This may result in reporting modified episode lengths and rewards, if other wrappers happen to modify these. Consider wrapping environment first with ``Monitor`` wrapper.
  UserWarning,<jupyter_text>Aparentemente, el entrenamiento ha ido bien, ya que la recompensa ha aumentado mucho.  Grabación de vídeo<jupyter_code># # Configuramos una pantalla falsa; de lo contrario, la renderización fallará
os.system("Xvfb :1 -screen 0 1024x768x24 &")
os.environ['DISPLAY'] = ':1'
def show_videos(video_path='', prefix=''):
  """
  Taken from https://github.com/eleurent/highway-env

  :param video_path: (str) Path to the folder containing videos
  :param prefix: (str) Filter the video, showing only the only starting with this prefix
  """
  html = []
  for mp4 in Path(video_path).glob("{}*.mp4".format(prefix)):
      video_b64 = base64.b64encode(mp4.read_bytes())
      html.append('''<video alt="{}" autoplay 
                    loop controls style="height: 400px;">
                    <source src="data:video/mp4;base64,{}" type="video/mp4" />
                </video>'''.format(mp4, video_b64.decode('ascii')))
  ipythondisplay.display(ipythondisplay.HTML(data="<br>".join(html)))
from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv

def record_video(env_id, model, video_length=500, prefix='', video_folder='videos/'):
  """
  :param env_id: (str)
  :param model: (RL model)
  :param video_length: (int)
  :param prefix: (str)
  :param video_folder: (str)
  """
  eval_env = DummyVecEnv([lambda: gym.make(env_id)])
  # Start the video at step=0 and record 500 steps
  eval_env = VecVideoRecorder(eval_env, video_folder=video_folder,
                              record_video_trigger=lambda step: step == 0, video_length=video_length,
                              name_prefix=prefix)

  obs = eval_env.reset()
  for _ in range(video_length):
    action, _ = model.predict(obs)
    obs, _, _, _ = eval_env.step(action)

  # Close the video recorder
  eval_env.close()
record_video('CartPole-v1', model, video_length=500, prefix='ppo2-cartpole')
show_videos('videos', prefix='ppo2')

model = PPO('MlpPolicy', "CartPole-v1", verbose=1).learn(1000)<jupyter_output>Using cuda device
Creating environment from the given name 'CartPole-v1'
Wrapping the env with a `Monitor` wrapper
Wrapping the env in a DummyVecEnv.
---------------------------------
| rollout/           |          |
|    ep_len_mean     | 21.9     |
|    ep_rew_mean     | 21.9     |
| time/              |          |
|    fps             | 676      |
|    iterations      | 1        |
|    time_elapsed    | 3        |
|    total_timesteps | 2048     |
---------------------------------
{'fraction_non_alphanumeric': 0.099812382739212, 'fraction_numerical': 0.02564102564102564, 'mean_word_length': 3.8696711327649207, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '61401', 'license': 'MIT', 'n_tokens_mistral': 2760, 'n_tokens_neox': 2588, 'n_words': 797, 'path': 'notebooks/code/35 - Reinforcement Learning.ipynb', 'repo_name': 'oobregon/aepi-intelig-artificial', 'size': '7995'}