<jupyter_start><jupyter_text>Day 1, Part C: Faster Learning  Learning goals- Learning can be accelerated by a number of factors- Test multi-environment training using Subprocess Vectorization  Definitions- **Simulation environment**: Notice that this is not the same as the python/conda environment.  The simulation environment is the simulated world where the reinforcement learning takes place.  It provides opportunities for an agent to learn and explore, and ideally provides challenges that aid in efficient learning.- **Agent (aka actor or policy)**: An entity in the simulation environment that performs actions.  The agent could be a person, a robot, a car, a thermostat, etc.- **State variable**: An observed variable in the simulation environment.  They can be coordinates of objects or entities, an amount of fuel in a tank, air temperature, wind speed, etc.- **Action variable**: An action that the agent can perform.  Examples: step forward, increase velocity to 552.5 knots, push object left with force of 212.3 N, etc.- **Reward**: A value given to the agent for doing something considered to be 'good'.  Reward is commonly assigned at each time step and cumulated during a learning episode.- **Episode**: A learning event consisting of multiple steps in which the agent can explore.  It starts with the unmodified environment and continues until the goal is achieved or something prevents further progress, such as a robot getting stuck in a hole.  Multiple episodes are typically run in loops until the model is fully trained.- **Model (aka policy or agent)**: An RL model is composed of the modeling architecture (e.g., neural network) and parameters or weights that define the unique behavior of the model.- **Policy (aka model or agent)**: The parameters of a model that encode the best choices to make in an environment.  The choices are not necessarily good ones until the model undergoes training.  The policy (or model) is the "brain" of the agent.- **Replay Buffer**: A place in memory to store state, action, reward and other variables describing environmental state transitions. It is effectively the agent's memory of past experiences. ![Reinforcement Learning Cycle](./images/Reinforcement-learning-diagram-01.png)  Some options for increasing learning rate- Multiprocessing (multiple instances of cartpole) - we go into this topic below where we create multiple environments and share learning across them.- Better reward functions, as we briefly touched on in earlier parts, and will return to in Day 4- Tricks like moving the agent farther from target as learning progresses or gradual task-adjustment environment changes- Add attention or other innovative algorithm enhancements from new research articles  Multiprocessing and Shared LearningBecause RL is focused on decision sequences in which an agent's actions influence future environment states, it is difficult to parallelize computations and speed up learning.  Multi-agent techniques, however, allow shared learning from identical agents.  OpenAI Gym provides a method for this by creating multiple identical environments, each with an identical agent.  By giving each environment a different random seed, they each begin learning in a different state.  These environments can run on seperate processes on a single computer at the same time, and the experiences of all of the agents will be shared.<jupyter_code>import os
import gym
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, SubprocVecEnv
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.utils import set_random_seed
from tqdm import trange<jupyter_output><empty_output><jupyter_text>This function helps us make multiple environments:<jupyter_code>def make_env(env_id, rank, seed=0):
    """
    Utility function for multiprocessed env.

    :param env_id: (str) the environment ID
    :param num_env: (int) the number of environments you wish to have in subprocesses
    :param seed: (int) the inital seed for RNG
    :param rank: (int) index of the subprocess
    """
    def _init():
        env = gym.make(env_id)
        env.seed(seed + rank)
        return env
    
    set_random_seed(seed)
    
    return _init<jupyter_output><empty_output><jupyter_text><jupyter_code>env_id = "CartPole-v1"<jupyter_output><empty_output><jupyter_text>Let's use multiple cpus, but avoid using all of them so that the computer remains stable<jupyter_code>num_cpu = os.cpu_count()
if num_cpu > 2:
    num_cpu -= 1<jupyter_output><empty_output><jupyter_text>Create the vectorized environment<jupyter_code>env = SubprocVecEnv([make_env(env_id, i) for i in range(num_cpu)])<jupyter_output><empty_output><jupyter_text>Start learning then display the result<jupyter_code>log_dir = "tmp/"
os.makedirs(log_dir, exist_ok=True)
model = PPO('MlpPolicy', env, verbose=1, tensorboard_log=log_dir, n_steps=256)
model.learn(total_timesteps=50000)<jupyter_output><empty_output><jupyter_text>Now try modifying the n_steps and total_timesteps above to see how it affects training time We now save the model and reload from a single cartpole env to view the results of our training<jupyter_code>model.save("ppo_cartpole")
del model
env = gym.make("CartPole-v1")
model = PPO.load("ppo_cartpole")
obs = env.reset()
for _ in range(1000):
    action, _states = model.predict(obs)
    obs, rewards, dones, info = env.step(action)
    env.render()
env.env.viewer.close()<jupyter_output><empty_output><jupyter_text>Run cell below to show training loss plots<jupyter_code>!tensorboard --logdir ./tmp/<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06543385490753911, 'fraction_numerical': 0.00604551920341394, 'mean_word_length': 5.181318681318682, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '419480', 'license': 'MIT', 'n_tokens_mistral': 1516, 'n_tokens_neox': 1458, 'n_words': 695, 'path': 'Course_Material/Day1_PartC.ipynb', 'repo_name': 'rshnn/Practical-RL', 'size': '5624'}