<jupyter_start><jupyter_text>Stable Baselines3 - Advanced Saving and LoadingGithub Repo: [https://github.com/DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3)[RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) is a collection of pre-trained Reinforcement Learning agents using Stable-Baselines3.It also provides basic scripts for training, evaluating agents, tuning hyperparameters and recording videos.Documentation is available online: [https://stable-baselines3.readthedocs.io/](https://stable-baselines3.readthedocs.io/) IntroductionIn this notebook, you will learn how to use some advanced features of stable baselines3 (SB3): how to easily create a test environment for periodic evaluation, use a policy independently from a model (and how to save it, load it) and save/load a replay buffer. Install Dependencies and Stable Baselines Using Pip```pip install stable-baselines3[extra]```<jupyter_code>!pip install stable-baselines3[extra]<jupyter_output>Collecting stable-baselines3[extra]
[?25l  Downloading https://files.pythonhosted.org/packages/76/7c/ec89fd9a51c2ff640f150479069be817136c02f02349b5dd27a6e3bb8b3d/stable_baselines3-0.10.0-py3-none-any.whl (145kB)

[K     |██▎                             | 10kB 20.1MB/s eta 0:00:01
[K     |████▌                           | 20kB 10.6MB/s eta 0:00:01
[K     |██████▊                         | 30kB 8.3MB/s eta 0:00:01
[K     |█████████                       | 40kB 7.5MB/s eta 0:00:01
[K     |███████████▎                    | 51kB 8.4MB/s eta 0:00:01
[K     |█████████████▌                  | 61kB 8.6MB/s eta 0:00:01
[K     |███████████████▊                | 71kB 9.2MB/s eta 0:00:01
[K     |██████████████████              | 81kB 9.5MB/s eta 0:00:01
[K     |████████████████████▏           | 92kB 8.2MB/s eta 0:00:01
[K     |██████████████████████▌         | 102kB 8.1MB/s eta 0:00:01
[K     |████████████████████████▊       | 112kB 8.1MB/s eta 0:00:01
[K     |█████████████████████████[...]<jupyter_text>Import policy, RL agent, ...<jupyter_code>from stable_baselines3 import SAC, TD3
from stable_baselines3.common.evaluation import evaluate_policy<jupyter_output><empty_output><jupyter_text>Create the Gym env and instantiate the agentFor this example, we will use Pendulum environment."The inverted pendulum swingup problem is a classic problem in the control literature. In this version of the problem, the pendulum starts in a random position, and the goal is to swing it up so it stays upright."Pendulum-v0 environment: [https://gym.openai.com/envs/Pendulum-v0/](https://gym.openai.com/envs/Pendulum-v0/)![Pendulum](https://gym.openai.com/videos/2019-10-21--mqt8Qj1mwo/Pendulum-v0/poster.jpg)We chose the MlpPolicy because input of Pendulum is a feature vector, not images.The type of action to use (discrete/continuous) will be automatically deduced from the environment action space  Create the environment and evaluation environmentStable-Baselines3 allows to automatically create an environment for evaluation.For that, you only need to specify `create_eval_env=True` when passing the Gym ID of the environment.<jupyter_code>model = SAC('MlpPolicy', 'Pendulum-v0', verbose=1, learning_rate=1e-3, create_eval_env=True)<jupyter_output>Using cuda device
Creating environment from the given name 'Pendulum-v0'
Creating environment from the given name 'Pendulum-v0'
Wrapping the env in a DummyVecEnv.<jupyter_text>Train the agent and evaluate it periodically on the test env.Behind the scene, SB3 uses an [EvalCallback](https://stable-baselines3.readthedocs.io/en/master/guide/callbacks.htmlevalcallback)<jupyter_code># Evaluate the model every 1000 steps on 5 test episodes and save the evaluation to the logs folder
model.learn(6000, eval_freq=1000, n_eval_episodes=5, eval_log_path="./logs/")<jupyter_output>Wrapping the env in a DummyVecEnv.
----------------------------------
| rollout/           |           |
|    ep_len_mean     | 200       |
|    ep_rew_mean     | -1.67e+03 |
| time/              |           |
|    episodes        | 4         |
|    fps             | 79        |
|    time_elapsed    | 10        |
|    total timesteps | 800       |
| train/             |           |
|    actor_loss      | 29        |
|    critic_loss     | 1.96      |
|    ent_coef        | 0.503     |
|    ent_coef_loss   | -1.06     |
|    learning_rate   | 0.001     |
|    n_updates       | 699       |
----------------------------------
Eval num_timesteps=1000, episode_reward=-1564.58 +/- 57.54
Episode length: 200.00 +/- 0.00
New best mean reward!
----------------------------------
| eval/              |           |
|    mean_ep_length  | 200       |
|    mean_reward     | -1.56e+03 |
| rollout/           |           |
|    ep_len_mean     | 200       |
|    ep_rew_mean     | -1.53e+03 |
| time/     [...]<jupyter_text>Save/Load the replay bufferBy default, the replay buffer is not saved when calling `model.save()`, in order to save space on the disk (a replay buffer can be up to several GB when using images).However, SB3 provides a `save_replay_buffer()` and `load_replay_buffer()` method to save it separately.<jupyter_code># save the model
model.save("sac_pendulum")

# the saved model does not contain the replay buffer
loaded_model = SAC.load("sac_pendulum")
print(f"The loaded_model has {loaded_model.replay_buffer.size()} transitions in its buffer")
# now save the replay buffer too
model.save_replay_buffer("sac_replay_buffer")

# load it into the loaded_model
loaded_model.load_replay_buffer("sac_replay_buffer")

# now the loaded replay is not empty anymore
print(f"The loaded_model has {loaded_model.replay_buffer.size()} transitions in its buffer")<jupyter_output>The loaded_model has 2079 transitions in its buffer<jupyter_text>Save the policy onlyIn SB3, you save the policy independently from the model if needed.Note: if you don't save the complete model, you cannot continue training afterward<jupyter_code>policy = model.policy
policy.save("sac_policy_pendulum.pkl")
env = model.get_env()
# Evaluate the policy
mean_reward, std_reward = evaluate_policy(policy, env, n_eval_episodes=10, deterministic=True)

print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")<jupyter_output>mean_reward=-1007.45 +/- 218.98353576660156<jupyter_text>Load the policy only<jupyter_code>from stable_baselines3.sac.policies import MlpPolicy
saved_policy = MlpPolicy.load("sac_policy_pendulum")
saved_policy = MlpPolicy.load("sac_policy_pendulum.pkl")
# Evaluate the loaded policy
mean_reward, std_reward = evaluate_policy(saved_policy, env, n_eval_episodes=10, deterministic=True)

print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1372871227965342, 'fraction_numerical': 0.04735584105168808, 'mean_word_length': 3.3530559167750327, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '290649', 'license': 'MIT', 'n_tokens_mistral': 2547, 'n_tokens_neox': 2198, 'n_words': 566, 'path': 'advanced_saving_loading(colab).ipynb', 'repo_name': 'StevenJokess/rl-colab-notebooks', 'size': '6694'}