<jupyter_start><jupyter_text>HVAC with Amazon SageMaker RL--- IntroductionHVAC stands for Heating, Ventilation and Air Conditioning and is responsible for keeping us warm and comfortable indoors.  HVAC takes up a whopping 50% of the energy in a building and accounts for 40% of energy use in the US [1, 2]. Several control system optimizations have been proposed to reduce energy usage while ensuring thermal comfort.Modern buildings collect data about the weather, occupancy and equipment use. All of this can be used to optimize HVAC energy usage. Reinforcement Learning (RL) is a good fit because it can learn how to interact with the environment and identify strategies to limit wasted energy. Several recent research efforts have shown that RL can reduce HVAC energy consumption by 15-20% [3, 4].As training an RL algorithm in a real HVAC system can take time to converge as well as potentially lead to hazardous settings as the agent explores its state space, we turn to a simulator to train the agent. [EnergyPlus](https://energyplus.net/) is an open source, state of the art HVAC simulator from the US Department of Energy. We use a simple example with this simulator to showcase how we can train an RL model easily with Amazon SageMaker RL.1. Objective: Control the data center HVAC system to reduce energy consumption while ensuring the room temperature stays within specified limits.2. Environment: We have a small single room datacenter that the HVAC system is cooling to ensure the compute equipment works properly. We will train our RL agent to control this HVAC system for one day subject to weather conditions in San Francisco.  The agent takes actions every 5 minutes for a 24 hour period. Hence, the episode is a fixed 120 steps. 3. State: The outdoor temperature, outdoor humidity and indoor room temperature.4. Action: The agent can set the heating and cooling setpoints. The cooling setpoint tells the HVAC system that it should start cooling the room if the room temperature goes above this setpoint. Likewise, the HVAC systems starts heating if the room temperature goes below the heating setpoint.5. Reward: The rewards has two components which are added together with coefficients:     1. It is proportional to the energy consumed by the HVAC system.    2. It gets a large penalty when the room temperature exceeds pre-specified lower or upper limits (as defined in `data_center_env.py`).References1. [sciencedirect.com](https://www.sciencedirect.com/science/article/pii/S0378778807001016)2. [environment.gov.au](https://www.environment.gov.au/system/files/energy/files/hvac-factsheet-energy-breakdown.pdf)3. Wei, Tianshu, <NAME>, and <NAME>. "Deep reinforcement learning for building hvac control." In Proceedings of the 54th Annual Design Automation Conference 2017, p. 22. ACM, 2017.4. <NAME>, and <NAME>. "Practical implementation and evaluation of deep reinforcement learning control for a radiant heating system." In Proceedings of the 5th Conference on Systems for Built Environments, pp. 148-157. ACM, 2018.  Pre-requisites  ImportsTo get started, we'll import the Python libraries we need, set up the environment with a few prerequisites for permissions and configurations.<jupyter_code>import sagemaker
import boto3
import sys
import os
import glob
import re
import subprocess
import numpy as np
from IPython.display import HTML
import time
from time import gmtime, strftime
sys.path.append("common")
from misc import get_execution_role, wait_for_s3_object
from docker_utils import build_and_push_docker_image
from sagemaker.rl import RLEstimator, RLToolkit, RLFramework<jupyter_output><empty_output><jupyter_text>Setup S3 bucketCreate a reference to the default S3 bucket that will be used for model outputs.<jupyter_code>sage_session = sagemaker.session.Session()
s3_bucket = sage_session.default_bucket()  
s3_output_path = 's3://{}/'.format(s3_bucket)
print("S3 bucket path: {}".format(s3_output_path))<jupyter_output>S3 bucket path: s3://sagemaker-us-east-1-912668378040/<jupyter_text>Define Variables We define a job below that's used to identify our jobs.<jupyter_code># create unique job name 
job_name_prefix = 'rl-hvac'<jupyter_output><empty_output><jupyter_text>Configure settingsYou can run your RL training jobs locally on the SageMaker notebook instance or on SageMaker training. In both of these scenarios, you can run in either 'local' (where you run the commands) or 'SageMaker' mode (on SageMaker training instances). 'local' mode uses the SageMaker Python SDK to run your code in Docker containers locally. It can speed up iterative testing and debugging while using the same familiar Python SDK interface. Just set `local_mode = True`. And when you're ready move to 'SageMaker' mode to scale things up.<jupyter_code># run local (on this machine)?
# or on sagemaker training instances?
local_mode = False

if local_mode:
    instance_type = 'local'
else:
    # choose a larger instance to avoid running out of memory
    instance_type = "ml.m4.4xlarge"<jupyter_output><empty_output><jupyter_text>Create an IAM roleEither get the execution role when running from a SageMaker notebook instance `role = sagemaker.get_execution_role()` or, when running from local notebook instance, use utils method `role = get_execution_role()` to create an execution role.<jupyter_code>try:
    role = sagemaker.get_execution_role()
except:
    role = get_execution_role()

print("Using IAM role arn: {}".format(role))<jupyter_output>Using IAM role arn: arn:aws:iam::912668378040:role/service-role/AmazonSageMaker-ExecutionRole-20200602T222454<jupyter_text>Install docker for `local` modeIn order to work in `local` mode, you need to have docker installed. When running from your local machine, please make sure that you have docker or docker-compose (for local CPU machines) and nvidia-docker (for local GPU machines) installed. Alternatively, when running from a SageMaker notebook instance, you can simply run the following script to install dependencies.Note, you can only run a single local notebook at one time.<jupyter_code># Only run from SageMaker notebook instance
if local_mode:
    !/bin/bash ./common/setup.sh<jupyter_output><empty_output><jupyter_text>Build docker containerSince we're working with a custom environment with custom dependencies, we create our own container for training. We:1. Fetch the base MXNet and Coach container image,2. Install EnergyPlus and its dependencies on top,3. Upload the new container image to AWS ECR.<jupyter_code>cpu_or_gpu = 'gpu' if instance_type.startswith('ml.p') else 'cpu'
repository_short_name = "sagemaker-hvac-coach-%s" % cpu_or_gpu
docker_build_args = {
    'CPU_OR_GPU': cpu_or_gpu, 
    'AWS_REGION': boto3.Session().region_name,
}
custom_image_name = build_and_push_docker_image(repository_short_name, build_args=docker_build_args)
print("Using ECR image %s" % custom_image_name)<jupyter_output>WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
Logged into ECR
Building docker image sagemaker-hvac-coach-cpu from Dockerfile
$ docker build -t sagemaker-hvac-coach-cpu -f Dockerfile . --build-arg CPU_OR_GPU=cpu --build-arg AWS_REGION=us-east-1
Sending build context to Docker daemon    7.3MB
Step 1/15 : ARG CPU_OR_GPU
Step 2/15 : ARG AWS_REGION
Step 3/15 : FROM 520713654638.dkr.ecr.${AWS_REGION}.amazonaws.com/sagemaker-rl-mxnet:coach0.11.0-${CPU_OR_GPU}-py3
coach0.11.0-cpu-py3: Pulling from sagemaker-rl-mxnet
7b8b6451c85f: Pulling fs layer
ab4d1096d9ba: Pulling fs layer
e6797d1788ac: Pulling fs layer
e25c5c290bde: Pulling fs layer
480449168c52: Pulling fs layer
ba06919a55f3: Pulling fs layer
6b5aafe7de05: Pulling fs layer
[...]<jupyter_text>Setup the environmentThe environment is defined in a Python file called `data_center_env.py` and for SageMaker training jobs, the file will be uploaded inside the `/src` directory.The environment implements the init(), step() and reset() functions that describe how the environment behaves. This is consistent with Open AI Gym interfaces for defining an environment.1. `init()` - initialize the environment in a pre-defined state2. `step()` - take an action on the environment3. `reset()` - restart the environment on a new episode  Configure the presets for RL algorithm The presets that configure the RL training jobs are defined in the “preset-energy-plus-clipped-ppo.py” file which is also uploaded as part of the `/src` directory. Using the preset file, you can define agent parameters to select the specific agent algorithm. You can also set the environment parameters, define the schedule and visualization parameters, and define the graph manager. The schedule presets will define the number of heat up steps, periodic evaluation steps, training steps between evaluations, etc.All of these can be overridden at run-time by specifying the `RLCOACH_PRESET` hyperparameter. Additionally, it can be used to define custom hyperparameters.<jupyter_code>!pygmentize src/preset-energy-plus-clipped-ppo.py<jupyter_output>[34mfrom[39;49;00m [04m[36mrl_coach.agents.clipped_ppo_agent[39;49;00m [34mimport[39;49;00m ClippedPPOAgentParameters
[34mfrom[39;49;00m [04m[36mrl_coach.architectures.layers[39;49;00m [34mimport[39;49;00m Dense
[34mfrom[39;49;00m [04m[36mrl_coach.base_parameters[39;49;00m [34mimport[39;49;00m VisualizationParameters, PresetValidationParameters, DistributedCoachSynchronizationType
[34mfrom[39;49;00m [04m[36mrl_coach.core_types[39;49;00m [34mimport[39;49;00m TrainingSteps, EnvironmentEpisodes, EnvironmentSteps, RunPhase
[34mfrom[39;49;00m [04m[36mrl_coach.environments.gym_environment[39;49;00m [34mimport[39;49;00m GymVectorEnvironment, mujoco_v2
[34mfrom[39;49;00m [04m[36mrl_coach.exploration_policies.additive_noise[39;49;00m [34mimport[39;49;00m AdditiveNoiseParameters
[34mfrom[39;49;00m [04m[36mrl_coach.exploration_policies.e_greedy[39;49;00m [34mimport[39;49;00m EGreedyParameters
[34mfrom[39;49;00m [04m[36mrl_coach.filters.observat[...]<jupyter_text>Write the Training Code The training code is written in the file “train-coach.py” which is uploaded in the /src directory. First import the environment files and the preset files, and then define the main() function.<jupyter_code>!pygmentize src/train-coach.py<jupyter_output>[34mfrom[39;49;00m [04m[36msagemaker_rl.coach_launcher[39;49;00m [34mimport[39;49;00m SageMakerCoachPresetLauncher


[34mclass[39;49;00m [04m[32mMyLauncher[39;49;00m(SageMakerCoachPresetLauncher):

    [34mdef[39;49;00m [32mdefault_preset_name[39;49;00m([36mself[39;49;00m):
        [33m"""This points to a .py file that configures everything about the RL job.[39;49;00m
[33m        It can be overridden at runtime by specifying the RLCOACH_PRESET hyperparameter.[39;49;00m
[33m        """[39;49;00m
        [34mreturn[39;49;00m [33m'[39;49;00m[33mpreset-energy-plus-clipped-ppo[39;49;00m[33m'[39;49;00m
    
    
[34mif[39;49;00m [31m__name__[39;49;00m == [33m'[39;49;00m[33m__main__[39;49;00m[33m'[39;49;00m:
    MyLauncher.train_main()<jupyter_text>Train the RL model using the Python SDK Script modeIf you are using local mode, the training will run on the notebook instance. When using SageMaker for training, you can select a GPU or CPU instance. The RLEstimator is used for training RL jobs. 1. Specify the source directory where the environment, presets and training code is uploaded.2. Specify the entry point as the training code 3. Specify the choice of RL toolkit and framework. This automatically resolves to the ECR path for the RL Container. 4. Define the training parameters such as the instance count, job name, S3 path for output and job name. 5. Specify the hyperparameters for the RL agent algorithm. The RLCOACH_PRESET can be used to specify the RL agent algorithm you want to use. 6. [optional] Define the metrics definitions that you are interested in capturing in your logs. These can also be visualized in CloudWatch and SageMaker Notebooks.<jupyter_code>%%time
estimator = RLEstimator(entry_point="train-coach.py",
                        source_dir='src',
                        dependencies=["common/sagemaker_rl"],
                        image_name=custom_image_name,
                        role=role,
                        train_instance_type=instance_type,
                        train_instance_count=1,
                        output_path=s3_output_path,
                        base_job_name=job_name_prefix,
                        hyperparameters = {
                            'save_model': 1
                        }
                    )

estimator.fit(wait=True)
job_name = estimator.latest_training_job.job_name
print("Training job: %s" % job_name)<jupyter_output>2020-06-29 22:19:46 Starting - Starting the training job...
2020-06-29 22:19:49 Starting - Launching requested ML instances......
2020-06-29 22:21:08 Starting - Preparing the instances for training......
2020-06-29 22:22:10 Downloading - Downloading input data
2020-06-29 22:22:10 Training - Downloading the training image......
2020-06-29 22:23:15 Training - Training image download completed. Training in progress..[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device[0m
[34mbash: no job control in this shell[0m
[34m2020-06-29 22:23:17,217 sagemaker-containers INFO     Imported framework sagemaker_mxnet_container.training[0m
[34m2020-06-29 22:23:17,221 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)[0m
[34m2020-06-29 22:23:17,235 sagemaker_mxnet_container.training INFO     MXNet training environment: {'SM_RESOURCE_CONFIG': '{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"}', 'SM_OUTPUT_INTERMEDIAT[...]<jupyter_text>Store intermediate training output and model checkpoints The output from the training job above is stored on S3. The intermediate folder contains gifs and metadata of the training.<jupyter_code>s3_url = "s3://{}/{}".format(s3_bucket,job_name)

if local_mode:
    output_tar_key = "{}/output.tar.gz".format(job_name)
else:
    output_tar_key = "{}/output/output.tar.gz".format(job_name)

intermediate_folder_key = "{}/output/intermediate/".format(job_name)
output_url = "s3://{}/{}".format(s3_bucket, output_tar_key)
intermediate_url = "s3://{}/{}".format(s3_bucket, intermediate_folder_key)

print("S3 job path: {}".format(s3_url))
print("Output.tar.gz location: {}".format(output_url))
print("Intermediate folder path: {}".format(intermediate_url))
    
tmp_dir = "/tmp/{}".format(job_name)
os.system("mkdir {}".format(tmp_dir))
print("Create local folder {}".format(tmp_dir))<jupyter_output>S3 job path: s3://sagemaker-us-east-1-912668378040/rl-hvac-2020-06-29-22-19-45-802
Output.tar.gz location: s3://sagemaker-us-east-1-912668378040/rl-hvac-2020-06-29-22-19-45-802/output/output.tar.gz
Intermediate folder path: s3://sagemaker-us-east-1-912668378040/rl-hvac-2020-06-29-22-19-45-802/output/intermediate/
Create local folder /tmp/rl-hvac-2020-06-29-22-19-45-802<jupyter_text>Visualization  Plot metrics for training jobWe can pull the reward metric of the training and plot it to see the performance of the model over time.<jupyter_code>%matplotlib inline
import pandas as pd

csv_file_name = "worker_0.simple_rl_graph.main_level.main_level.agent_0.csv"
key = os.path.join(intermediate_folder_key, csv_file_name)
wait_for_s3_object(s3_bucket, key, tmp_dir)

csv_file = "{}/{}".format(tmp_dir, csv_file_name)
df = pd.read_csv(csv_file)
df = df.dropna(subset=['Training Reward'])
x_axis = 'Episode #'
y_axis = 'Training Reward'

plt = df.plot(x=x_axis,y=y_axis, figsize=(12,5), legend=True, style='b-')
plt.set_ylabel(y_axis);
plt.set_xlabel(x_axis);<jupyter_output>Waiting for s3://sagemaker-us-east-1-912668378040/rl-hvac-2020-06-29-22-19-45-802/output/intermediate/worker_0.simple_rl_graph.main_level.main_level.agent_0.csv...
Downloading rl-hvac-2020-06-29-22-19-45-802/output/intermediate/worker_0.simple_rl_graph.main_level.main_level.agent_0.csv
{'fraction_non_alphanumeric': 0.09307905259920024, 'fraction_numerical': 0.05641341125807444, 'mean_word_length': 5.115876598946576, 'pattern_counts': {'":': 3, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '435470', 'license': 'Apache-2.0', 'n_tokens_mistral': 5742, 'n_tokens_neox': 4966, 'n_words': 1653, 'path': 'rl/rl_hvac_coach_energyplus/rl_hvac_coach_energyplus.ipynb', 'repo_name': 'Laurenstc/partner-ml-bootcamp', 'size': '16273'}