<jupyter_start><jupyter_text>Profiling PyTorch training job using Amazon SageMaker Debugger In this notebook we will demonstrate the deep profiling capability of Amazon SageMaker Debugger1. Setup2. Train a PyTorch Tensorflow model for weather prediction with debugger enabled. 3. Analyze and visualize the system and framework metrics generated by the profiler.4. Analyze the profiler report generated by SageMaker Debugger.5. Review and implement recommendations from the profiler report.  Step 1 - Setup  1.1 Import libraries<jupyter_code>import sagemaker
import boto3
from sagemaker import image_uris
from sagemaker.session import Session
from sagemaker.inputs import TrainingInput

from sagemaker.pytorch import PyTorch
from sagemaker.debugger import ProfilerConfig, FrameworkProfile<jupyter_output><empty_output><jupyter_text>1.2 Define variables<jupyter_code>#Set the s3_bucket to the correct bucket name created in your datascience environment
s3_bucket = 'datascience-environment-notebookinstance--06dc7a0224df'
s3_prefix = 'prepared'
region = boto3.Session().region_name<jupyter_output><empty_output><jupyter_text>1.3 Setup service clients<jupyter_code>sagemaker_client = boto3.client("sagemaker")
s3_client = boto3.client('s3', region_name=region)<jupyter_output><empty_output><jupyter_text>1.4 Training and validation data inputs to training job<jupyter_code>##Get the file name at index from the 'prefix' folder
def get_file_in_bucket(prefix,index):
    response = s3_client.list_objects(
        Bucket=s3_bucket,
        Prefix=s3_prefix + "/" + prefix
    )
    ## At '0' index you will find the SUCCESS/FAILURE of file uploades to S3. First data file is at index 1
    file_name = response['Contents'][index]['Key']
    print("Returing file name : " + file_name)
    return file_name
content_type = "csv"

# Define the data type and paths to the training and validation datasets

#Since we are using powerful CPU/GPU instances for training over hours, you can choose to use a single file 
#for training and validation instead of the entrie dataset to save some time and trainging costs.  Change the variable
#use_full_data to True to use the complete dataset
use_full_data=False

#Different train and validation inputs
#define the data type and paths to the training and validation datasets
if use_full_data == False:
    ##Update the csv file names to match the contents in your S3 bucket
    #train_input = TrainingInput("s3://{}/{}/{}/part-00000-2554f113-947e-46bd-be31-9cd75cb4661c-c000.csv".format(s3_bucket, s3_prefix, 'train'), content_type=content_type)
    #validation_input = TrainingInput("s3://{}/{}/{}/part-00000-85addac2-a753-4bc2-b157-26ff8f5d5952-c000.csv".format(s3_bucket, s3_prefix, 'validation'), content_type=content_type)
    train_input = TrainingInput("s3://{}/{}".format(s3_bucket, get_file_in_bucket('train',1)), content_type=content_type)
    validation_input = TrainingInput("s3://{}/{}".format(s3_bucket, get_file_in_bucket('validation',1)), content_type=content_type)
else:
    train_input = TrainingInput("s3://{}/{}/{}/".format(s3_bucket, s3_prefix, 'train'), content_type=content_type, distribution='ShardedByS3Key')
    validation_input = TrainingInput("s3://{}/{}/{}/".format(s3_bucket, s3_prefix, 'validation'), content_type=content_type, distribution='ShardedByS3Key')<jupyter_output>Returing file name : prepared/train/part-00000-2554f113-947e-46bd-be31-9cd75cb4661c-c000.csv
Returing file name : prepared/validation/part-00000-85addac2-a753-4bc2-b157-26ff8f5d5952-c000.csv<jupyter_text>Step 2 - Train a PyTorch Tensorflow model for weather prediction with debugger enabled.  2.1 Define training related variables<jupyter_code>train_instance_type = "ml.p3.2xlarge"
instance_count = 2<jupyter_output><empty_output><jupyter_text>2.2 Define profiler configurationWith the following profiler_config parameter configuration, Debugger calls the default settings of monitoring, collecting system metrics every 500 milliseconds. For collecting framework metrics, you can set target steps and target time intervals in detail.<jupyter_code>profiler_config = ProfilerConfig(
    framework_profile_params=FrameworkProfile(start_step=2, num_steps=7)
)<jupyter_output><empty_output><jupyter_text>With this profiler_config settings, Debugger will collect system metrics every 500 milliseconds and framework metrics on the specified steps (from step 2 to 9). For a complete list of parameters and profiling configurations, see Configure Debugger Using Amazon SageMaker Python SDK in the Amazon SageMaker Debugger developer guide.  2.3 Create PyTorch Estimator<jupyter_code>hyperparameters = {
    "nproc_per_node": 4,
    "nnodes": 2,
}

pt_estimator = PyTorch(
    entry_point="train_pytorch.py",
    source_dir="code",
    role=sagemaker.get_execution_role(),
    instance_count=instance_count,
    instance_type=train_instance_type,
    framework_version="1.6",
    py_version="py3",
    volume_size=1024,
    hyperparameters=hyperparameters,
    #Debugger-specific parameters
    profiler_config=profiler_config,
)
## Kickoff training
pt_estimator.fit({'train': train_input, 'test': validation_input})<jupyter_output>2021-08-09 03:35:03 Starting - Starting the training job...
2021-08-09 03:35:26 Starting - Launching requested ML instancesProfilerReport-1628480102: InProgress
...
2021-08-09 03:35:58 Starting - Preparing the instances for training...............
2021-08-09 03:38:27 Downloading - Downloading input data
2021-08-09 03:38:27 Training - Downloading the training image.......[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device[0m
[34mbash: no job control in this shell[0m
[34m2021-08-09 03:39:33,041 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training[0m
[34m2021-08-09 03:39:33,065 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.[0m
[35mbash: cannot set terminal process group (-1): Inappropriate ioctl for device[0m
[35mbash: no job control in this shell[0m
[35m2021-08-09 03:39:38,777 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training[...]<jupyter_text>Step 3 - Analyze and visualize the system and framework metrics generated by the profiler.  3.1 Manual Analysis<jupyter_code>from smdebug.profiler.system_metrics_reader import S3SystemMetricsReader
import time
#All collected metrics are persisted in S3.  Define path to the profiler artifacts
path = pt_estimator.latest_job_profiler_artifacts_path()
#Create a reader for the system metrics
system_metrics_reader = S3SystemMetricsReader(path)
training_job_name = pt_estimator.latest_training_job.name
print(f"Training job name: {training_job_name}")
training_job_status = ""
training_job_secondary_status = ""
##Wait till the profiler data is available
while system_metrics_reader.get_timestamp_of_latest_available_file() == 0:
    system_metrics_reader.refresh_event_file_list()
    client = sagemaker_client.describe_training_job(TrainingJobName=training_job_name)
    if "TrainingJobStatus" in client:
        training_job_status = f"TrainingJobStatus: {client['TrainingJobStatus']}"
    if "SecondaryStatus" in client:
        training_job_secondary_status = f"TrainingJobSecondaryStatus: {client['SecondaryStatus']}"

    print(
        f"Profiler data from system not available yet. {training_job_status}. {training_job_secondary_status}."
    )
    time.sleep(20)

print("\n\nProfiler data from system is available")
from datetime import datetime

def timestamp_to_utc(timestamp):
    utc_dt = datetime.utcfromtimestamp(timestamp)
    return utc_dt.strftime("%Y-%m-%d %H:%M:%S")<jupyter_output><empty_output><jupyter_text>Now that the data is available we can query and inspect it. We get the latest available timestamp and query all the events within the given time range:<jupyter_code>system_metrics_reader.refresh_event_file_list()
#Get the latest event
last_timestamp = system_metrics_reader.get_timestamp_of_latest_available_file()
events = system_metrics_reader.get_events(0, last_timestamp * 1000000)  # UTC time in micro seconds

print(
    "Found",
    len(events),
    "recorded system metric events. Latest recorded event:",
    timestamp_to_utc(last_timestamp / 1000000),
)  # UTC time in seconds to datetime<jupyter_output>[2021-08-09 04:27:11.737 ip-172-16-11-227:19699 INFO metrics_reader_base.py:134] Getting 42 event files
Found 126056 recorded system metric events. Latest recorded event: 2021-08-09 03:58:00<jupyter_text>We can iterate over the list of recorded events. Let's have a look on the first event.<jupyter_code>#Show the first system metric event collected
print(
    "Event name:",
    events[0].name,
    "\nTimestamp:",
    timestamp_to_utc(events[0].timestamp),
    "\nValue:",
    events[0].value,
)<jupyter_output>Event name: gpu0 
Timestamp: 2021-08-09 03:38:01 
Value: 0.0<jupyter_text>3.2 GPU and CPU usageMetricHistogram computes a histogram on GPU and CPU utilization values. Bins are between 0 and 100. Good system utilization means that the center of the distribution should be between 80 to 90. In case of multi-GPU training: if distributions of GPU utilization values are not similar it indicates an issue with workload distribution.The following cell will plot the histograms per metric. In order to only plot specific metrics, define the list select_dimensions and select_events. A dimension can be CPUUtilization, GPUUtilization, GPUMemoryUtilization IOPS. With CPUUtilization dimension, CPU uiltization histogram for each single core and total CPU usage will be plotted. In case of GPU, it will visualize utilization and memory for each GPU. In case of IOPS, it will plot IO wait time per CPU. If select_events is specified then only metrics that match the name in select_metrics will be shown. If neither select_dimensions nor select_events are specified, all available metrics will be visualized. One can also specify a start and endtime.<jupyter_code>from smdebug.profiler.analysis.notebook_utils.metrics_histogram import MetricsHistogram

system_metrics_reader.refresh_event_file_list()
metrics_histogram = MetricsHistogram(system_metrics_reader)
metrics_histogram.plot()<jupyter_output><empty_output><jupyter_text>3.3 Read profiling data: framework annotations<jupyter_code>from smdebug.profiler.algorithm_metrics_reader import S3AlgorithmMetricsReader

framework_metrics_reader = S3AlgorithmMetricsReader(path)

events = []
while framework_metrics_reader.get_timestamp_of_latest_available_file() == 0 or len(events) == 0:
    framework_metrics_reader.refresh_event_file_list()
    last_timestamp = framework_metrics_reader.get_timestamp_of_latest_available_file()
    events = framework_metrics_reader.get_events(0, last_timestamp)

    print("Profiler data from framework not available yet")
    time.sleep(20)

print("\n\n Profiler data from framework is available")<jupyter_output>[2021-08-09 04:27:29.444 ip-172-16-11-227:19699 INFO algorithm_metrics_reader.py:192] S3AlgorithmMetricsReader created with bucket:sagemaker-us-west-2-802439482869 and prefix:pytorch-training-2021-08-09-03-35-01-831/profiler-output/framework/
[2021-08-09 04:27:29.665 ip-172-16-11-227:19699 INFO metrics_reader_base.py:134] Getting 40 event files
[2021-08-09 04:27:47.400 ip-172-16-11-227:19699 INFO trace_event_file_parser.py:197] Start time for events in uSeconds = 1628480410657345
[2021-08-09 04:27:48.277 ip-172-16-11-227:19699 INFO trace_event_file_parser.py:197] Start time for events in uSeconds = 1628480411445444
[2021-08-09 04:27:49.047 ip-172-16-11-227:19699 INFO trace_event_file_parser.py:197] Start time for events in uSeconds = 1628480410657345
[2021-08-09 04:27:50.022 ip-172-16-11-227:19699 INFO trace_event_file_parser.py:197] Start time for events in uSeconds = 1628480411445444
[2021-08-09 04:27:51.025 ip-172-16-11-227:19699 INFO trace_event_file_parser.py:197] Start time for e[...]<jupyter_text>The following code cell retrieves all recorded events from Amazon S3.<jupyter_code>framework_metrics_reader.refresh_event_file_list()
last_timestamp = framework_metrics_reader.get_timestamp_of_latest_available_file()
events = framework_metrics_reader.get_events(0, last_timestamp)

print(
    "Found",
    len(events),
    "recorded framework annotations. Latest event recorded ",
    timestamp_to_utc(last_timestamp / 1000000),
)<jupyter_output><empty_output><jupyter_text>Like before we can inspect the recorded events. Since we are reading framework metrics there is now a start and end time for each event.<jupyter_code>print(
    "Event name:",
    events[0].event_name,
    "\nStart time:",
    timestamp_to_utc(events[0].start_time / 1000000000),
    "\nEnd time:",
    timestamp_to_utc(events[0].end_time / 1000000000),
    "\nDuration:",
    events[0].duration,
    "nanosecond",
)<jupyter_output><empty_output><jupyter_text>3.4 Outliers in step durationStepHistogram creates histograms of step duration values. Significant outliers are indication of system bottlenecks. In contrast to SetpTimelineChart it helps identify clusters of step duration values. As a simple example: time spent during training phase (forward and backward pass) will likely be different to time spent during validation phase (forward pass), so we would expect at least two clusters.<jupyter_code>from smdebug.profiler.analysis.notebook_utils.step_histogram import StepHistogram

framework_metrics_reader.refresh_event_file_list()
step_histogram = StepHistogram(framework_metrics_reader)
step_histogram.plot()<jupyter_output><empty_output><jupyter_text>3.5 HeatmapThe following code cell creates a heatmap where each row corresponds to one metric (CPU core and GPU utilizations) and x-axis is the duration of the training job. It allows you to more easily spot CPU bottlenecks (utilization on GPU is low but a utilization of one or more cores is high).<jupyter_code>from smdebug.profiler.analysis.notebook_utils.heatmap import Heatmap

view_heatmap = Heatmap(
    system_metrics_reader,
    framework_metrics_reader,
    select_dimensions=["CPU", "GPU"],  # optional - comment this line out to see all dimensions.
    # select_events=["total"],                 # optional - comment this line out to see all events.
    plot_height=900,
)<jupyter_output><empty_output><jupyter_text>Section 4 - Analyze the profiler report generated by SageMaker Debugger.In this section we will analyze the report generated by debugger.  We will showcase a few sections of the report.  For complete details, please download the report from the S3 bucket and review.Also note that the exact details in the report generated for your training job may be different from what you see in this section.    4.1 View the location of the report generated.<jupyter_code>rule_output_path = pt_estimator.output_path + pt_estimator.latest_training_job.job_name + "/rule-output"
print(
    f"You will find the profiler report under `{rule_output_path}/` after the training has finished"
)<jupyter_output><empty_output><jupyter_text>To check if the report is generated, list directories and files recursively<jupyter_code>! aws s3 ls {rule_output_path} --recursive<jupyter_output><empty_output><jupyter_text>Download the report and rule output files recursively using `aws s3 cp` The following command saves all of the rule output files to the **ProfilerReport-1234567890** folder under your current working directory.<jupyter_code>! aws s3 cp {rule_output_path} ./ --recursive<jupyter_output><empty_output><jupyter_text>The following script automatically finds the **ProfilerReport** folder name and returns a link to the downloaded report.<jupyter_code>from IPython.display import FileLink

profiler_report_name = [
    rule["RuleConfigurationName"]
    for rule in pt_estimator.latest_training_job.rule_job_summary()
    if "Profiler" in rule["RuleConfigurationName"]
][0]
profiler_report_name
display(
    "Click link below to view the profiler report",
    FileLink(profiler_report_name + "/profiler-output/profiler-report.html"),
)<jupyter_output><empty_output><jupyter_text>For more information about how to find, download, and browse Debugger profiling reports, see SageMaker Debugger Profiling Report in the Amazon SageMaker Debugger developer guide.  Section 5 - Analyze and Implement recommendations from the report The **Rules Summary** section of the report aggregates all of the rule evaluation results, analysis, rule descriptions, and suggestions. The following table shows a summary of the executed profiler rules. The table is sorted by the rules that triggered most frequently. In training job this was the case for rule LowGPUUtilization. It has processed 1001 datapoints and triggered 8 times.You may see a different rule summary based on the data and the training configuration you use.  From the analysis so far and the top recommendations from the table above, there seems to be scope for improving resource utilization and make our training efficient.  Based on this change the training configuration settings and re run the training.  5.1 Implement recommendations from the reportNext, we will rerun the training job with the changed configuration. We will keep the same training cluster of 2 p3.ml.2xlarge and update the batch_size to 1024 increasing it from the default value of 64After second training job with the new settings is complete, there are new system metrics, framework metrics and a new report generated.<jupyter_code>##Change the batch size to 512 from the default 64
hyperparameters = {
    "batch_size": 1024,
    "nproc_per_node": 4,
    "nnodes": 2,
}
pt_estimator_updated_batch_size = PyTorch(
    entry_point="train_pytorch.py",
    source_dir="code",
    role=sagemaker.get_execution_role(),
    instance_count=instance_count,
    instance_type=train_instance_type,
    framework_version="1.6",
    py_version="py3",
    volume_size=1024,
    hyperparameters=hyperparameters,
    # Debugger-specific parameters
    profiler_config=profiler_config,
)
#Kickoff training
pt_estimator_updated_batch_size.fit({'train': train_input, 'test': validation_input})<jupyter_output><empty_output><jupyter_text>5.3 Compare the training jobsTo understand the impact of the training configuration changes, compare the report analysis from the two training jobs.  Repeat the process of analyzing the profiler report, implementing the recommendations and comparing with the previous run, till you are satisfied.<jupyter_code>rule_output_path = (
    pt_estimator_updated_batch_size.output_path + pt_estimator_updated_batch_size.latest_training_job.job_name + "/rule-output"
)
print(
    f"You will find the profiler report under {rule_output_path}/ after the training has finished"
)<jupyter_output><empty_output><jupyter_text>Download the new report and files recursively using `aws s3 cp`<jupyter_code>! aws s3 cp {rule_output_path} ./ --recursive<jupyter_output><empty_output><jupyter_text>Retrieve a file link to the new profiling report.<jupyter_code>from IPython.display import FileLink

profiler_report_name = [
    rule["RuleConfigurationName"]
    for rule in pt_estimator_updated_batch_size.latest_training_job.rule_job_summary()
    if "Profiler" in rule["RuleConfigurationName"]
][0]
profiler_report_name
display(
    "Click link below to view the profiler report",
    FileLink(profiler_report_name + "/profiler-output/profiler-report.html"),
)<jupyter_output><empty_output><jupyter_text>You should see that the training time has decreased for the training job executed with the updated configuration.  But there is still scope for improvement.  For the next round of triaining, lets keep the updated batch size and change the training instance to a less powerful instance "ml.p2.8xlarge"  Next round of training<jupyter_code>train_instance_type = "ml.p2.8xlarge"

pt_estimator_updated_batch_size_instance_size = PyTorch(
    entry_point="train_pytorch.py",
    source_dir="code",
    role=sagemaker.get_execution_role(),
    instance_count=instance_count,
    instance_type=train_instance_type,
    framework_version="1.6",
    py_version="py3",
    volume_size=1024,
    hyperparameters=hyperparameters,
    # Debugger-specific parameters
    profiler_config=profiler_config,
)
pt_estimator_updated_batch_size_instance_size.fit({'train': train_input, 'test': validation_input})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07859811246559183, 'fraction_numerical': 0.045320487613055446, 'mean_word_length': 5.8089022757697455, 'pattern_counts': {'":': 5, '<': 96, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '488421', 'license': 'MIT', 'n_tokens_mistral': 6579, 'n_tokens_neox': 6007, 'n_words': 1873, 'path': 'Chapter07/weather-prediction-debugger-profiler.ipynb', 'repo_name': 'bdonkey/Amazon-SageMaker-Best-Practices', 'size': '20344'}