<jupyter_start><jupyter_text>Profiling TensorFlow Multi GPU Multi Node  Training Job with Amazon SageMaker DebuggerThis notebook will walk you through creating a TensorFlow training job with the SageMaker Debugger profiling feature enabled. It will create a multi GPU multi node training using Horovod.   (Optional) Install SageMaker and SMDebug Python SDKsTo use the new Debugger profiling features released in December 2020, ensure that you have the latest versions of SageMaker and SMDebug SDKs installed. Use the following cell to update the libraries and restarts the Jupyter kernel to apply the updates.<jupyter_code>import sys
import IPython
install_needed = False  # should only be True once
if install_needed:
    print("installing deps and restarting kernel")
    !{sys.executable} -m pip install -U sagemaker smdebug
    IPython.Application.instance().kernel.do_shutdown(True)<jupyter_output><empty_output><jupyter_text>1. Create a Training Job with Profiling EnabledYou will use the standard [SageMaker Estimator API for Tensorflow](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.htmltensorflow-estimator) to create training jobs. To enable profiling, create a `ProfilerConfig` object and pass it to the `profiler_config` parameter of the `TensorFlow` estimator.  Define parameters for distributed training This parameter tells SageMaker how to configure and run horovod. If you want to use more than 4 GPUs per node then change the process_per_host paramter accordingly.<jupyter_code>distributions = {
    "mpi": {
        "enabled": True,
        "processes_per_host": 4,
        "custom_mpi_options": "-verbose -x HOROVOD_TIMELINE=./hvd_timeline.json -x NCCL_DEBUG=INFO -x OMPI_MCA_btl_vader_single_copy_mechanism=none",
    }
}<jupyter_output><empty_output><jupyter_text>Configure rulesWe specify the following rules:- loss_not_decreasing: checks if loss is decreasing and triggers if the loss has not decreased by a certain persentage in the last few iterations- LowGPUUtilization: checks if GPU is under-utilizated - ProfilerReport: runs the entire set of performance rules and create a final output report with further insights and recommendations.<jupyter_code>from sagemaker.debugger import Rule, ProfilerRule, rule_configs

rules = [
    Rule.sagemaker(rule_configs.loss_not_decreasing()),
    ProfilerRule.sagemaker(rule_configs.LowGPUUtilization()),
    ProfilerRule.sagemaker(rule_configs.ProfilerReport()),
]<jupyter_output><empty_output><jupyter_text>Specify a profiler configurationThe following configuration will capture system metrics at 500 milliseconds. The system metrics include utilization per CPU, GPU, memory utilization per CPU, GPU as well I/O and network.Debugger will capture detailed profiling information from step 5 to step 15. This information includes Horovod metrics, dataloading, preprocessing, operators running on CPU and GPU.<jupyter_code>from sagemaker.debugger import ProfilerConfig, FrameworkProfile

profiler_config = ProfilerConfig(
    system_monitor_interval_millis=500,
    framework_profile_params=FrameworkProfile(
        local_path="/opt/ml/output/profiler/", start_step=5, num_steps=10
    ),
)<jupyter_output><empty_output><jupyter_text>Get the image URIThe image that we will is dependent on the region that you are running this notebook in.<jupyter_code>import boto3

session = boto3.session.Session()
region = session.region_name

image_uri = f"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04"<jupyter_output><empty_output><jupyter_text>Define estimatorTo enable profiling, you need to pass the Debugger profiling configuration (`profiler_config`), a list of Debugger rules (`rules`), and the image URI (`image_uri`) to the estimator. Debugger enables monitoring and profiling while the SageMaker estimator requests a training job.<jupyter_code>import sagemaker
from sagemaker.tensorflow import TensorFlow

estimator = TensorFlow(
    role=sagemaker.get_execution_role(),
    image_uri=image_uri,
    instance_count=2,
    instance_type="ml.p3.8xlarge",
    entry_point="tf-hvd-train.py",
    source_dir="entry_point",
    profiler_config=profiler_config,
    distribution=distributions,
    rules=rules,
)<jupyter_output><empty_output><jupyter_text>Start training jobThe following `estimator.fit()` with `wait=False` argument initiates the training job in the background. You can proceed to run the dashboard or analysis notebooks.<jupyter_code>estimator.fit(wait=False)<jupyter_output><empty_output><jupyter_text>2.  Analyze Profiling DataCopy outputs of the following cell (`training_job_name` and `region`) to run the analysis notebooks `profiling_generic_dashboard.ipynb`, `analyze_performance_bottlenecks.ipynb`, and `profiling_interactive_analysis.ipynb`.<jupyter_code>training_job_name = estimator.latest_training_job.name
print(f"Training jobname: {training_job_name}")
print(f"Region: {region}")<jupyter_output><empty_output><jupyter_text>While the training is still in progress you can visualize the performance data in SageMaker Studio or in the notebook.Debugger provides utilities to plot system metrics in form of timeline charts or heatmaps. Checkout out the notebook [profiling_interactive_analysis.ipynb](analysis_tools/profiling_interactive_analysis.ipynb) for more details. In the following code cell we plot the total CPU and GPU utilization as timeseries charts. To visualize other metrics such as I/O, memory, network you simply need to extend the list passed to `select_dimension` and `select_events`.  Install the SMDebug client library to use Debugger analysis tools<jupyter_code>import pip


def import_or_install(package):
    try:
        __import__(package)
    except ImportError:
        pip.main(["install", package])


import_or_install("smdebug")<jupyter_output><empty_output><jupyter_text>Access the profiling data using the SMDebug `TrainingJob` utility class<jupyter_code>from smdebug.profiler.analysis.notebook_utils.training_job import TrainingJob

tj = TrainingJob(training_job_name, region)
tj.wait_for_sys_profiling_data_to_be_available()<jupyter_output><empty_output><jupyter_text>Plot time line chartsThe following code shows how to use the SMDebug `TrainingJob` object, refresh the object if new event files are available, and plot time line charts of CPU and GPU usage.<jupyter_code>from smdebug.profiler.analysis.notebook_utils.timeline_charts import TimelineCharts

system_metrics_reader = tj.get_systems_metrics_reader()
system_metrics_reader.refresh_event_file_list()

view_timeline_charts = TimelineCharts(
    system_metrics_reader,
    framework_metrics_reader=None,
    select_dimensions=["CPU", "GPU"],
    select_events=["total"],
)<jupyter_output><empty_output><jupyter_text>3. Download Debugger Profiling Report The `ProfilerReport()` rule creates an html report `profiler-report.html` with a summary of builtin rules and recommenades of next steps. You can find this report in your S3 bucket.<jupyter_code>rule_output_path = estimator.output_path + estimator.latest_training_job.job_name + "/rule-output"
print(f"You will find the profiler report in {rule_output_path}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06969654981294167, 'fraction_numerical': 0.007066648191769434, 'mean_word_length': 6.365306122448979, 'pattern_counts': {'":': 4, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '1117', 'license': 'Apache-2.0', 'n_tokens_mistral': 2163, 'n_tokens_neox': 2087, 'n_words': 659, 'path': 'sagemaker-debugger/tensorflow_profiling/tf-resnet-profiling-multi-gpu-multi-node.ipynb', 'repo_name': 'Amirosimani/amazon-sagemaker-examples', 'size': '7217'}