<jupyter_start><jupyter_text>SageMaker Model ProfilingHere we will see how we can use Sagemaker Profiling to see our training system metrics as well as generate a Profiler Report.First we will need to install `smdebug`.  `pytorch_cifar_profiling.py`   Click here to see the full code for the script ```pythonimport argparseimport timeimport numpy as npimport torchimport torch.nn as nnimport torch.optim as optimimport torchvisionimport torchvision.models as modelsimport torchvision.transforms as transformsfrom smdebug import modesfrom smdebug.profiler.utils import str2boolfrom smdebug.pytorch import get_hookdef train(args, net, device):    hook = get_hook(create_if_not_exists=True)    batch_size = args.batch_size    epoch = args.epoch    transform_train = transforms.Compose(        [            transforms.RandomCrop(32, padding=4),            transforms.RandomHorizontalFlip(),            transforms.ToTensor(),            transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))        ]    )    transform_valid = transforms.Compose(        [            transforms.ToTensor(),            transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))        ]    )    trainset = torchvision.datasets.CIFAR10(        root="./data", train=True, download=True, transform=transform_train    )    trainloader = torch.utils.data.DataLoader(        trainset,        batch_size=batch_size,        shuffle=True    )    validset = torchvision.datasets.CIFAR10(        root="./data", train=False, download=True, transform=transform_valid    )    validloader = torch.utils.data.DataLoader(        validset,        batch_size=batch_size,        shuffle=False    )    loss_optim = nn.CrossEntropyLoss()    optimizer = optim.SGD(net.parameters(), lr=1.0, momentum=0.9)    epoch_times = []    if hook:        hook.register_loss(loss_optim)     train the model    for i in range(epoch):        print("START TRAINING")        if hook:            hook.set_mode(modes.TRAIN)        start = time.time()        net.train()        train_loss = 0        for _, (inputs, targets) in enumerate(trainloader):            inputs, targets = inputs.to(device), targets.to(device)            optimizer.zero_grad()            outputs = net(inputs)            loss = loss_optim(outputs, targets)            loss.backward()            optimizer.step()            train_loss += loss.item()        print("START VALIDATING")        if hook:            hook.set_mode(modes.EVAL)        net.eval()        val_loss = 0        with torch.no_grad():            for _, (inputs, targets) in enumerate(validloader):                inputs, targets = inputs.to(device), targets.to(device)                outputs = net(inputs)                loss = loss_optim(outputs, targets)                val_loss += loss.item()        epoch_time = time.time() - start        epoch_times.append(epoch_time)        print(            "Epoch %d: train loss %.3f, val loss %.3f, in %.1f sec"            % (i, train_loss, val_loss, epoch_time)        )     calculate training time after all epoch    p50 = np.percentile(epoch_times, 50)    return p50def main():    parser = argparse.ArgumentParser()    parser.add_argument("--batch_size", type=int, default=128)    parser.add_argument("--epoch", type=int, default=1)    parser.add_argument("--gpu", type=str2bool, default=True)    parser.add_argument("--model", type=str, default="resnet50")    opt = parser.parse_args()    for key, value in vars(opt).items():        print(f"{key}:{value}")     create model    net = models.__dict__[opt.model](pretrained=True)    if opt.gpu == 1:        device = torch.device("cuda")    else:        device = torch.device("cpu")    net.to(device)     Start the training.    median_time = train(opt, net, device)    print("Median training time per Epoch=%.1f sec" % median_time)if __name__ == "__main__":    main()```<jupyter_code># install dependencies
!pip install smdebug<jupyter_output>Requirement already satisfied: smdebug in /opt/conda/lib/python3.6/site-packages (1.0.9)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from smdebug) (21.3)
Requirement already satisfied: pyinstrument>=3.1.3 in /opt/conda/lib/python3.6/site-packages (from smdebug) (3.4.2)
Requirement already satisfied: numpy>=1.16.0 in /opt/conda/lib/python3.6/site-packages (from smdebug) (1.19.1)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from smdebug) (3.19.1)
Requirement already satisfied: boto3>=1.10.32 in /opt/conda/lib/python3.6/site-packages (from smdebug) (1.20.24)
Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /opt/conda/lib/python3.6/site-packages (from boto3>=1.10.32->smdebug) (0.5.0)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3>=1.10.32->smdebug) (0.10.0)
Requirement already satisfied: botocore<1.24.0,>=1.23.24 in /opt/conda/lib/pytho[...]<jupyter_text>Next we will need to specify the metrics that we want to track and create the profiler rules. Below you can see that I have specified to track 3 metrics: Loss not decreasing, Low GPU Utilization and also to generate the profiler report. I have also specified that these metrics should be tracked every 500 milliseconds.<jupyter_code>from sagemaker.debugger import Rule, ProfilerRule, rule_configs

rules = [
    Rule.sagemaker(rule_configs.loss_not_decreasing()),
    ProfilerRule.sagemaker(rule_configs.LowGPUUtilization()),
    ProfilerRule.sagemaker(rule_configs.ProfilerReport()),
]
from sagemaker.debugger import ProfilerConfig, FrameworkProfile

profiler_config = ProfilerConfig(
    system_monitor_interval_millis=500, framework_profile_params=FrameworkProfile(num_steps=10)
)<jupyter_output><empty_output><jupyter_text>Now that we have specified our profiler rules, we can create our hyperparameter dict and estimator to perform training. We will also need to specify our profiler rules and configs in the estimator.<jupyter_code>hyperparameters = {
    "batch_size": 2048,
    "gpu": True,
    "epoch": 2,
    "model": "resnet50",
}
import sagemaker
from sagemaker.pytorch import PyTorch

estimator = PyTorch(
    role=sagemaker.get_execution_role(),
    instance_count=1,
    instance_type="ml.p3.2xlarge",
    source_dir="scripts",
    entry_point="pytorch_cifar_profiling.py",
    framework_version="1.8",
    py_version="py36",
    hyperparameters=hyperparameters,
    profiler_config=profiler_config,
    rules=rules,
)
estimator.fit(wait=True)
import boto3

session = boto3.session.Session()
region = session.region_name

training_job_name = estimator.latest_training_job.name
print(f"Training jobname: {training_job_name}")
print(f"Region: {region}")<jupyter_output>Training jobname: pytorch-training-2022-01-09-20-07-45-811
Region: us-east-1<jupyter_text>Checking System UtilizationBelow is some boilerplate code to get the training job object using the training job name and display the system metrics. The plots may not show up in the classroom, but it will show up when you train the model in SageMaker Studio.<jupyter_code>from smdebug.profiler.analysis.notebook_utils.training_job import TrainingJob

tj = TrainingJob(training_job_name, region)
tj.wait_for_sys_profiling_data_to_be_available()
from smdebug.profiler.analysis.notebook_utils.timeline_charts import TimelineCharts

system_metrics_reader = tj.get_systems_metrics_reader()
system_metrics_reader.refresh_event_file_list()

view_timeline_charts = TimelineCharts(
    system_metrics_reader,
    framework_metrics_reader=None,
    select_dimensions=["CPU", "GPU"],
    select_events=["total"],
)<jupyter_output><empty_output><jupyter_text>Profiler ReportNext we will fetch the profiler report from the S3 bucket where it was stored and display it. The profiler report may not display in the notebook, but you can take a look at it from the ProfilerReport folder.<jupyter_code>rule_output_path = estimator.output_path + estimator.latest_training_job.job_name + "/rule-output"
print(f"You will find the profiler report in {rule_output_path}")
! aws s3 ls {rule_output_path} --recursive
! aws s3 cp {rule_output_path} ./ --recursive
import os

# get the autogenerated folder name of profiler report
profiler_report_name = [
    rule["RuleConfigurationName"]
    for rule in estimator.latest_training_job.rule_job_summary()
    if "Profiler" in rule["RuleConfigurationName"]
][0]
import IPython

IPython.display.HTML(filename=profiler_report_name + "/profiler-output/profiler-report.html")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10528160785230194, 'fraction_numerical': 0.027342837111474644, 'mean_word_length': 3.9761627906976744, 'pattern_counts': {'":': 5, '<': 22, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '567320', 'license': 'MIT', 'n_tokens_mistral': 2750, 'n_tokens_neox': 2656, 'n_words': 660, 'path': 'machine-learning-engineering/sagemaker-debugger-profiler/sample-profiler/model_profiling.ipynb', 'repo_name': 'edgarinvillegas/my-ml-playground', 'size': '8558'}