<jupyter_start><jupyter_text>Distributed data parallel YOLOv5 training with PyTorch and SageMaker distributed[Amazon SageMaker's distributed library](https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training.html) can be used to train deep learning models faster and cheaper. The [data parallel](https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel.html) feature in this library (`smdistributed.dataparallel`) is a distributed data parallel training framework for PyTorch, TensorFlow, and MXNet.This notebook demonstrates how to use `smdistributed.dataparallel` with PyTorch(version 1.10.2) on [Amazon SageMaker](https://aws.amazon.com/sagemaker/) to train a YOLOv5 model on [BCCD dataset](https://public.roboflow.com/object-detection/bccd/4) using [Amazon FSx for Lustre file-system](https://aws.amazon.com/fsx/lustre/) as data source.The outline of steps is as follows:1. Stage BCCD dataset on [Amazon S3](https://aws.amazon.com/s3/)2. Create Amazon FSx Lustre file-system and import data into the file-system from S33. Build Docker training image and push it to [Amazon ECR](https://aws.amazon.com/ecr/)4. Configure data input channels for SageMaker5. Configure hyper-parameters6. Define training metrics7. Define training job and start training**NOTE:** With large training dataset, we recommend using [Amazon FSx](https://aws.amazon.com/fsx/) as the input file system for the SageMaker training job. FSx file input to SageMaker significantly cuts down training start up time on SageMaker because it avoids downloading the training data each time you start the training job (as done with S3 input for SageMaker training job) and provides good data read throughput.  Amazon SageMaker InitializationInitialize the notebook instance. Get the AWS Region and a SageMaker execution role. SageMaker roleThe following code cell defines `role` which is the IAM role ARN used to create and run SageMaker training and hosting jobs. This is the same IAM role used to create this SageMaker Notebook instance. `role` must have permission to create a SageMaker training job and host a model. For granular policies you can use to grant these permissions, see [Amazon SageMaker Roles](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html).As described above, since we will be using FSx, please make sure to attach `FSx Access` permission to this IAM role.<jupyter_code>%%time
! python3 -m pip install --upgrade sagemaker
import sagemaker
from sagemaker import get_execution_role
from sagemaker.estimator import Estimator
import boto3

sagemaker_session = sagemaker.Session()
bucket = sagemaker_session.default_bucket()

role = (
    get_execution_role()
)  # provide a pre-existing role ARN as an alternative to creating a new role
role_name = role.split(["/"][-1])
print(f"SageMaker Execution Role: {role}")
print(f"The name of the Execution role: {role_name[-1]}")

client = boto3.client("sts")
account = client.get_caller_identity()["Account"]
print(f"AWS account: {account}")

session = boto3.session.Session()
region = session.region_name
print(f"AWS region: {region}")<jupyter_output><empty_output><jupyter_text>To verify that the role above has required permissions:1. Go to the IAM console: https://console.aws.amazon.com/iam/home.2. Select **Roles**.3. Enter the role name in the search box to search for that role. 4. Select the role.5. Use the **Permissions** tab to verify this role has required permissions attached.  Prepare SageMaker Training ImagesSageMaker by default use the latest [Amazon Deep Learning Container Images (DLC)](https://github.com/aws/deep-learning-containers/blob/master/available_images.md) PyTorch training image. In this step, we use it as a base image and install additional dependencies required for training YOLOv5 model. Build and Push Docker Image to ECRRun the below command build the docker image and push it to ECR.<jupyter_code>image = "YOLOv5-smdataparallel-sagemaker"  # use any image name you want
tag = "pt1.10.2"  # use any tag name you want
import subprocess

subprocess.Popen(
    "aws ecr get-login-password --region "
    + region
    + " docker login --username AWS --password-stdin <PASSWORD>r."
    + region
    + ".amazonaws.com",
    shell=True,
)
!pygmentize ./Dockerfile
!pygmentize ./build_and_push.sh
%%time
! chmod +x build_and_push.sh; bash build_and_push.sh {region} {image} {tag}
import subprocess

subprocess.Popen(
    "rm -rf SMDDP-Examples && git clone https://github.com/HerringForks/SMDDP-Examples", shell=True
)<jupyter_output><empty_output><jupyter_text>Preparing FSx Input for SageMaker1. Download and prepare your training dataset on S3.2. [Create a FSx linked with your S3 bucket with training data](https://docs.aws.amazon.com/fsx/latest/LustreGuide/create-fs-linked-data-repo.html). Make sure to add an endpoint to your VPC allowing S3 access.3. [Configure your SageMaker training job to use FSx](https://aws.amazon.com/blogs/machine-learning/speed-up-training-on-amazon-sagemaker-using-amazon-efs-or-amazon-fsx-for-lustre-file-systems).  Important Caveats1. You need to use the same `subnet` and `vpc` and `security group` used with FSx when launching the SageMaker notebook instance. The same configurations will be used by your SageMaker training job.2. Make sure you [set appropriate inbound/output rules in the `security group`](https://docs.aws.amazon.com/fsx/latest/LustreGuide/limit-access-security-groups.html). Especially, opening up these ports is necessary for SageMaker to access the FSx file system in the training job. 3. Make sure `SageMaker IAM Role` used to launch this SageMaker training job has access to `AmazonFSx`.  SageMaker PyTorch Estimator function optionsIn the following code block, you can update the estimator function to use a different instance type, instance count, and distribution strategy. You're also passing in the training script you reviewed in the previous cell.**Instance types**1. ml.p3.16xlarge1. ml.p3dn.24xlarge [Recommended]1. ml.p4d.24xlarge [Recommended]**Instance count**You should use at least 2 instances, but you can also use 1 for testing this example.**Distribution strategy**Note that to use DDP mode, you need to update the "distribution" strategy.<jupyter_code>import os
from sagemaker.pytorch import PyTorch
instance_type = "ml.p4d.24xlarge"  # Other supported instance type: ml.p3.16xlarge, ml.p4d.24xlarge
instance_count = 8  # You can use 2, 4, 8 etc.
docker_image = f"{account}.dkr.ecr.{region}.amazonaws.com/{image}:{tag}"  # YOUR_ECR_IMAGE_BUILT_WITH_ABOVE_DOCKER_FILE
username = "AWS"
subnets = ["<subnet-xxx>"]  # Should be same as Subnet used for FSx. Example: subnet-0f9XXXX
security_group_ids = ["<sg-xxx>"]  # Should be same as Security group used for FSx. sg-03ZZZZZZ
job_name = "<job-xxx>"  # This job name is used as prefix to the sagemaker training job. Makes it easy for your look for your training job in SageMaker Training job console.
file_system_id = "<fs-xxx>"  # FSx file system ID with your training dataset. Example: 'fs-0bYYYYYY'
hyperparameters = {
    "data": "data/bccd.yaml",
    "cfg": "models/yolov5x.yaml",
    "epochs": 10,
    "batch-size": 1024,
}
estimator = PyTorch(
    entry_point="train.py",
    role=role,
    image_uri=docker_image,  # your image
    source_dir="./SMDDP-Examples/pytorch/object_detection/yolov5",
    instance_count=instance_count,
    instance_type=instance_type,
    framework_version="1.10.2",
    py_version="py38",
    sagemaker_session=sagemaker_session,
    hyperparameters=hyperparameters,
    subnets=subnets,
    security_group_ids=security_group_ids,
    debugger_hook_config=False,
    distribution={"smdistributed": {"dataparallel": {"enabled": True}}},
)
# Configure FSx Input for your SageMaker Training job

from sagemaker.inputs import FileSystemInput

file_system_directory_path = "<xxx>"  # NOTE: use the fsx path to your data
file_system_access_mode = "rw"
file_system_type = "FSxLustre"
train_fs = FileSystemInput(
    file_system_id=file_system_id,
    file_system_type=file_system_type,
    directory_path=file_system_directory_path,
    file_system_access_mode=file_system_access_mode,
)
data_channels = {"train": train_fs}
# Submit SageMaker training job
estimator.fit(inputs=data_channels, job_name=job_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09180842079864882, 'fraction_numerical': 0.010857763300760043, 'mean_word_length': 5.059941520467836, 'pattern_counts': {'":': 8, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '225249', 'license': 'Apache-2.0', 'n_tokens_mistral': 2623, 'n_tokens_neox': 2517, 'n_words': 869, 'path': 'training/distributed_training/pytorch/data_parallel/yolov5/yolov5.ipynb', 'repo_name': 'qidewenwhen/amazon-sagemaker-examples', 'size': '8298'}