<jupyter_start><jupyter_text>StepFunctions Data Science SDK for BYO Container Here we demonstrate how to use the StepFunction Data Science SDK to do an end-to-end data science workflow where you bring your own code and deploy a model on SageMaker.The Steps are as follows:1. Create a Lambda function which launches a CodeBuild job that launches the creation of your Docker container. (Steps for this are included separately)2. Launch the Lambda function as a Step Functions workflow. 3. Once the Docker container is built, launch a SageMaker training job using SF DS SDK.4. Use the DS SDK to deploy the trained model.  Create an execution role for Step FunctionsYou need an execution role so that you can create and execute workflows in Step Functions.1. Go to the IAM console2. Select Roles and then Create role.3. Under Choose the service that will use this role select Step Functions4. Choose Next until you can enter a Role name5. Enter a name such as StepFunctionsWorkflowExecutionRole and then select Create roleAttach a policy to the role you created. The following steps attach a policy that provides full access to Step Functions, however as a good practice you should only provide access to the resources you need.1. Under the Permissions tab, click Add inline policy2. Enter the following in the JSON tab```json{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "sagemaker:CreateTransformJob",                "sagemaker:DescribeTransformJob",                "sagemaker:StopTransformJob",                "sagemaker:CreateTrainingJob",                "sagemaker:DescribeTrainingJob",                "sagemaker:StopTrainingJob",                "sagemaker:CreateHyperParameterTuningJob",                "sagemaker:DescribeHyperParameterTuningJob",                "sagemaker:StopHyperParameterTuningJob",                "sagemaker:CreateModel",                "sagemaker:CreateEndpointConfig",                "sagemaker:CreateEndpoint",                "sagemaker:DeleteEndpointConfig",                "sagemaker:DeleteEndpoint",                "sagemaker:UpdateEndpoint",                "sagemaker:ListTags",                "lambda:InvokeFunction",                "sqs:SendMessage",                "sns:Publish",                "ecs:RunTask",                "ecs:StopTask",                "ecs:DescribeTasks",                "dynamodb:GetItem",                "dynamodb:PutItem",                "dynamodb:UpdateItem",                "dynamodb:DeleteItem",                "batch:SubmitJob",                "batch:DescribeJobs",                "batch:TerminateJob",                "glue:StartJobRun",                "glue:GetJobRun",                "glue:GetJobRuns",                "glue:BatchStopJobRun"            ],            "Resource": "*"        },        {            "Effect": "Allow",            "Action": [                "iam:PassRole"            ],            "Resource": "*",            "Condition": {                "StringEquals": {                    "iam:PassedToService": "sagemaker.amazonaws.com"                }            }        },        {            "Effect": "Allow",            "Action": [                "events:PutTargets",                "events:PutRule",                "events:DescribeRule"            ],            "Resource": [                "arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule",                "arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule",                "arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTuningJobsRule",                "arn:aws:events:*:*:rule/StepFunctionsGetEventsForECSTaskRule",                "arn:aws:events:*:*:rule/StepFunctionsGetEventsForBatchJobsRule"            ]        }    ]}```1. Choose Review policy and give the policy a name such as StepFunctionsWorkflowExecutionPolicy2. Choose Create policy. You will be redirected to the details page for the role.2. Note the Role ARN at the top of the Summary, as you'll need it later.  Import necessary libraries<jupyter_code># Import required libraries and upload the training data to S3
import boto3
import os
import numpy as np
import pandas as pd
import sagemaker
from sagemaker import get_execution_role
import time
import uuid



role = get_execution_role()
sess = sagemaker.Session()
bucket = sess.default_bucket() # or feel free to replace with a bucket of your choosing
WORK_DIRECTORY = 'PennFudanPed'
key = 'BYO-Mask-RCNN'
prefix = 'key/{}'.format(key, WORK_DIRECTORY)
print(bucket)
print(WORK_DIRECTORY)
print(key)
print(prefix)
# install StepFunctions SDK
import sys
!{sys.executable} -m pip install --upgrade stepfunctions<jupyter_output>Collecting stepfunctions
[?25l  Downloading https://files.pythonhosted.org/packages/a5/75/c3c9c9321c5fec4588625dc810ac56ff11c9babb10038ff5cf09d7f4b502/stepfunctions-1.0.0.6.tar.gz (58kB)
[K    100% |████████████████████████████████| 61kB 21.4MB/s ta 0:00:01
[?25hRequirement not upgraded as not directly required: sagemaker>=1.42.8 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from stepfunctions) (1.50.9.post0)
Requirement not upgraded as not directly required: boto3>=1.9.213 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from stepfunctions) (1.11.14)
Requirement not upgraded as not directly required: pyyaml in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from stepfunctions) (5.2)
Requirement not upgraded as not directly required: requests<3,>=2.20.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from sagemaker>=1.42.8->stepfunctions) (2.20.0)
Requirement not upgraded as not directly required: [...]<jupyter_text>Find your StepFunctionsWorkflowExecutionRole ARNGo back to IAM and find the ARN of the role where you created the inline policy above. You can find the full ARN in the top of that IAM page.<jupyter_code>import stepfunctions
import logging
from stepfunctions.steps import (LambdaStep, Retry, Catch, Fail, Chain, TrainingStep, ModelStep, EndpointConfigStep, EndpointStep)
from stepfunctions.workflow import Workflow
from stepfunctions.template.pipeline import TrainingPipeline
from stepfunctions.inputs import ExecutionInput

stepfunctions.set_stream_logger(level=logging.INFO)

# paste the StepFunctionsWorkflowExecutionRole ARN from above in "<>"
workflow_execution_role = "arn:aws:iam::166914487755:role/StepFunctionsWorkflowExecutionRole"<jupyter_output><empty_output><jupyter_text>Download the training data<jupyter_code>!rm -rf PennFudanPed*
!wget "https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip"
!unzip PennFudanPed.zip<jupyter_output>--2020-02-24 17:07:39--  https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip
Resolving www.cis.upenn.edu (www.cis.upenn.edu)... 192.168.3.11, fd00:c2b6:b24b:be67:2827:688d:e6a1:6a3b
Connecting to www.cis.upenn.edu (www.cis.upenn.edu)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 53723336 (51M) [application/zip]
Saving to: ‘PennFudanPed.zip’

PennFudanPed.zip    100%[===================>]  51.23M  4.71MB/s    in 9.0s    

2020-02-24 17:07:48 (5.66 MB/s) - ‘PennFudanPed.zip’ saved [53723336/53723336]

Archive:  PennFudanPed.zip
   creating: PennFudanPed/
  inflating: PennFudanPed/added-object-list.txt  
   creating: PennFudanPed/Annotation/
  inflating: PennFudanPed/Annotation/FudanPed00001.txt  
  inflating: PennFudanPed/Annotation/FudanPed00002.txt  
  inflating: PennFudanPed/Annotation/FudanPed00003.txt  
  inflating: PennFudanPed/Annotation/FudanPed00004.txt  
  inflating: PennFudanPed/Annotation/FudanPed00005.txt  
  inflating: PennFudanPed/Annotati[...]<jupyter_text>Upload the training dataset to Amazon S3After executing the cell below, wait until the `data_location` variable is printed, or until the S3 bucket is populated. You can check this by opening S3 in another browser tab.<jupyter_code>data_location = sess.upload_data(WORK_DIRECTORY, bucket=bucket, key_prefix=prefix)
print(data_location)<jupyter_output>s3://sagemaker-us-east-1-166914487755/key/BYO-Mask-RCNN<jupyter_text>Define Estimator and StepFunctions Workflow<jupyter_code>account = sess.boto_session.client('sts').get_caller_identity()['Account']
region = sess.boto_session.region_name

# Note that this image name will work below provided you have made the changes to the Environment variables in the Lambda
# defintion as suggested in the workshop Readme. If not please make those first.

image = '{}.dkr.ecr.{}.amazonaws.com/sm-container-maskrcnn:torch'.format(account, region) 

maskrcnn = sagemaker.estimator.Estimator(image,
                       role, 1, 'ml.p3.16xlarge', # --- ml.p3.2xlarge --- feel free to modify with your own. A cost estimate is provided in Readme.
                       output_path="s3://{}/{}/output".format(sess.default_bucket(), key),
                       sagemaker_session=sess)

maskrcnn.set_hyperparameters(num_epochs = 1,
                              num_classes = 2)

#maskrcnn.fit(os.path.dirname(data_location))<jupyter_output><empty_output><jupyter_text>Create StepFunction Pipeline Remember to replace the `FunctionName` variable below to the Lambda function you created through Cloudformation.<jupyter_code>lambda_state = LambdaStep(
    state_id="Calls CodeBuild to Build Container",
    parameters={  
        "FunctionName": "lambda-build-docker", #replace with the name of the Lambda function you created
        "Payload": {  
           "input": "HelloWorld"
        }
    }
)

lambda_state.add_retry(Retry(
    error_equals=["States.TaskFailed"],
    interval_seconds=15,
    max_attempts=2,
    backoff_rate=4.0
))

lambda_state.add_catch(Catch(
    error_equals=["States.TaskFailed"],
    next_step=Fail("LambdaTaskFailed")
))
execution_input = ExecutionInput(schema={
    'JobName': str, 
    'ModelName': str,
    'EndpointName': str
})
train_step = TrainingStep(
    'Train Step', 
    estimator=maskrcnn,
#    role=workflow_execution_role,
    data=os.path.dirname(data_location),
    job_name=execution_input['JobName']
)
model_step = ModelStep(
    'Save model',
    model=train_step.get_expected_model(),
    model_name=execution_input['ModelName'] 
)
endpoint_config_step = EndpointConfigStep(
    "Create Endpoint Config",
    endpoint_config_name=execution_input['ModelName'],
    model_name=execution_input['ModelName'],
    initial_instance_count=1,
    instance_type='ml.m5.large'
)
endpoint_step = EndpointStep(
    "Create Endpoint",
    endpoint_name=execution_input['EndpointName'],
    endpoint_config_name=execution_input['ModelName']
)
workflow_definition = Chain([
    lambda_state,
    train_step,
    model_step,
    endpoint_config_step,
    endpoint_step
])

# Next, we define the workflow
workflow = Workflow(
    name="MyWorkflow-BYOC-MaskRCNN-{}".format(uuid.uuid1().hex),
    definition=workflow_definition,
    role=workflow_execution_role
)
print(workflow.definition.to_json(pretty=True))
workflow.render_graph()
workflow.create()
execution = workflow.execute(
    inputs={
        'JobName': 'BYOC-Mask-RCNN-{}'.format(uuid.uuid1().hex), # Each Sagemaker Job requires a unique name
        'ModelName': 'BYOC-Mask-RCNN-{}'.format(uuid.uuid1().hex), # Each Model requires a unique name,
        'EndpointName': 'BYOC-Mask-RCNN-{}'.format(uuid.uuid1().hex) # Each Endpoint requires a unique name,
    }
)<jupyter_output>[32m[INFO] Workflow execution started successfully on AWS Step Functions.[0m<jupyter_text>Watch the progress of your workflow here<jupyter_code>from IPython.display import display, display_html
from stepfunctions.workflow import ExecutionStatus

while execution.status == ExecutionStatus.Running:
    display_html(execution.render_progress())
    time.sleep(30)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1040486856563266, 'fraction_numerical': 0.027977347646014707, 'mean_word_length': 3.605683145192682, 'pattern_counts': {'":': 17, '<': 27, '<?xml version=': 0, '>': 32, 'https://': 3, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '424505', 'license': 'MIT-0', 'n_tokens_mistral': 3856, 'n_tokens_neox': 3535, 'n_words': 870, 'path': 'StepFunctions_BYOC_Workflow.ipynb', 'repo_name': 'luisonthekeyboard/aws-stepfunctions-byoc-mlops-using-data-science-sdk', 'size': '11818'}