<jupyter_start><jupyter_text>Customer Churn Prediction with XGBoost_**Using Gradient Boosted Trees to Predict Mobile Customer Departure**_------ Contents1. [Background](Background)1. [Setup](Setup)1. [Data](Data)1. [Train](Train)1. [Host](Host_The_Model)1. [Evaluate](Evaluate)1. [(Must) Clean-up](clean_up)1. [Predict in batch mode using Batch Transform](batch_transform)1. [(Optional) Bonus 1 - Tune the model using different/additional hyperparameters](bonus_1)1. [(Optional) Bonus 2 - Use SageMaker automatic model tuning](bonus_2)--- Background_This notebook has been adapted from an [AWS blog post](https://aws.amazon.com/blogs/ai/predicting-customer-churn-with-amazon-machine-learning/)_Losing customers is costly for any business.  Identifying unhappy customers early on gives you a chance to offer them incentives to stay.  This notebook describes using machine learning (ML) for the automated identification of unhappy customers, also known as customer churn prediction. ML models rarely give perfect predictions though, so this notebook is also about how to incorporate the relative costs of prediction mistakes when determining the financial outcome of using ML.We use an example of churn that is familiar to all of us–leaving a mobile phone operator.  Seems like I can always find fault with my provider du jour! And if my provider knows that I’m thinking of leaving, it can offer timely incentives–I can always use a phone upgrade or perhaps have a new feature activated–and I might just stick around. Incentives are often much more cost effective than losing and reacquiring a customer.--- SetupSpecify your name (will be displayed in the class live leaderboard, So nothing too funny plz :)<jupyter_code>!pip install sagemaker --upgrade
my_name = YOUR_NAME_HERE_AS_STRING

#### 

from classroom import init_classroom, report_progress, report_algo_objective
init_classroom(my_name, call_home=True)

report_progress(0)<jupyter_output><empty_output><jupyter_text>Run the below cell to Specify:- The S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting. In this case we'll use Sagemaker's default bucket.- The IAM role arn used to give training and hosting access to your data.<jupyter_code># Define IAM role
import boto3
import re
from sagemaker import get_execution_role
import sagemaker


role = get_execution_role()
session = sagemaker.Session()
bucket = session.default_bucket()
prefix = 'sagemaker/DEMO-xgboost-churn'
print(f'Using bucket: {bucket}')<jupyter_output><empty_output><jupyter_text>Next, we'll import the Python libraries we'll need for the remainder of the exercise.In particular note:1. [Pandas](https://pandas.pydata.org/) - A Python library for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.2. [Numpy](http://www.numpy.org/) - A Python library that adds support for large, multi-dimensional arrays and matrices.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import io
import os
import sys
import time
import json
from IPython.display import display
from time import strftime, gmtime
import sagemaker<jupyter_output><empty_output><jupyter_text>--- DataMobile operators have historical records on which customers ultimately ended up churning and which continued using the service. We can use this historical information to construct an ML model of one mobile operator’s churn using a process called training. After training the model, we can pass the profile information of an arbitrary customer (the same profile information that we used to train the model) to the model, and have the model predict whether this customer is going to churn. Of course, we expect the model to make mistakes–after all, predicting the future is tricky business! But I’ll also show how to deal with prediction errors.The dataset we use is publicly available and was mentioned in the book [Discovering Knowledge in Data](https://www.amazon.com/dp/0470908742/) by <NAME>. It is attributed by the author to the University of California Irvine Repository of Machine Learning Datasets.  Let's download and read that dataset in now:<jupyter_code>!aws s3 cp s3://floor28/churn.zip ./
!unzip -o churn.zip
!ls -l 'Data sets/churn.txt'<jupyter_output><empty_output><jupyter_text>---Load into a Pandas Dataframe:  churn.txt is a CSV file, load it into a Pandas Dataframe by using the relevant [Pandas method for reading input](https://pandas.pydata.org/pandas-docs/stable/api.html).<jupyter_code>churn = pd.REPLACE_ME('./Data sets/churn.txt') # Use pandas to read in CSV format
pd.set_option('display.max_columns', 500)
report_progress(1)
churn.head(5) # Print first five records<jupyter_output><empty_output><jupyter_text>By modern standards, it’s a relatively small dataset, with only 3,333 records, where each record uses 21 attributes to describe the profile of a customer of an unknown US mobile operator. The attributes are:- `State`: the US state in which the customer resides, indicated by a two-letter abbreviation; for example, OH or NJ- `Account Length`: the number of days that this account has been active- `Area Code`: the three-digit area code of the corresponding customer’s phone number- `Phone`: the remaining seven-digit phone number- `Int’l Plan`: whether the customer has an international calling plan: yes/no- `VMail Plan`: whether the customer has a voice mail feature: yes/no- `VMail Message`: presumably the average number of voice mail messages per month- `Day Mins`: the total number of calling minutes used during the day- `Day Calls`: the total number of calls placed during the day- `Day Charge`: the billed cost of daytime calls- `Eve Mins, Eve Calls, Eve Charge`: the billed cost for calls placed during the evening- `Night Mins`, `Night Calls`, `Night Charge`: the billed cost for calls placed during nighttime- `Intl Mins`, `Intl Calls`, `Intl Charge`: the billed cost for international calls- `CustServ Calls`: the number of calls placed to Customer Service- `Churn?`: whether the customer left the service: true/falseThe last attribute, `Churn?`, is known as the target attribute–the attribute that we want the ML model to predict.  Because the target attribute is binary, our model will be performing binary prediction, also known as binary classification. ---As Data scientists, before plugging our data into an ML algorithm, we must first explore our data to understand:1. Which features are categorical and which are numeric. Which features categorical Numpy wrongly classified as numeric.2. Each feature values distribution and cardinality.3. Which features we should drop from the dataset because we can easily see they would not contribute to the model, or because they are highly correlated with other features.You should really explore all features, but in this notebook, we decided to focus on specific features we know are interesting.  Let's start exploring:<jupyter_code>churn.State.value_counts(sort=True).plot.pie()
plt.show()<jupyter_output><empty_output><jupyter_text>Note `State` appears to be quite evenly distributed<jupyter_code>def crosstab(column):
    display(pd.crosstab(index=churn[column], columns='% observations', normalize='columns'))
crosstab('Phone')<jupyter_output><empty_output><jupyter_text>`Phone` takes on too many unique values to be of any practical use. It's possible parsing out the prefix could have some value, but without more context on how these are allocated, we should avoid using it.  Let's drop the `Phone` column:<jupyter_code>churn = churn.drop('Phone', axis=1)
churn['Churn?'].value_counts(sort=True).plot.pie(autopct='%1.f%%')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see only 14% of customers churned, so there is some class imabalance, but nothing extreme.<jupyter_code># Histograms for each numeric features
#display(churn.describe())
%matplotlib inline
hist = churn.hist(bins=30, sharey=True, figsize=(10, 10))<jupyter_output><empty_output><jupyter_text>Most of the numeric features are surprisingly nicely distributed, with many showing bell-like gaussianity.  `VMail Message` being a notable exception (and `Area Code` showing up as a feature we should convert to non-numeric).  Let's convert `Area Code` to non-numeric:<jupyter_code>churn['Area Code'] = churn['Area Code'].astype(object)<jupyter_output><empty_output><jupyter_text>Next let's look at the relationship between each of the features and our target variable.<jupyter_code>for column in churn.select_dtypes(include=['object']).columns:
    if column != 'Churn?':
        display(pd.crosstab(index=churn[column], columns=churn['Churn?'], normalize='index'))

for column in churn.select_dtypes(exclude=['object']).columns:
    print(column)
    hist = churn[[column, 'Churn?']].hist(by='Churn?', bins=30)
    plt.show()<jupyter_output><empty_output><jupyter_text>Interestingly we see that churners appear:- Fairly evenly distributed geographically- More likely to have an international plan- Less likely to have a voicemail plan- To exhibit some bimodality in daily minutes (either higher or lower than the average for non-churners)- To have a larger number of customer service calls (which makes sense as we'd expect customers who experience lots of problems may be more likely to churn)In addition, we see that churners take on very similar distributions for features like `Day Mins` and `Day Charge`.  That's not surprising as we'd expect minutes spent talking to correlate with charges.  Let's dig deeper into the relationships between our features.<jupyter_code># Show the correlation between each possible features pair as a scatter plot
pd.plotting.scatter_matrix(churn, figsize=(16, 16))
plt.show()<jupyter_output><empty_output><jupyter_text>We see several features that essentially have 100% correlation with one another.  Including these feature pairs in some machine learning algorithms can create catastrophic problems, while in others it will only introduce minor redundancy and bias.  Let's remove one feature from each of the highly correlated pairs: Day Charge from the pair with Day Mins, Night Charge from the pair with Night Mins, Intl Charge from the pair with Intl Mins:<jupyter_code>churn = churn.drop(['Day Charge', 'Eve Charge', 'Night Charge', 'Intl Charge'], axis=1)<jupyter_output><empty_output><jupyter_text>Now that we've cleaned up our dataset, let's determine which algorithm to use.  As mentioned above, there appear to be some variables where both high and low (but not intermediate) values are predictive of churn.  In order to accommodate this in an algorithm like linear regression, we'd need to generate polynomial (or bucketed) terms.  Instead, let's attempt to model this problem using gradient boosted trees.  Amazon SageMaker provides an XGBoost container that we can use to train in a managed, distributed setting, and then host as a real-time prediction endpoint.  XGBoost uses gradient boosted trees which naturally account for non-linear relationships between features and the target variable, as well as accommodating complex interactions between features.Amazon SageMaker XGBoost can train on data in either a CSV or LibSVM format.  For this example, we'll stick with CSV.  It should:- Have the predictor variable in the first column- Not have a header rowBut first, let's convert our categorical features into numeric features, as the algorithm expects numerical values only.This means adding each possible catagorical value as a new colume.Replace `REPLACE_ME` with the pandas method that Converts categorical variable into dummy/indicator variables.<jupyter_code>model_data = pd.REPLACE_ME(churn)
# creates a new dataframe with the 'Churn?_True.' colume being the first column, the trailed by 
# all other columes except 'Churn?_False.' and 'Churn?_True.'.
model_data = pd.concat([model_data['Churn?_True.'], 
                        model_data.drop(['Churn?_False.', 'Churn?_True.'], axis=1)], axis=1)
model_data.head(5) # notice we now have many more columes generated to reperesent all categorical values<jupyter_output><empty_output><jupyter_text>And now let's split the data into training, validation, and test sets.  This will help prevent us from overfitting the model, and allow us to test the models accuracy on data it hasn't already seen.<jupyter_code>train_data, validation_data, test_data = np.split(
    model_data.sample(frac=1, random_state=1729), 
    [int(0.7 * len(model_data)), 
     int(0.9 * len(model_data))])
train_data.to_csv('train.csv', header=False, index=False)
validation_data.to_csv('validation.csv', header=False, index=False)<jupyter_output><empty_output><jupyter_text>Now we'll upload these files to S3.<jupyter_code>boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train/train.csv')).upload_file('train.csv')
boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'validation/validation.csv')).upload_file('validation.csv')<jupyter_output><empty_output><jupyter_text>--- TrainAmazon SageMaker provides several [built-in machine learning algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html) that you can use for a variety of problem types. Amazon SageMaker algorithms are packaged as Docker images. This gives you the flexibility to use almost any algorithm code with Amazon SageMaker, regardless of implementation language, dependent libraries, frameworks, and so on.Each built-in algorithm packed in its own Docker image found in the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html). We'll specify the relevant container image for each training job.  Moving onto training, first we'll need to specify the image for the XGBoost algorithm container.In the next cell replace **algorithm_name** with the relevant algorithm name from the "Training Image and Inference Image Registry Path" colume in this [table](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html). Enter the algoritm name in lower case and without the trailing colon.<jupyter_code>report_progress(2)
from sagemaker.amazon.amazon_estimator import get_image_uri
container = get_image_uri(boto3.Session().region_name, 'replace_me_with_algorithm_name')
display(container)<jupyter_output><empty_output><jupyter_text>[SageMaker Python SDK](https://sagemaker.readthedocs.io/en/latest/)  is an open source library for training and deploying machine-learned models on Amazon SageMaker. The SDK provides several high-level abstractions for working with Amazon SageMaker. These are:* [Estimators](https://sagemaker.readthedocs.io/en/latest/estimators.html): Encapsulate training on SageMaker.* [Models](https://sagemaker.readthedocs.io/en/latest/model.html): Encapsulate a trained ML model. Can be deployed to an endpoint.* [Predictors](https://sagemaker.readthedocs.io/en/latest/predictors.html): Provide real-time inference and transformation using Python data-types against a SageMaker endpoint.* [Session](https://sagemaker.readthedocs.io/en/latest/session.html): Provides a collection of methods for working with SageMaker resources.We'll start by creating the [xgboost Estimator](https://sagemaker.readthedocs.io/en/latest/estimators.html). The mandatory paramters are: image_name (str), role (str), sagemaker_session (session), train_instance_type (str), and train_instance_count (int).For this training job, provide these parameters: **image_name = container, role=role, train_instance_count = 1, train_instance_type = 'ml.m4.xlarge', sagemaker_session = session**.<jupyter_code># Creating the SageMaker Estimator object
xgb = sagemaker.estimator.Estimator(image_name = ...,
                                    role = ..., 
                                    train_instance_count = ..., 
                                    train_instance_type = ...,
                                    output_path = 's3://{}/{}/output'.format(bucket, prefix),
                                    sagemaker_session = ...)<jupyter_output><empty_output><jupyter_text>An ML algorithm is configured and tuned with specific hyperparameters, the hyperparameters changes the way the algorithm works ([what is a hyperpramaeter?](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning%29)).The XGBoost hyperparamaters are described in the [XGBoost documentation](https://xgboost.readthedocs.io/en/latest/parameter.html).For this example, the required hyperparameters for the XGBoost algorithm are:* `objective` - Specifies the learning task and the corresponding learning objective. please use **binary:logistic** for binary classification task.  * `num_round` - Controls the number of boosting rounds. This is essentially the subsequent models that are trained using the residuals of previous iterations. More rounds should produce a better fit on the training data, but can be computationally expensive or lead to overfitting.A few other key hyperparameters are:* `max_depth` - Controls how deep each tree within the algorithm can be built. Deeper trees can lead to better fit, but are more computationally expensive and can lead to overfitting. There is typically some trade-off in model performance that needs to be explored between a large number of shallow trees and a smaller number of deeper trees.* `subsample` controls - Sampling of the training data. This technique can help reduce overfitting, but setting it too low can also starve the model of data.* `eta` - Controls how aggressive each round of boosting is. Larger values lead to more conservative boosting.* `gamma` - Controls how aggressively trees are grown. Larger values lead to more conservative models.Use the [xgb.set_hyperparameters](https://sagemaker.readthedocs.io/en/latest/estimators.html) to set the hyperparameters value you choose after reading the documentation.<jupyter_code># set the hyperparameters
xgb.set_hyperparameters(objective=...,
                        num_round=...
)<jupyter_output><empty_output><jupyter_text>Since we're training with CSV file format, we'll create [`s3_input`](https://sagemaker.readthedocs.io/en/latest/session.html?highlight=sagemaker.session.s3_input) objects that our training function can use as a pointer to the files type and location in S3.Run the following for the training data input and the validation data sets:<jupyter_code># Configuring the data inputs
s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket, prefix), content_type='csv')
s3_input_validation = sagemaker.s3_input(s3_data='s3://{}/{}/validation'.format(bucket, prefix), content_type='csv')<jupyter_output><empty_output><jupyter_text>Finally we are ready to train.To train use the [xgb.fit()](https://sagemaker.readthedocs.io/en/latest/estimators.html) function.<jupyter_code># Training the model
xgb.fit({'train': s3_input_train, 'validation': s3_input_validation})
report_progress(3)<jupyter_output><empty_output><jupyter_text>--- You've completed training - pay attention to the final validation-error. A lower value is better.  Let's post this value to the classroom leaderboard:<jupyter_code>df = xgb.training_job_analytics.dataframe()
validation_error = df[df.metric_name.isin(['validation:error'])].at[0, "value"]

display('validation_error='+str(validation_error))

#import importlib, classroom
#importlib.reload(classroom)
#init_classroom(my_name, call_home=True)
#from classroom import report_algo_objective

report_algo_objective(validation_error, xgb.hyperparam_dict)<jupyter_output><empty_output><jupyter_text>--- Host The ModelNow that we've trained the algorithm, let's create a model and deploy it to a hosted endpoint.We'll do this using [estimator](https://sagemaker.readthedocs.io/en/latest/estimators.html) `deploy()` method.Provide these parameters: **initial_instance_count = 1** and **instance_type = 'ml.m4.xlarge'**<jupyter_code># Deploy the model
xgb_predictor = xgb.deploy(...)<jupyter_output><empty_output><jupyter_text>--- EvaluateNow that we have a hosted endpoint running, we can make real-time predictions from our model very easily, simply by making an http POST request.  But first, we'll need to setup serializers and deserializers for passing our `test_data` NumPy arrays to the model behind the endpoint.<jupyter_code>from sagemaker.predictor import csv_serializer
xgb_predictor.content_type = 'text/csv'
xgb_predictor.serializer = csv_serializer
xgb_predictor.deserializer = None<jupyter_output><empty_output><jupyter_text>Now, we'll use a simple function to:1. Loop over our test dataset1. Split it into mini-batches of rows 1. Convert those mini-batchs to CSV string payloads1. Retrieve mini-batch predictions by invoking the XGBoost endpoint1. Collect predictions and convert from the CSV output, our model provides, into a NumPy array<jupyter_code>def predict(data, rows=500):
    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))
    predictions = ''
    for array in split_array:
        current_predictions = xgb_predictor.predict(please complete...).decode('utf-8')
        predictions = ','.join([predictions, current_predictions])

    return np.fromstring(predictions[1:], sep=',') # csv to numpy array

predictions = predict(test_data.as_matrix()[:, 1:])

assert(len(predictions)==334) # Just checking<jupyter_output><empty_output><jupyter_text>There are many ways to compare the performance of a machine learning model, but let's start by simply by comparing actual to predicted values.  In this case, we're simply predicting whether the customer churned (`1`) or not (`0`), which produces a simple [confusion matrix](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/).<jupyter_code>report_progress(4)
confusion_matrix = pd.crosstab(index=test_data.iloc[:, 0], columns=np.round(predictions), rownames=['actual'], colnames=['predictions'], margins='true')
display(confusion_matrix)<jupyter_output><empty_output><jupyter_text>_Note, due to randomized elements of the algorithm, your results may differ slightly._Of the 48 churners, we've correctly predicted ~39 of them (_True Positives (TP)_). And, we incorrectly predicted 4 customers would churn who then ended up not doing so (_False Positives (FP)_).  There are also ~9 customers who ended up churning, that we predicted would not (_False Negatives (FN)_).Now let's calculate the accuracy, precision and recall (these are common mesurments that assists in evaluating the quality of our model and also compare between differnt models):_Accuracy_: Overall, how often is the classifier correct? (TP+TN)/total_Precision_: When it predicts yes, how often is it correct? TP/(predicted yes)_Recall_: When it's actually yes, how often does it predict yes? TP/(actual yes)<jupyter_code>total = confusion_matrix['All'][2]
TP = [please complete...]
TN = [please complete...]
accuracy = ([please complete...]
precision = [please complete...]
recall = [please complete...]
print ('Accuracy: {}'.format(accuracy))
print ('Precision: {}'.format(precision))
print ('Recall: {}'.format(recall))<jupyter_output><empty_output><jupyter_text>An important point here is that because of the `np.round()` function above we are using a simple threshold (or cutoff) of 0.5.  Our predictions from `xgboost` come out as continuous values between 0 and 1 and we force them into the binary classes that we began with.  However, because a customer that churns is expected to cost the company more than proactively trying to retain a customer who we think might churn, we should consider adjusting this cutoff.  That will almost certainly increase the number of false positives, but it can also be expected to increase the number of true positives and reduce the number of false negatives. --- (Must) Clean-upIf you're ready to be done with this notebook, please run the cell below.  This will remove the hosted endpoint you created and avoid any charges from a stray instance being left on.<jupyter_code>sagemaker.Session().delete_endpoint(xgb_predictor.endpoint)<jupyter_output><empty_output><jupyter_text>--- Create an automated workflow using StepFunction data science SDKThe AWS Step Functions Data Science SDK is an open-source library that allows data scientists to easily create workflows that process and publish machine learning models using Amazon SageMaker and AWS Step Functions. You can create machine learning workflows in Python that orchestrate AWS infrastructure at scale, without having to provision and integrate the AWS services separately.Workflow - A sequence of steps designed to perform some workStep - A unit of work within a workflowML Pipeline - A type of workflow used in data science to create and train machine learning modelsThe AWS Step Functions Data Science SDK enables you to do the following.Easily construct and run machine learning workflows that use AWS infrastructure directly in PythonInstantiate common training pipelinesCreate standard machine learning workflows in a Jupyter notebook from templates<jupyter_code>## Install the SDK
!pip install stepfunctions<jupyter_output><empty_output><jupyter_text>Build a machine learning workflow You can use a workflow to create a machine learning pipeline. The AWS Data Science Workflows SDK provides several AWS SageMaker workflow steps that you can use to construct an ML pipeline. In this tutorial you will use the Train and Transform steps.* [**TrainingStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.TrainingStep) - Starts a Sagemaker training job and outputs the model artifacts to S3.* [**ModelStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.ModelStep) - Creates a model on SageMaker using the model artifacts from S3.* [**TransformStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.TransformStep) - Starts a SageMaker transform job* [**EndpointConfigStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.EndpointConfigStep) - Defines an endpoint configuration on SageMaker.* [**EndpointStep**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.EndpointStep) - Deploys the trained model to the configured endpoint.  Define the input schema for a workflow executionThe [**ExecutionInput**](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/placeholders.htmlstepfunctions.inputs.ExecutionInput) API defines the options to dynamically pass information to a workflow at runtime.The following cell defines the fields that must be passed to your workflow when starting an execution.While the workflow is usually static after it is defined, you may want to pass values dynamically that are used by steps in your workflow. To help with this, the SDK provides a way to create placeholders when you define your workflow. These placeholders can be dynamically assigned values when you execute your workflow.ExecutionInput values are accessible to each step of your workflow. You have the ability to define a schema for this placeholder collection, as shown in the cell below. When you execute your workflow the SDK will verify if the dynamic input conforms to the schema you defined.<jupyter_code>from stepfunctions.inputs import ExecutionInput
from stepfunctions.steps import TrainingStep, ModelStep, EndpointConfigStep, EndpointStep, Chain
# SageMaker expects unique names for each job, model and endpoint. 
# If these names are not unique the execution will fail. Pass these
# dynamically for each execution using placeholders.
execution_input = ExecutionInput(schema={
    'JobName': str, 
    'ModelName': str,
    'EndpointName': str,
    'TrainDataLocation': str,
    'ValidationDataLocation': str
})<jupyter_output><empty_output><jupyter_text>Create the training step In the following cell we create the training step and pass the estimator we defined above. See  [TrainingStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.TrainingStep) in the AWS Step Functions Data Science SDK documentation.<jupyter_code>training_step = TrainingStep(
    'Train Step', 
    estimator=xgb,
    data={
        'train': sagemaker.s3_input(execution_input['TrainDataLocation'] , content_type='csv'),
        'validation': sagemaker.s3_input(execution_input['ValidationDataLocation'] , content_type='csv')
    },
    job_name=execution_input['JobName']  
)<jupyter_output><empty_output><jupyter_text>Create the model step In the following cell we define a model step that will create a model in SageMaker using the artifacts created during the TrainingStep. See  [ModelStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.ModelStep) in the AWS Step Functions Data Science SDK documentation.The model creation step typically follows the training step. The Step Functions SDK provides the [get_expected_model](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.TrainingStep.get_expected_model) method in the TrainingStep class to provide a reference for the trained model artifacts. Please note that this method is only useful when the ModelStep directly follows the TrainingStep.<jupyter_code>model_step = ModelStep(
    'Save model',
    model=training_step.get_expected_model(),
    model_name=execution_input['ModelName']  
)<jupyter_output><empty_output><jupyter_text>Create an endpoint configuration stepIn the following cell we create an endpoint configuration step. See [EndpointConfigStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.EndpointConfigStep) in the AWS Step Functions Data Science SDK documentation.<jupyter_code>endpoint_config_step = EndpointConfigStep(
    "Create Endpoint Config",
    endpoint_config_name=execution_input['ModelName'],
    model_name=execution_input['ModelName'],
    initial_instance_count=1,
    instance_type='ml.m5.large'
)<jupyter_output><empty_output><jupyter_text>Create an endpointIn the following cell we create a step to deploy the trained model to an endpoint in AWS SageMaker. See [EndpointStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.sagemaker.EndpointStep) in the AWS Step Functions Data Science SDK documentation.<jupyter_code>endpoint_step = EndpointStep(
    "Create Endpoint",
    endpoint_name=execution_input['EndpointName'],
    endpoint_config_name=execution_input['ModelName']
)<jupyter_output><empty_output><jupyter_text>Chain together steps for your workflowCreate your workflow definition by chaining the steps together. See [Chain](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/sagemaker.htmlstepfunctions.steps.states.Chain) in the AWS Step Functions Data Science SDK documentation.<jupyter_code>workflow_definition = Chain([
    training_step,
    model_step,
    endpoint_config_step,
    endpoint_step
])<jupyter_output><empty_output><jupyter_text>Create your workflow using the workflow definition above, and render the graph with [render_graph](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Workflow.render_graph).<jupyter_code>from stepfunctions.workflow import Workflow
workflow = Workflow(
    name='MyTrainTransformDeploy_v1',
    definition=workflow_definition,
    role=role,
    execution_input=execution_input
)
workflow.render_graph()<jupyter_output><empty_output><jupyter_text>Create the workflow in AWS Step Functions with [create](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Workflow.create).<jupyter_code>workflow.create()<jupyter_output><empty_output><jupyter_text>Run the workflow with [execute](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Workflow.execute).<jupyter_code>import uuid
execution = workflow.execute(
    inputs={
        'JobName': 'churn-{}'.format(uuid.uuid1().hex), # Each Sagemaker Job requires a unique name
        'ModelName': 'churn-{}'.format(uuid.uuid1().hex), # Each Model requires a unique name,
        'EndpointName': 'churn-{}'.format(uuid.uuid1().hex), # Each Endpoint requires a unique name,
        'TrainDataLocation': 's3://{}/{}/train'.format(bucket, prefix),
        'ValidationDataLocation': 's3://{}/{}/validation'.format(bucket, prefix)
    }
)<jupyter_output><empty_output><jupyter_text>Render workflow progress with the [render_progress](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Execution.render_progress).This generates a snapshot of the current state of your workflow as it executes. This is a static image. Run the cell again to check progress.<jupyter_code>execution.render_progress()<jupyter_output><empty_output><jupyter_text>Use [list_events](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Execution.list_events) to list all events in the workflow execution.<jupyter_code>execution.list_events(html=True)<jupyter_output><empty_output><jupyter_text>Use [list_executions](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Workflow.list_executions) to list all executions for a specific workflow.<jupyter_code>workflow.list_executions(html=True)<jupyter_output><empty_output><jupyter_text>Use [list_workflows](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.htmlstepfunctions.workflow.Workflow.list_workflows) to list all workflows in your AWS account.<jupyter_code>Workflow.list_workflows(html=True)<jupyter_output><empty_output><jupyter_text>--- Predict in batch mode using Batch TransformWhat if you want to get inference (predict) over very large quantities of data? For example your input data is in S3, and you would like the inference results to be stored in S3 as well.What if you need to split the work over several instances to reduce job duration?  With some hard work you could do it by coding a custom workflow like this one:1. Start N required hosted real-time endpoints (instances)2. Starts a driver program (that we need to code) to read records from S3 (add support for several popular input formats of course)3. Sends a single/batch of records to each fo the hosted endpoints, possibly in parallel - so use async IO or multi-threaded approach4. Retries if something fails. Expose parameters to control driver behavior. Log the driver activity. Establish error handling. Report progress.5. Collect results to a driver temp storage and store final results in S36. Shutdown endpoints to save $As it happens this is exactly what [SageMaker Batch Transform](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-batch.html) provides.    Let's demo how to use it:<jupyter_code># Blow up our full dataset by 1000x to make things more interesting from scale perspective
model_data.drop(['Churn?_True.'], axis=1).to_csv('features_without_label.csv', header=False, index=False)

import os
total_size_mb = int(os.path.getsize('features_without_label.csv')*1000/1024/1024.0)
print('Upload '+str(total_size_mb)+'MB of input data to S3 (as 1000 CSV files).')

for i in range(1000):
    boto3.Session().resource('s3').Bucket(bucket).Object(
        os.path.join(prefix, 'batch-input/features_without_label_'+str(i)+'.csv')).upload_file('features_without_label.csv')<jupyter_output><empty_output><jupyter_text>With the input data in place, let's create and start a batch transform job (should take about 5min to complete using two instances):<jupyter_code>batch_input = 's3://{}/{}/batch-input'.format(bucket, prefix) # The location of the dataset to transform
batch_output = 's3://{}/{}/batch-output'.format(bucket, prefix) # The location to store the results of the batch transform job

# Define the job
# Doc: https://sagemaker.readthedocs.io/en/stable/estimators.html#sagemaker.estimator.EstimatorBase.transformer
transformer = xgb.transformer(
    instance_count=2,
    instance_type='ml.m5.large', 
    strategy='MultiRecord', # Send a bulk of records in each inference request
    max_concurrent_transforms=2, # m5.large has two cores. Keep both of them busy
    output_path=batch_output)

# Start the job
# Doc: https://sagemaker.readthedocs.io/en/stable/transformer.html#sagemaker.transformer.Transformer.transform
transformer.transform(
    data=batch_input, 
    data_type='S3Prefix', 
    content_type='text/csv', 
    split_type='Line')

# Job is running in the background. You can view its metrics and logs in the console: https://console.aws.amazon.com/sagemaker/home?#/transform-jobs
# Let's wait for it to finish
transformer.wait()<jupyter_output><empty_output><jupyter_text>The job completed, let's take a look at the job output data output:<jupyter_code>batch_output = 's3://{}/{}/batch-output'.format(bucket, prefix)
batch_output
display("Here's our 1000 output files, one per input file:")
!aws s3 ls {batch_output}/ | wc -l 
display('\nfirst 10 predictions from the last file:')
!aws s3 cp {batch_output}/features_without_label_999.csv.out - | head -10<jupyter_output><empty_output><jupyter_text>congrats! you were able to transform/infer 1000 CSV quickly by splitting the work on N machines. The only code you wrote is triggering the Batch Transform API. Batch transform can be part of your ML pipeline, or run on-demand. It can help with feature engineering, or with actual inference.   Learn more [here](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-batch.html). --- (Optional) Bonus 1 - Tune the model using different/additional hyperparametersUse the [Estimator.set_hyperparameters](https://sagemaker.readthedocs.io/en/latest/estimators.html) and explore other hyperparameters, check out the [recommanded values](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html) for each hyper parameter.Try to get a lower validation-error.<jupyter_code>#Bonus 1 code here<jupyter_output><empty_output><jupyter_text>--- (Optional) Bonus 2 - Use SageMaker automatic model tuningSee the [blog post on Automatic model tuning](https://aws.amazon.com/blogs/aws/sagemaker-automatic-model-tuning/)Call auto model tuning to achieve the lowest validation-error. Sit back, relax and appreciate how auto model tuning is far easier than manual tuning.<jupyter_code>#Bonus 2 code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08758183411516057, 'fraction_numerical': 0.005670395381205217, 'mean_word_length': 5.586148361907996, 'pattern_counts': {'":': 0, '<': 190, '<?xml version=': 0, '>': 190, 'https://': 47, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '47', 'dir': 'jupyter-structured-clean-dedup', 'id': '520444', 'license': 'Apache-2.0', 'n_tokens_mistral': 11139, 'n_tokens_neox': 10522, 'n_words': 3952, 'path': 'xgboost_customer_churn-student.ipynb', 'repo_name': 'gilinachum/ChurnWorkshop', 'size': '38808'}