<jupyter_start><jupyter_text>Direct Marketing with Amazon SageMaker AutoPilotLast update: February 6th, 2020<jupyter_code>import sagemaker

print (sagemaker.__version__)

sess   = sagemaker.Session()
bucket = sess.default_bucket()                     
prefix = 'sagemaker/DEMO-automl-dm'
import numpy as np 
import pandas as pd
%%sh
wget -N https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip
unzip -o bank-additional.zip<jupyter_output><empty_output><jupyter_text>Let's read the CSV file into a Pandas data frame and take a look at the first few lines.<jupyter_code># https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html
data = pd.read_csv('./bank-additional/bank-additional-full.csv', sep=';')
pd.set_option('display.max_columns', 500)     # Make sure we can see all of the columns
pd.set_option('display.max_rows', 50)         # Keep the output on one page
data[:10] # Show the first 10 lines
data.shape # (number of lines, number of columns)<jupyter_output><empty_output><jupyter_text>Splitting the datasetWe split the dataset into training (95%) and test (5%) datasets. We will use the training dataset for AutoML, where it will be automatically split again for training and validation. Once the model has been deployed, we'll use the test dataset to evaluate its performance.<jupyter_code># Set the seed to 123 for reproductibility
# https://pandas.pydata.org/pandas-docs/version/0.25/generated/pandas.DataFrame.sample.html
# https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.split.html
train_data, test_data, _ = np.split(data.sample(frac=1, random_state=123), 
                                                  [int(0.95 * len(data)), int(len(data))])  

# Save to CSV files
train_data.to_csv('automl-train.csv', index=False, header=True) # Need to keep column names
test_data.to_csv('automl-test.csv', index=False, header=True)
!ls -l automl*.csv<jupyter_output><empty_output><jupyter_text>**No preprocessing needed!** AutoML will take care of this, so let's just copy the training set to S3.<jupyter_code>s3_input_data = sess.upload_data(path="automl-train.csv", key_prefix=prefix + "/input")
print(s3_input_data)<jupyter_output><empty_output><jupyter_text>Setting up the SageMaker AutoPilot jobAfter uploading the dataset to S3, we can invoke SageMaker AutoPilot to find the best ML pipeline to train a model on this dataset. The required inputs for invoking a SageMaker AutoML job are the dataset location in S3, the name of the column of the dataset you want to predict (`y` in this case) and an IAM role.<jupyter_code>from sagemaker.automl.automl import AutoML
# https://sagemaker.readthedocs.io/en/stable/automl.html

role = sagemaker.get_execution_role()

auto_ml_job = AutoML(
    role = role,                                              # IAM permissions for SageMaker
    sagemaker_session = sess,                                 # 
    target_attribute_name = 'y',                              # The column we want to predict
    output_path = 's3://{}/{}/output'.format(bucket,prefix),  # Save artefacts here
    max_runtime_per_training_job_in_seconds = 600, 
    total_job_runtime_in_seconds = 3600
)<jupyter_output><empty_output><jupyter_text>Launching the SageMaker AutoPilot jobWe can now launch the job by calling the `fit()` API.<jupyter_code>auto_ml_job.fit(inputs=s3_input_data, logs=False, wait=False)
auto_ml_job.describe_auto_ml_job()<jupyter_output><empty_output><jupyter_text>Tracking the progress of the AutoPilot jobSageMaker AutoPilot job consists of four high-level steps : * Data Preprocessing, where the dataset is split into train and validation sets.* Recommending Pipelines, where the dataset is analyzed and SageMaker AutoPilot comes up with a list of ML pipelines that should be tried out on the dataset.* Automatic Feature Engineering, where SageMaker AutoPilot performs feature transformation on individual features of the dataset as well as at an aggregate level.* ML pipeline selection and hyperparameter tuning, where the top performing pipeline is selected along with the optimal hyperparameters for the training algorithm (the last stage of the pipeline).<jupyter_code>from time import sleep

job = auto_ml_job.describe_auto_ml_job()
job_status = job['AutoMLJobStatus']
job_sec_status = job['AutoMLJobSecondaryStatus']

if job_status not in ('Stopped', 'Failed'):
    while job_status in ('InProgress') and job_sec_status in ('AnalyzingData'):
        sleep(30)
        job = auto_ml_job.describe_auto_ml_job()
        job_status = job['AutoMLJobStatus']
        job_sec_status = job['AutoMLJobSecondaryStatus']
        print (job_status, job_sec_status)
    print("Data analysis complete")<jupyter_output><empty_output><jupyter_text>Viewing notebooks generated by SageMaker AutoPilotOnce data analysis is complete, SageMaker AutoPilot generates two notebooks: * Data exploration,* Candidate definition.<jupyter_code>job = auto_ml_job.describe_auto_ml_job()
job_candidate_notebook = job['AutoMLJobArtifacts']['CandidateDefinitionNotebookLocation']
job_data_notebook = job['AutoMLJobArtifacts']['DataExplorationNotebookLocation']

print(job_candidate_notebook)
print(job_data_notebook)<jupyter_output><empty_output><jupyter_text>Let's copy these two notebooks.<jupyter_code>%%sh -s $job_candidate_notebook $job_data_notebook
aws s3 cp $1 .
aws s3 cp $2 .<jupyter_output><empty_output><jupyter_text>Go back to the folder view, and open these notebooks. Lots of useful information in there! SageMaker AutoPilot then launches feature engineering, and prepares different training and validation datasets.<jupyter_code>job = auto_ml_job.describe_auto_ml_job()
job_status = job['AutoMLJobStatus']
job_sec_status = job['AutoMLJobSecondaryStatus']

if job_status not in ('Stopped', 'Failed'):
    while job_status in ('InProgress') and job_sec_status in ('FeatureEngineering'):
        sleep(30)
        job = auto_ml_job.describe_auto_ml_job()
        job_status = job['AutoMLJobStatus']
        job_sec_status = job['AutoMLJobSecondaryStatus']
        print (job_status, job_sec_status)
    print("Feature engineering complete")<jupyter_output><empty_output><jupyter_text>Once feature engineering is complete, SageMaker AutoPilot launches Automatic Model Tuning on the different candidates. While model tuning is running, we can explore its progress with SageMaker Experiments.<jupyter_code>import pandas as pd
from sagemaker.analytics import ExperimentAnalytics

exp = ExperimentAnalytics(
    sagemaker_session=sess, 
    experiment_name=job['AutoMLJobName'] + '-aws-auto-ml-job'
)

df = exp.dataframe()
print("Number of jobs: ", len(df))

# Move metric to first column
df = pd.concat([df['ObjectiveMetric - Max'], df.drop(['ObjectiveMetric - Max'], axis=1)], axis=1)
# Show top 5 jobs
df.sort_values('ObjectiveMetric - Max', ascending=0)[:5]
job = auto_ml_job.describe_auto_ml_job()
job_status = job['AutoMLJobStatus']
job_sec_status = job['AutoMLJobSecondaryStatus']

if job_status not in ('Stopped', 'Failed'):
    while job_status in ('InProgress') and job_sec_status in ('ModelTuning'):
        sleep(30)
        job = auto_ml_job.describe_auto_ml_job()
        job_status = job['AutoMLJobStatus']
        job_sec_status = job['AutoMLJobSecondaryStatus']
        print (job_status, job_sec_status)
    print("Model tuning complete")<jupyter_output><empty_output><jupyter_text>Deploying the best candidateNow that we have successfully completed the AutoML job on our dataset and visualized the trials, we can create a model from any of the trials with a single API call and then deploy that model for online or batch prediction using [Inference Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html). For this notebook, we deploy only the best performing trial for inference. The best candidate is the one we're really interested in.<jupyter_code>job_best_candidate = auto_ml_job.best_candidate()
print(job_best_candidate['CandidateName'])
print(job_best_candidate['FinalAutoMLJobObjectiveMetric'])
from time import strftime, gmtime
timestamp = strftime('%d-%H-%M-%S', gmtime())

endpoint_name = job['AutoMLJobName']+'-'+timestamp
auto_ml_job.deploy(
    initial_instance_count = 1,
    instance_type = 'ml.m4.xlarge',
    endpoint_name = endpoint_name
)<jupyter_output><empty_output><jupyter_text>Scoring the best candidate Let's predict and score the validation set. We'll compute metrics ourselves just for fun.<jupyter_code>from sagemaker.predictor import csv_serializer, RealTimePredictor
from sagemaker.content_types import CONTENT_TYPE_CSV

predictor = RealTimePredictor(
    endpoint=endpoint_name, 
    sagemaker_session=sess, 
    serializer=csv_serializer,
    content_type=CONTENT_TYPE_CSV, 
    accept='text/csv'
)
import sys

tp = tn = fp = fn = count = 0

with open('automl-test.csv') as f:
    lines = f.readlines()
    for l in lines[1:]:   # Skip header
        l = l.split(',')  # Split CSV line into feature array
        label = l[-1]     # Store 'yes'/'no' label
        l = l[:-1]        # Remove label
        l = ','.join(l)   # Rebuild CSV line without label
                
        response = predictor.predict(l)
        response = response.decode("utf-8")
        #print ("label %s response %s" %(label,response))

        if 'yes' in label:
            # Sample is positive
            if 'yes' in response:
                # True positive
                tp=tp+1
            else:
                # False negative
                fn=fn+1
        else:
            # Sample is negative
            if 'no' in response:
                # True negative
                tn=tn+1
            else:
                # False positive
                fp=fp+1
        count = count+1
        if (count % 100 == 0):   
            sys.stdout.write(str(count)+' ')
            
print ("Done")
#Confusion matrix
print ("%d %d" % (tn, fp))
print ("%d %d" % (fn, tp))

accuracy  = (tp+tn)/(tp+tn+fp+fn)
precision = tp/(tp+fp)
recall    = tp/(tp+fn)
f1        = (2*precision*recall)/(precision+recall)

print ("%.4f %.4f %.4f %.4f" % (accuracy, precision, recall, f1))<jupyter_output><empty_output><jupyter_text>Deleting the endpointOnce that we're done predicting, we can delete the endpoint (and stop paying for it).<jupyter_code>sess.delete_endpoint(predictor.endpoint)<jupyter_output><empty_output><jupyter_text>The SageMaker AutoML job creates many underlying artifacts such as dataset splits, preprocessing scripts, preprocessed data, etc. Let's delete them.<jupyter_code>import boto3

job_outputs_prefix = '{}/output/{}'.format(prefix, job['AutoMLJobName'])
print(job_outputs_prefix)

s3_bucket =boto3.resource('s3').Bucket(bucket)
s3_bucket.objects.filter(Prefix=job_outputs_prefix).delete()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09761066864234118, 'fraction_numerical': 0.008890535284311909, 'mean_word_length': 4.278103616813294, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '37845', 'license': 'MIT', 'n_tokens_mistral': 3437, 'n_tokens_neox': 3274, 'n_words': 927, 'path': 'sdkv1/ch3/example2.ipynb', 'repo_name': 'absynthe/Learn-Amazon-SageMaker', 'size': '10798'}