<jupyter_start><jupyter_text>Regression with Amazon SageMaker XGBoost (Parquet input)This notebook exhibits the use of a Parquet dataset for use with the SageMaker XGBoost algorithm. The example here is almost the same as [Regression with Amazon SageMaker XGBoost algorithm](xgboost_abalone.ipynb).This notebook tackles the exact same problem with the same solution, but has been modified for a Parquet input. The original notebook provides details of dataset and the machine learning use-case.<jupyter_code>import os
import boto3
import re
from sagemaker import get_execution_role

role = get_execution_role()
region = boto3.Session().region_name

# S3 bucket for saving code and model artifacts.
# Feel free to specify a different bucket here if you wish.
bucket = sagemaker.Session().default_bucket()
prefix = 'sagemaker/DEMO-xgboost-parquet'
bucket_path = 'https://s3-{}.amazonaws.com/{}'.format(region, bucket)<jupyter_output><empty_output><jupyter_text>We will use [PyArrow](https://arrow.apache.org/docs/python/) library to store the Abalone dataset in the Parquet format.<jupyter_code>!pip install pyarrow
%%time

import numpy as np
import pandas as pd
import urllib.request
from sklearn.datasets import load_svmlight_file

# Download the dataset and load into a pandas dataframe
FILE_DATA = 'abalone'
urllib.request.urlretrieve("https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data", FILE_DATA)
feature_names=['Sex', 
               'Length', 
               'Diameter', 
               'Height', 
               'Whole weight', 
               'Shucked weight', 
               'Viscera weight', 
               'Shell weight', 
               'Rings']
data = pd.read_csv('abalone.csv', 
                   header=None, 
                   names=feature_names)

# SageMaker XGBoost has the convention of label in the first column
data = data[feature_names[-1:] + feature_names[:-1]]

# Split the downloaded data into train/test dataframes
train, test = np.split(data.sample(frac=1), [int(.8*len(data))])

# requires PyArrow installed
train.to_parquet('abalone_train.parquet')
test.to_parquet('abalone_test.parquet')
%%time
import sagemaker
sagemaker.Session().upload_data('abalone_train.parquet', 
                                bucket=bucket, 
                                key_prefix=prefix+'/'+'train')

sagemaker.Session().upload_data('abalone_test.parquet', 
                                bucket=bucket, 
                                key_prefix=prefix+'/'+'test')<jupyter_output><empty_output><jupyter_text>We obtain the new container by specifying the framework version (0.90-1). This version specifies the upstream XGBoost framework version (0.90) and an additional SageMaker version (1). If you have an existing XGBoost workflow based on the previous (0.72) container, this would be the only change necessary to get the same workflow working with the new container.<jupyter_code>from sagemaker.amazon.amazon_estimator import get_image_uri
container = get_image_uri(region, 'xgboost', '0.90-1')<jupyter_output><empty_output><jupyter_text>After setting training parameters, we kick off training, and poll for status until training is completed, which in this example, takes between 5 and 6 minutes.<jupyter_code>%%time
import time
import boto3
from time import gmtime, strftime

job_name = 'xgboost-parquet-example-training-' + strftime("%Y-%m-%d-%H-%M-%S", gmtime())
print("Training job", job_name)

#Ensure that the training and validation data folders generated above are reflected in the "InputDataConfig" parameter below.

create_training_params = {
    "AlgorithmSpecification": {
        "TrainingImage": container,
        "TrainingInputMode": "Pipe"
    },
    "RoleArn": role,
    "OutputDataConfig": {
        "S3OutputPath": bucket_path + "/" + prefix + "/single-xgboost"
    },
    "ResourceConfig": {
        "InstanceCount": 1,
        "InstanceType": "ml.m5.24xlarge",
        "VolumeSizeInGB": 20
    },
    "TrainingJobName": job_name,
    "HyperParameters": {
        "max_depth":"5",
        "eta":"0.2",
        "gamma":"4",
        "min_child_weight":"6",
        "subsample":"0.7",
        "silent":"0",
        "objective":"reg:linear",
        "num_round":"10"
    },
    "StoppingCondition": {
        "MaxRuntimeInSeconds": 3600
    },
    "InputDataConfig": [
        {
            "ChannelName": "train",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": bucket_path + "/" + prefix + "/train",
                    "S3DataDistributionType": "FullyReplicated"
                }
            },
            "ContentType": "application/x-parquet",
            "CompressionType": "None"
        },
        {
            "ChannelName": "validation",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": bucket_path + "/" + prefix + "/test",
                    "S3DataDistributionType": "FullyReplicated"
                }
            },
            "ContentType": "application/x-parquet",
            "CompressionType": "None"
        }
    ]
}


client = boto3.client('sagemaker', region_name=region)
client.create_training_job(**create_training_params)

status = client.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']
print(status)
while status !='Completed' and status!='Failed':
    time.sleep(60)
    status = client.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']
    print(status)
%matplotlib inline
from sagemaker.analytics import TrainingJobAnalytics

metric_name = 'validation:rmse'

metrics_dataframe = TrainingJobAnalytics(training_job_name=job_name, metric_names=[metric_name]).dataframe()
plt = metrics_dataframe.plot(kind='line', figsize=(12,5), x='timestamp', y='value', style='b.', legend=False)
plt.set_ylabel(metric_name);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11347993956689609, 'fraction_numerical': 0.010575793184488837, 'mean_word_length': 3.128898128898129, 'pattern_counts': {'":': 39, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '15059', 'license': 'Apache-2.0', 'n_tokens_mistral': 1811, 'n_tokens_neox': 1721, 'n_words': 426, 'path': 'introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb', 'repo_name': 'danny/amazon-sagemaker-examples', 'size': '5957'}