<jupyter_start><jupyter_text>![MLU Logo](../data/MLU_Logo.png)  Machine Learning Accelerator - Natural Language Processing - Lecture 2 Sagemaker built-in Training and Deployment with LinearLearnerIn this notebook, we use Sagemaker's built-in machine learning model __LinearLearner__ to predict the __isPositive__ field of our review dataset.Overall dataset schema:* __reviewText:__ Text of the review* __summary:__ Summary of the review* __verified:__ Whether the purchase was verified (True or False)* __time:__ UNIX timestamp for the review* __log_votes:__ Logarithm-adjusted votes log(1+votes)* __isPositive:__ Whether the review is positive or negative (1 or 0)__Notes on AWS SageMaker__* Fully managed machine learning service, to quickly and easily get you started on building and training machine learning models - we have seen that already! Integrated Jupyter notebook instances, with easy access to data sources for exploration and analysis, abstract away many of the messy infrastructural details needed for hands-on ML - you don't have to manage servers, install libraries/dependencies, etc.!* Apart from easily building end-to-end machine learning models in SageMaker notebooks, like we did so far, SageMaker also provides a few __build-in common machine learning algorithms__ (check "SageMaker Examples" from your SageMaker instance top menu for a complete updated list) that are optimized to run efficiently against extremely large data in a distributed environment. __LinearLearner__ build-in algorithm in SageMaker is extremely fast at inference and can be trained at scale, in mini-batch fashion over GPU(s). The trained model can then be directly deployed into a production-ready hosted environment for easy access at inference. We will follow these steps:1. Read the dataset2. Exploratory Data Analysis3. Text Processing: Stop words removal and stemming4. Training - Validation - Test Split5. Data processing with Pipeline and ColumnTransform6. Train a classifier with SageMaker build-in algorithm7. Model evaluation8. Deploy the model to an endpoint9. Test the enpoint10. Clean up model artifacts<jupyter_code>#Upgrade dependencies
!pip install --upgrade pip
!pip install --upgrade scikit-learn<jupyter_output>Requirement already up-to-date: pip in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (20.1.1)
Requirement already up-to-date: scikit-learn in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (0.23.1)
Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from scikit-learn) (1.14.3)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from scikit-learn) (0.16.0)
Requirement already satisfied, skipping upgrade: scipy>=0.19.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from scikit-learn) (1.1.0)
Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from scikit-learn) (2.1.0)<jupyter_text>1. Reading the dataset(Go to top)We will use the __pandas__ library to read our dataset.<jupyter_code>import pandas as pd

df = pd.read_csv('../data/examples/AMAZON-REVIEW-DATA-CLASSIFICATION.csv')

print('The shape of the dataset is:', df.shape)<jupyter_output>The shape of the dataset is: (70000, 6)<jupyter_text>Let's look at the first five rows in the dataset.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>2. Exploratory Data Analysis(Go to top) Let's look at the target distribution for our datasets.<jupyter_code>df["isPositive"].value_counts()<jupyter_output><empty_output><jupyter_text>Checking the number of missing values:<jupyter_code>print(df.isna().sum())<jupyter_output>reviewText    11
summary       14
verified       0
time           0
log_votes      0
isPositive     0
dtype: int64<jupyter_text>We have missing values in our text fields.  3. Text Processing: Stop words removal and stemming(Go to top)<jupyter_code># Install the library and functions
import nltk

nltk.download('punkt')
nltk.download('stopwords')<jupyter_output>[nltk_data] Downloading package punkt to /home/ec2-user/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data]     /home/ec2-user/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>We will create the stop word removal and text cleaning processes below. NLTK library provides a list of common stop words. We will use the list, but remove some of the words from that list. It is because those words are actually useful to understand the sentiment in the sentence.<jupyter_code>import nltk, re
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer
from nltk.tokenize import word_tokenize

# Let's get a list of stop words from the NLTK library
stop = stopwords.words('english')

# These words are important for our problem. We don't want to remove them.
excluding = ['against', 'not', 'don', "don't",'ain', 'aren', "aren't", 'couldn', "couldn't",
             'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 
             'haven', "haven't", 'isn', "isn't", 'mightn', "mightn't", 'mustn', "mustn't",
             'needn', "needn't",'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', 
             "weren't", 'won', "won't", 'wouldn', "wouldn't"]

# New stop word list
stop_words = [word for word in stop if word not in excluding]

snow = SnowballStemmer('english')

def process_text(texts): 
    final_text_list=[]
    for sent in texts:
        
        # Check if the sentence is a missing value
        if isinstance(sent, str) == False:
            sent = ""
            
        filtered_sentence=[]
        
        sent = sent.lower() # Lowercase 
        sent = sent.strip() # Remove leading/trailing whitespace
        sent = re.sub('\s+', ' ', sent) # Remove extra space and tabs
        sent = re.compile('<.*?>').sub('', sent) # Remove HTML tags/markups:
        
        for w in word_tokenize(sent):
            # We are applying some custom filtering here, feel free to try different things
            # Check if it is not numeric and its length>2 and not in stop words
            if(not w.isnumeric()) and (len(w)>2) and (w not in stop_words):  
                # Stem and add to filtered list
                filtered_sentence.append(snow.stem(w))
        final_string = " ".join(filtered_sentence) #final string of cleaned words
 
        final_text_list.append(final_string)
        
    return final_text_list<jupyter_output><empty_output><jupyter_text>4. Training - Validation - Test Split(Go to top)Let's split our dataset into training (80%), validation (10%) and test (10%) using sklearn's [__train_test_split()__](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(df[["reviewText", "summary", "time", "log_votes"]],
                                                  df["isPositive"],
                                                  test_size=0.20,
                                                  shuffle=True,
                                                  random_state=324
                                                 )

X_val, X_test, y_val, y_test = train_test_split(X_val,
                                                y_val,
                                                test_size=0.5,
                                                shuffle=True,
                                                random_state=324)
print("Processing the reviewText fields")
X_train["reviewText"] = process_text(X_train["reviewText"].tolist())
X_val["reviewText"] = process_text(X_val["reviewText"].tolist())
X_test["reviewText"] = process_text(X_test["reviewText"].tolist())

print("Processing the summary fields")
X_train["summary"] = process_text(X_train["summary"].tolist())
X_val["summary"] = process_text(X_val["summary"].tolist())
X_test["summary"] = process_text(X_test["summary"].tolist())<jupyter_output>Processing the reviewText fields
Processing the summary fields<jupyter_text>Our process_text() method in section 3 uses empty string for missing values.  5. Data processing with Pipeline and ColumnTransform(Go to top)In the previous examples, we have seen how to use pipeline to prepare a data field for our machine learning model. This time, we will focus on multiple fields: numeric and text fields.    * For the numerical features pipeline, the __numerical_processor__ below, we use a MinMaxScaler (don't have to scale features when using Decision Trees, but it's a good idea to see how to use more data transforms). If different processing is desired for different numerical features, different pipelines should be built - just like shown below for the two text features.   * For the text features pipeline, the __text_processor__ below, we use CountVectorizer() for the text fields.   The selective preparations of the dataset features are then put together into a collective ColumnTransformer, to be finally used in a Pipeline along with an estimator. This ensures that the transforms are performed automatically on the raw data when fitting the model and when making predictions, such as when evaluating the model on a validation dataset via cross-validation or making predictions on a test dataset in the future.<jupyter_code># Grab model features/inputs and target/output
numerical_features = ['time',
                      'log_votes']

text_features = ['summary',
                 'reviewText']

model_features = numerical_features + text_features
model_target = 'isPositive'
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import MinMaxScaler
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

### COLUMN_TRANSFORMER ###
##########################

# Preprocess the numerical features
numerical_processor = Pipeline([
    ('num_imputer', SimpleImputer(strategy='mean')),
    ('num_scaler', MinMaxScaler()) 
                                ])
# Preprocess 1st text feature
text_processor_0 = Pipeline([
    ('text_vect_0', CountVectorizer(binary=True, max_features=50))
                                ])

# Preprocess 2nd text feature (larger vocabulary)
text_precessor_1 = Pipeline([
    ('text_vect_1', CountVectorizer(binary=True, max_features=150))
                                ])

# Combine all data preprocessors from above (add more, if you choose to define more!)
# For each processor/step specify: a name, the actual process, and finally the features to be processed
data_preprocessor = ColumnTransformer([
    ('numerical_pre', numerical_processor, numerical_features),
    ('text_pre_0', text_processor_0, text_features[0]),
    ('text_pre_1', text_precessor_1, text_features[1])
                                    ]) 

### DATA PREPROCESSING ###
##########################

print('Datasets shapes before processing: ', X_train.shape, X_val.shape, X_test.shape)

X_train = data_preprocessor.fit_transform(X_train).toarray()
X_val = data_preprocessor.transform(X_val).toarray()
X_test = data_preprocessor.transform(X_test).toarray()

print('Datasets shapes after processing: ', X_train.shape, X_val.shape, X_test.shape)<jupyter_output>Datasets shapes before processing:  (56000, 4) (7000, 4) (7000, 4)
Datasets shapes after processing:  (56000, 202) (7000, 202) (7000, 202)<jupyter_text>6. Train a classifier with SageMaker build-in algorithm(Go to top) We will call the Sagemaker `LinearLearner()` below. * __Compute power:__ We will use `train_instance_count` and `train_instance_type` parameters. This example uses `ml.m4.xlarge` resource for training. We can change the instance type for our needs (For example GPUs for neural networks). * __Model type:__ `predictor_type` is set to __`binary_classifier`__, as we have a binary classification problem here; __`multiclass_classifier`__ could be used if there are 3 or more classes involved, or __'regressor'__ for a regression problem.<jupyter_code>import sagemaker

# Call the LinearLearner estimator object
linear_classifier = sagemaker.LinearLearner(role=sagemaker.get_execution_role(),
                                           train_instance_count=1,
                                           train_instance_type='ml.m4.xlarge',
                                           predictor_type='binary_classifier')<jupyter_output><empty_output><jupyter_text>We are using the `record_set()` function of our binary_estimator to set the training, validation, test parts of the estimator.<jupyter_code>train_records = linear_classifier.record_set(X_train.astype("float32"),
                                            y_train.values.astype("float32"),
                                            channel='train')
val_records = linear_classifier.record_set(X_val.astype("float32"),
                                          y_val.values.astype("float32"),
                                          channel='validation')
test_records = linear_classifier.record_set(X_test.astype("float32"),
                                           y_test.values.astype("float32"),
                                           channel='test')<jupyter_output><empty_output><jupyter_text>`fit()` function applies a distributed version of the Stochastic Gradient Descent (SGD) algorithm and we are sending the data to it. We disabled logs with `logs=False`. You can remove that parameter to see more details about the process. __This process takes about 3-4 minutes on a ml.m4.xlarge instance.__<jupyter_code>%%time
linear_classifier.fit([train_records,
                       val_records,
                       test_records],
                      logs=False)<jupyter_output>2020-07-08 03:58:33 Starting - Starting the training job
2020-07-08 03:58:35 Starting - Launching requested ML instances............
2020-07-08 03:59:43 Starting - Preparing the instances for training.................
2020-07-08 04:01:12 Downloading - Downloading input data..
2020-07-08 04:01:26 Training - Downloading the training image...
2020-07-08 04:01:45 Training - Training image download completed. Training in progress........
2020-07-08 04:02:27 Uploading - Uploading generated training model
2020-07-08 04:02:34 Completed - Training job completed
CPU times: user 258 ms, sys: 29.6 ms, total: 288 ms
Wall time: 4min 1s<jupyter_text>7. Model Evaluation(Go to top)We can use Sagemaker analytics to get some performance metrics of our choice on the test set. This doesn't require us to deploy our model. Since this is a binary classfication problem, we can check the accuracy.<jupyter_code>sagemaker.analytics.TrainingJobAnalytics(linear_classifier._current_job_name, 
                                         metric_names = ['test:binary_classification_accuracy']
                                        ).dataframe()<jupyter_output><empty_output><jupyter_text>8. Deploy the model to an endpoint(Go to top)In the last part of this exercise, we will deploy our model to another instance of our choice. This will allow us to use this model in production environment. Deployed endpoints can be used with other AWS Services such as Lambda and API Gateway. A nice walkthrough is available here: https://aws.amazon.com/blogs/machine-learning/call-an-amazon-sagemaker-model-endpoint-using-amazon-api-gateway-and-aws-lambda/ if you are interested.Run the following cell to deploy the model. We can use different instance types such as: `ml.t2.medium`, `ml.c4.xlarge` etc. __This will take some time to complete (Approximately 7-8 minutes).__<jupyter_code>%%time
linear_classifier_predictor = linear_classifier.deploy(initial_instance_count = 1,
                                                       instance_type = 'ml.t2.medium',
                                                       endpoint_name = 'LinearLearnerEndpoint'
                                                      )<jupyter_output>---------------!CPU times: user 226 ms, sys: 15.7 ms, total: 242 ms
Wall time: 7min 31s<jupyter_text>9. Test the endpoint(Go to top)Let's use the deployed endpoint. We will send our test data and get predictions of it.<jupyter_code>import numpy as np

# Let's get test data in batch size of 25 and make predictions.
prediction_batches = [linear_classifier_predictor.predict(batch)
                      for batch in np.array_split(X_test.astype("float32"), 25)
                     ]

# Let's get a list of predictions
print([pred.label['score'].float32_tensor.values[0] for pred in prediction_batches[0]])<jupyter_output>[0.7902330160140991, 0.1961013525724411, 0.9616415500640869, 0.6506044268608093, 0.6324111819267273, 0.13196150958538055, 0.7690600752830505, 0.03198748826980591, 0.9872618317604065, 0.9839351773262024, 0.7172891497612, 0.5460348725318909, 0.8144757747650146, 0.7077612280845642, 0.8950422406196594, 0.9754360914230347, 0.6271064877510071, 0.24714428186416626, 0.4907993972301483, 0.9344369769096375, 0.6512829661369324, 0.7926424741744995, 0.18350689113140106, 0.743069589138031, 0.28049173951148987, 0.4696404039859772, 0.948711633682251, 0.0935184508562088, 0.9856142401695251, 0.17812573909759521, 0.8229093551635742, 0.8398733735084534, 0.6779384016990662, 0.9942814111709595, 0.099876768887043, 0.9922278523445129, 0.419038861989975, 0.8894866704940796, 0.06670814007520676, 0.26895591616630554, 0.9754182696342468, 0.27426987886428833, 0.6335780024528503, 0.7747470736503601, 0.6460452675819397, 0.16709531843662262, 0.9967218041419983, 0.9504019618034363, 0.9899674654006958, 0.78151547908782[...]<jupyter_text>10. Clean up model artifacts(Go to top)You can run the following to delete the endpoint after you are done using it.<jupyter_code>sagemaker_session = sagemaker.Session()
sagemaker_session.delete_endpoint(linear_classifier_predictor.endpoint)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09263657957244656, 'fraction_numerical': 0.06661879246533724, 'mean_word_length': 3.541896638233818, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 63, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 54, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '203084', 'license': 'MIT-0', 'n_tokens_mistral': 5997, 'n_tokens_neox': 5167, 'n_words': 1650, 'path': 'notebooks/MLA-NLP-Lecture2-Sagemaker.ipynb', 'repo_name': 'dsMOOC/MLA-NLP', 'size': '18103'}