<jupyter_start><jupyter_text>Sentiment Analysis with TensorFlow 2Amazon SageMaker provides both built-in algorithms and an easy path to train your own custom models.  Although the built-in algorithms cover many domains (computer vision, natural language processing etc.) and are easy to use (just provide your data), sometimes training a custom model is the preferred approach.  This notebook will focus on training a custom model using TensorFlow 2.  Sentiment analysis is a very common text analytics task that involves determining whether a text sample is positive or negative about its subject.  There are several different algorithms for performing this task, including statistical algorithms and deep learning algorithms.  With respect to deep learning, a 1D Convolutional Neural Net (CNN) is sometimes used for this purpose.  In this notebook we'll use a CNN built with TensorFlow 2 to perform sentiment analysis in Amazon SageMaker on the IMDB dataset, which consists of movie reviews labeled as having positive or negative sentiment. Three aspects of Amazon SageMaker will be demonstrated:- How to use a SageMaker prebuilt TensorFlow 2 container with a custom model training script similar to one you would use outside SageMaker. This feature is known as Script Mode.  - Local Mode training:  this allows you to prototype and test your code on a low-cost notebook instance before creating a full scale training job on more powerful and expensive instances.- Hosted training: for full scale training on a complete dataset.  - Distributed training:  using a single, multi-GPU instance to speed up training.  - Batch Transform for offline, asynchronous predictions on large batches of data.   Prepare the datasetWe'll begin by loading the reviews dataset, and padding the reviews so all reviews have the same length.  Each review is represented as an array of numbers, where each number represents an indexed word.  We'll also pad shorter reviews to match a maximum specified length.<jupyter_code>!pip install --user -qU 'sagemaker[local]'
!pip install matplotlib
%matplotlib inline

import logging
logging.getLogger("tensorflow").setLevel(logging.ERROR)
import numpy as np
import os

from tensorflow.keras.preprocessing import sequence
from tensorflow.python.keras.datasets import imdb

max_features = 20000
maxlen = 400

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
print(len(x_train), 'train sequences')
print(len(x_test), 'test sequences')

x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)<jupyter_output>25000 train sequences
25000 test sequences
x_train shape: (25000, 400)
x_test shape: (25000, 400)<jupyter_text>Training data for both Local Mode training and hosted Training must be saved as files.  Accordingly, we'll save the padded data to files, locally for now, and later to Amazon S3.<jupyter_code>data_dir = os.path.join(os.getcwd(), 'data')
os.makedirs(data_dir, exist_ok=True)

train_dir = os.path.join(os.getcwd(), 'data/train')
os.makedirs(train_dir, exist_ok=True)

test_dir = os.path.join(os.getcwd(), 'data/test')
os.makedirs(test_dir, exist_ok=True)

csv_test_dir = os.path.join(os.getcwd(), 'data/csv-test')
os.makedirs(csv_test_dir, exist_ok=True)

np.save(os.path.join(train_dir, 'x_train.npy'), x_train)
np.save(os.path.join(train_dir, 'y_train.npy'), y_train)
np.save(os.path.join(test_dir, 'x_test.npy'), x_test)
np.save(os.path.join(test_dir, 'y_test.npy'), y_test)
np.savetxt(os.path.join(csv_test_dir, 'csv-test.csv'), np.array(x_test[:100], dtype=np.int32), fmt='%d', delimiter=",")<jupyter_output><empty_output><jupyter_text>Local Mode Training & Git IntegrationAmazon SageMaker’s Local Mode training feature is a convenient way to make sure your code is working as expected before moving on to full scale, hosted training. With Local Mode, you can run quick tests with just a sample of training data, and/or a small number of epochs (passes over the full training set), while avoiding the time and expense of attempting full scale hosted training using possibly buggy code.   Prepare for Local ModeTo train in Local Mode, it is necessary to have docker-compose or nvidia-docker-compose (for GPU) installed. Running the following script will install docker-compose or nvidia-docker-compose and configure the notebook environment for you.<jupyter_code>!wget -q https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-script-mode/master/local_mode_setup.sh
!wget -q https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-script-mode/master/daemon.json    
!/bin/bash ./local_mode_setup.sh<jupyter_output>/bin/sh: wget: command not found
/bin/sh: wget: command not found
/bin/bash: ./local_mode_setup.sh: No such file or directory<jupyter_text>Git ConfigurationNow we need a training script that can be used to train the model in Amazon SageMaker.  In this example, we'll use Git integration. That is, you can specify a training script that is stored in a GitHub, AWS CodeCommit or another Git repository as the entry point so that you don't have to download the scripts locally. If you do so, any source directory and dependencies should be in the same repo if they are needed.To use Git integration, pass a dict `git_config` as a parameter when you create an Amazon SageMaker Estimator object. In the `git_config` parameter, you specify the fields `repo`, `branch` and `commit` to locate the specific repo you want to use. If you do not specify `commit` in `git_config`, the latest commit of the specified repo and branch will be used by default.  Also, if authentication is required to access the repo, you can specify fields `2FA_enabled`, `username`, `password` and `token` accordingly.The script that we will use in this example is stored a public GitHub repo so we don't need authentication to access it. Let's specify the `git_config` argument here:<jupyter_code>git_config = {'repo': 'https://github.com/aws-samples/amazon-sagemaker-script-mode', 
              'branch': 'master'}<jupyter_output><empty_output><jupyter_text>Local Mode Estimator setupThe next step is to set up a TensorFlow Estimator for Local Mode training. The Estimator will be used to configure and start a training job.  A key parameters for the Estimator is the `train_instance_type`, which is the kind of hardware on which training will run. In the case of Local Mode, we simply set this parameter to `local_gpu` to invoke Local Mode training on the GPU, or to `local` if the instance has a CPU. Other parameters of note are the algorithm’s hyperparameters, which are passed in as a dictionary, and a Boolean parameter indicating that we are using Script Mode.<jupyter_code>import sagemaker
from sagemaker.tensorflow import TensorFlow


model_dir = '/opt/ml/model'
train_instance_type = 'local'
hyperparameters = {'epochs': 1, 'batch_size': 64, 'learning_rate': 0.01}
local_estimator = TensorFlow(
                       git_config=git_config,
                       source_dir='tf-sentiment-script-mode',
                       entry_point='sentiment.py',
                       model_dir=model_dir,
                       train_instance_type=train_instance_type,
                       train_instance_count=1,
                       hyperparameters=hyperparameters,
                       role='arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881',
                       base_job_name='tf-sentiment',
                       framework_version='2.1',
                       py_version='py3',
                       script_mode=True)<jupyter_output><empty_output><jupyter_text>Now we'll briefly train the model in Local Mode.  Since this is just to make sure the code is working, we'll train for only one epoch.  On a notebook instance using a general purpose CPU-based instance type such as m5.xlarge, this one epoch will take about 3 or 4 minutes.  As you'll see from the logs below the cell when training is complete, even when trained for only one epoch, the accuracy of the model on training data is around 80%.<jupyter_code>inputs = {'train': f'file://{train_dir}',
          'test': f'file://{test_dir}'}

local_estimator.fit(inputs)<jupyter_output><empty_output><jupyter_text>Hosted TrainingAfter we've confirmed our code is working using Local Mode training, we can move on to use SageMaker's hosted training, which typically uses more powerful compute resources separate from your less powerful, lower cost notebook instance.  Hosted training spins up one or more instances (i.e. a cluster) for training, and then tears the cluster down when training is complete. In general, hosted training is preferred for doing actual large-scale training on more powerful instances, especially for distributed training on a single instance with multiple GPUs or multiple instances each having multiple GPUs.  Upload data to S3Before starting hosted training, the data must be present in storage that can be accessed by SageMaker. The storage options are:  Amazon S3 (object storage service), Amazon EFS (elastic NFS file system service), and Amazon FSx for Lustre (high-performance file system service). For this example, we'll upload the data to S3.<jupyter_code>import sagemaker

s3_prefix = 'tf-keras-sentiment'

traindata_s3_prefix = '{}/data/train'.format(s3_prefix)
testdata_s3_prefix = '{}/data/test'.format(s3_prefix)

train_s3 = sagemaker.Session().upload_data(path='./data/train/', key_prefix=traindata_s3_prefix)
test_s3 = sagemaker.Session().upload_data(path='./data/test/', key_prefix=testdata_s3_prefix)

inputs = {'train':train_s3, 'test': test_s3}
print(inputs)<jupyter_output>'upload_data' method will be deprecated in favor of 'S3Uploader' class (https://sagemaker.readthedocs.io/en/stable/s3.html#sagemaker.s3.S3Uploader) in SageMaker Python SDK v2.<jupyter_text>Hosted training Estimator setupWith the training data now in S3, we're ready to set up an Estimator object for hosted training. It is similar to the Local Mode Estimator, except the `train_instance_type` has been set to a ML instance type instead of a local type for Local Mode. Additionally, we've increased the number of epochs for actual training, as opposed to just testing the code in Local Mode.<jupyter_code>from sagemaker.tensorflow import TensorFlow

model_dir = '/opt/ml/model'
train_instance_type = 'ml.p3.8xlarge'
hyperparameters = {'epochs': 10, 'batch_size': 256, 'learning_rate': 0.01}

estimator = TensorFlow(
                       git_config=git_config,
                       source_dir='tf-sentiment-script-mode',
                       entry_point='sentiment.py',
                       model_dir=model_dir,
                       train_instance_type=train_instance_type,
                       train_instance_count=1,
                       hyperparameters=hyperparameters,
                       role=sagemaker.get_execution_role(),
                       base_job_name='tf-sentiment',
                       framework_version='2.1',
                       py_version='py3',
                       script_mode=True)<jupyter_output><empty_output><jupyter_text>Distributed training on a single multi-GPU instanceThe instance type selected above, p3.8xlarge, contains four GPUs based on NVIDIA's V100 Tensor Core architecture.  This presents an opportunity to do distributed training within a single multi-GPU instance, utilizing all four GPUs to reduce total training time compared to using a single GPU.  Although using multiple instances also is a possibility, using a single multi-GPU instance may be more performant because it avoids extra network traffic necessary to coordinate multiple instances.  For larger datasets and more complex models, using multiple instances may be a necessity, however, that is not the case here.To utilize all four GPUs on the instance, you don't need to do anything special in Amazon SageMaker:  TensorFlow 2 itself will handle the details under the hood.  TensorFlow 2 includes several native distribution strategies, including MirroredStrategy, which is well-suited for training a model using multiple GPUs on a single instance.  To enable MirroredStrategy, simply add the following lines of code in your training script before you define and compile the model:```pythondef get_model(learning_rate):    mirrored_strategy = tf.distribute.MirroredStrategy()        with mirrored_strategy.scope():        embedding_layer = tf.keras.layers.Embedding(max_features,                                                    embedding_dims,                                                    input_length=maxlen)        ....        model.compile(loss='binary_crossentropy',                      optimizer=optimizer,                      metrics=['accuracy'])        return model```Additionally, the batch size was increased in the Estimator hyperparameters to account for the fact that batches are divided among multiple GPUs.  If you are interested in reviewing the rest of the training code, it is at the GitHub repository referenced above in the `git_config` variable.    Start the hosted training jobWith the change in training instance type and increase in epochs, we simply call `fit` to start the actual hosted training.  The training job should take around 5 minutes, including the time needed to spin up the training instance.  At the end of hosted training, you'll see from the logs below the cell that accuracy on the training set has greatly increased, and accuracy on the validation set is approaching 90%.<jupyter_code>estimator.fit(inputs)<jupyter_output><empty_output><jupyter_text>Although the training accuracy has improved, the validation accuracy plateaued, so the model might be overfitting now:  it could be less able to generalize to data it has not yet seen.  This is the case even though we are employing dropout as a regularization technique to reduce the possibility of overfitting.  (See the training script at the GitHub repository referenced above.)  For a production model, further experimentation would be necessary.TensorFlow 2's tf.keras API provides a convenient way to capture the history of model training.  When the model was saved after training, the history was saved alongside it.  To retrieve the history, we first download the trained model from the S3 bucket where SageMaker stored it.  Models trained by SageMaker are always accessible in this way to be run anywhere.  Next, we can unzip it to gain access to the history data structure, and then simply load the history as JSON:<jupyter_code>import json 

!aws s3 cp {estimator.model_data} ./model/model.tar.gz
!tar -xzf ./model/model.tar.gz -C ./model

with open('./model/history.p', "r") as f:
    history = json.load(f)<jupyter_output><empty_output><jupyter_text>Now we can plot the training curves based on the history, with separate graphs for model accuracy and model loss.  We can see that training converged relatively smoothly to higher model accuracy and correspondingly lower model loss.<jupyter_code>import matplotlib.pyplot as plt

def plot_training_curves(history): 
    fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharex=True)
    ax = axes[0]
    ax.plot(history['accuracy'], label='train')
    ax.set(title='model accuracy', ylabel='accuracy', xlabel='epoch')
    ax.legend()
    ax = axes[1]
    ax.plot(history['loss'], label='train')
    ax.set(title='model loss', ylabel='loss', xlabel='epoch')
    ax.legend()
    fig.tight_layout()
    
plot_training_curves(history)<jupyter_output><empty_output><jupyter_text>Batch PredictionIf our use case requires individual predictions in near real-time, SageMaker hosted endpoints can be created. Hosted endpoints also can be used for pseudo-batch prediction, but the process is more involved than simply using SageMaker's Batch Transform feature, which is designed for large-scale, asynchronous batch inference.To use Batch Transform, we first upload to S3 some input test data to be transformed.  The data can be in any format accepted by your model; in this case, it is CSV.<jupyter_code>csvtestdata_s3_prefix = '{}/data/csv-test'.format(s3_prefix)
csvtest_s3 = sagemaker.Session().upload_data(path='./data/csv-test/', key_prefix=csvtestdata_s3_prefix)
print(csvtest_s3)<jupyter_output><empty_output><jupyter_text>A Transformer object must be set up to describe the Batch Transform job, including the amount and type of inference hardware to be used.  Then the actual transform job itself is started with a call to the `transform` method of the Transformer.  When setting up Batch Transform jobs, hardware considerations are important, just as they are for training:- `instance_count`:  Batch Transform can spin up a cluster of multiple instances; at the end of the job, the cluster is torn down automatically.  Since this dataset is small, we'll just use one instance.- `instance_type`:  When doing inference for smaller models, such as this one, often CPU-based instance types can be used instead of GPU instance types, allowing significant cost savings.  Note, however, that the choice of specific CPU instance type can significantly affect inference speed:  although we could use a general purpose instance here such as a m5.xlarge, if we use a compute-optimized c5.xlarge instance, the total batch inference time is cut in half.<jupyter_code>transformer = estimator.transformer(instance_count=1, 
                                    instance_type='ml.c5.xlarge')

transformer.transform(csvtest_s3, content_type='text/csv')
print('Waiting for transform job: ' + transformer.latest_transform_job.job_name)
transformer.wait()<jupyter_output><empty_output><jupyter_text>We can now download the batch predictions from S3 to the local filesystem on the notebook instance; the predictions are contained in a file with a .out extension, and are embedded in JSON.  Next we'll load the JSON and examine the predictions, which are confidence scores from 0.0 to 1.0 where numbers close to 1.0 indicate positive sentiment, while numbers close to 0.0 indicate negative sentiment.<jupyter_code>import json

batch_output = transformer.output_path
!mkdir -p batch_data/output
!aws s3 cp --recursive $batch_output/ batch_data/output/

with open('batch_data/output/csv-test.csv.out', 'r') as f:
    jstr = json.load(f)
    results = [float('%.3f'%(item)) for sublist in jstr['predictions'] for item in sublist]
    print(results)<jupyter_output><empty_output><jupyter_text>Now let's look at the text of some actual reviews to see the predictions in action.  First, we have to convert the integers representing the words back to the words themselves by using a reversed dictionary.  Next we can decode the reviews, taking into account that the first 3 indices were reserved for "padding", "start of sequence", and "unknown", and removing a string of unknown tokens from the start of the review.<jupyter_code>import re

regex = re.compile(r'^[\?\s]+')

word_index = imdb.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
first_decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_test[0]])
regex.sub('', first_decoded_review)<jupyter_output><empty_output><jupyter_text>Overall, this review looks fairly negative.  Let's compare the actual label with the prediction:<jupyter_code>def get_sentiment(score):
    return 'positive' if score > 0.5 else 'negative' 

print('Labeled sentiment for this review is {}, predicted sentiment is {}'.format(get_sentiment(y_test[0]), 
                                                                                  get_sentiment(results[0])))<jupyter_output><empty_output><jupyter_text>Training deep learning models is a stochastic process, so your results may vary -- there is no guarantee that the predicted result will match the actual label. However, it is likely that the sentiment prediction agrees with the label for this review.  Let's now examine another review:<jupyter_code>second_decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_test[5]])
regex.sub('', second_decoded_review)
print('Labeled sentiment for this review is {}, predicted sentiment is {}'.format(get_sentiment(y_test[5]), 
                                                                                  get_sentiment(results[5])))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07212246099499559, 'fraction_numerical': 0.008536944362672947, 'mean_word_length': 4.222393031001793, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 67, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '280742', 'license': 'Apache-2.0', 'n_tokens_mistral': 5609, 'n_tokens_neox': 5337, 'n_words': 2322, 'path': '07_train/archive/tf-sentiment-script-mode/sentiment-analysis.ipynb', 'repo_name': 'MarcusFra/workshop', 'size': '20382'}