<jupyter_start><jupyter_text>Ludwig TransformerIn this notebook we will use [ludwig](https://ludwig-ai.github.io/ludwig-docs/) to create a deep learning model built on tensorflow 2.4 for classifiying reviews.This notebook takes you through building custom containers for training and deployment in Amazon SageMaker.  * Please add managed policy `AmazonEC2ContainerRegistryFullAccess` to your notebook role to publish to ECR.One you have built the base container, you can jump to directly update the:* [Model Container](Model-container)<jupyter_code>%%capture
!pip install -U sagemaker seaborn
!pip install sagemaker-experiments<jupyter_output><empty_output><jupyter_text>Data PrepDownload a sample form the [Amazon Customer Reviews](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) dataset<jupyter_code>!aws s3 cp s3://amazon-reviews-pds/tsv/amazon_reviews_us_Apparel_v1_00.tsv.gz reviews.tsv.gz<jupyter_output><empty_output><jupyter_text>Load the compressed reviews into pandas, selecting the review headling, body and star rating<jupyter_code>import pandas as pd

# Load a sample of the rows
df_reviews = pd.read_csv('reviews.tsv.gz', compression='gzip', error_bad_lines=False,  nrows=15000,
                         sep='\t', usecols=['review_headline', 'review_body', 'star_rating']).dropna()
df_reviews.info()<jupyter_output><empty_output><jupyter_text>Inspect the dataset<jupyter_code>df_reviews.head()<jupyter_output><empty_output><jupyter_text>Look at the distribution between of scores.<jupyter_code>df_reviews.describe()<jupyter_output><empty_output><jupyter_text>Plot the distribution of ratings<jupyter_code>import seaborn as sns

sns.displot(df_reviews, x='star_rating')<jupyter_output><empty_output><jupyter_text>Split the dataset into train (90%) / validation (5%) / test (5%) and save to file local files.<jupyter_code>from sklearn.model_selection import train_test_split

train_df, val_df = train_test_split(df_reviews, test_size=0.1, random_state=42) 
val_df, test_df = train_test_split(val_df, test_size=0.5, random_state=42)
print('split train: {}, val: {}, test: {} '.format(train_df.shape[0], val_df.shape[0], test_df.shape[0]))<jupyter_output><empty_output><jupyter_text>Save files with headers<jupyter_code>train_df.to_csv('train.csv', index=False, header=True)
val_df.to_csv('validation.csv', index=False, header=True)
test_df.to_csv('test.csv', index=False, header=True)<jupyter_output><empty_output><jupyter_text>Build and Push containersDefine the shell script to build and push images to ECR registry (requires IAM permissions)<jupyter_code>%%writefile build_and_push.sh
#!/usr/bin/env bash

# This script shows how to build the Docker image and push it to ECR to be ready for use
# by SageMaker.

# The argument to this script is the image name. This will be used as the image on the local
# machine and combined with the account and region to form the repository name for ECR.
image=$1
dockerfile=${2:-Dockerfile}
inputdir=${3:-container}

if [ "$image" == "" ]
then
    echo "Usage: $0 <image-name> [dockerfile]"
    exit 1
fi


# Get the account number associated with the current IAM credentials
account=$(aws sts get-caller-identity --query Account --output text)

if [ $? -ne 0 ]
then
    exit 255
fi


# Get the region defined in the current configuration (default to us-west-2 if none defined)
region=$(aws configure get region)
region=${region:-ap-southeast-2}

fullname="${account}.dkr.ecr.${region}.amazonaws.com/${image}:latest"

# If the repository doesn't exist in ECR, create it.

aws ecr describe-repositories --repository-names "${image}" > /dev/null 2>&1

if [ $? -ne 0 ]
then
    aws ecr create-repository --repository-name "${image}" > /dev/null
fi

# Get the login command from ECR and execute it directly
aws ecr get-login-password --region "${region}" | docker login --username AWS --password-stdin "${account}".dkr.ecr."${region}".amazonaws.com

# Build the docker image locally with the image name and then push it to ECR
# with the full name.

docker build -t ${image} -f ${inputdir}/${dockerfile} ${inputdir}
docker tag ${image} ${fullname}

docker push ${fullname}<jupyter_output><empty_output><jupyter_text>Training base container Write local files for Docker training and inference under a new `container` folder.<jupyter_code>!mkdir -p container
%%writefile container/Dockerfile-training
#
# Ludwig Docker image with full set of pre-requiste packages to support these capabilities
#   text features
#   image features
#   audio features
#   visualizations
#   hyperparameter optimization
#   distributed training
#   model serving
#
FROM tensorflow/tensorflow:2.4.0-gpu

# Set a docker label to advertise multi-model support on the container
LABEL com.amazonaws.sagemaker.capabilities.multi-models=false
# Set a docker label to enable container to use SAGEMAKER_BIND_TO_PORT environment variable if present
LABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true    
    
RUN apt-get -y update && apt-get -y install \
    git \
    libsndfile1 \
    cmake \
    libcudnn7=7.6.5.32-1+cuda10.1 \
    libnccl2=2.7.8-1+cuda10.1 \
    libnccl-dev=2.7.8-1+cuda10.1

RUN git clone --depth=1 https://github.com/ludwig-ai/ludwig.git \
    && cd ludwig/ \
    && HOROVOD_GPU_OPERATIONS=NCCL \
       HOROVOD_WITH_TENSORFLOW=1 \
       HOROVOD_WITHOUT_MPI=1 \
       HOROVOD_WITHOUT_PYTORCH=1 \
       HOROVOD_WITHOUT_MXNET=1 \
    && pip install --no-cache-dir '.[full]'

# install the SageMaker Inference & Training Toolkits
RUN pip install --upgrade pip
RUN pip --no-cache-dir install sagemaker-training

# add custom transformer directory
COPY transformer /opt/ml/code/transformer

# set the working directory to be where code is
RUN mkdir -p /opt/ml/code
WORKDIR /opt/ml/code

# copy the training script and configuration
COPY train.py /opt/ml/code/train.py

# define train.py as the script entry point
ENV SAGEMAKER_PROGRAM train.py<jupyter_output><empty_output><jupyter_text>Download a HuggingFace [Tensorflow pre-trained model](https://huggingface.co/models?filter=tf) to be included as part of our container so we do not have to download this on-demand.<jupyter_code>!aws s3 sync s3://models.huggingface.co/bert/gilf/english-yelp-sentiment ./container/transformer<jupyter_output><empty_output><jupyter_text>Create the training file<jupyter_code>%%writefile container/train.py
import argparse
import json
import numpy as np
import pandas as pd
import os

def read_csv(input_dir):
    # Take the set of 1 or more files and read them all into a single pandas dataframe
    input_files = [ os.path.join(input_dir, file) for file in os.listdir(input_dir) if file.endswith('csv') ]
    if len(input_files) == 0:
        raise ValueError(f'No csv files found in {input_dir}')
    df = pd.concat([pd.read_csv(file) for file in input_files])
    print(f'Loaded {len(input_files)} files from {input_dir}, shape: {df.shape}')
    return df
    
def train(args):
    # output directories
    print(f'input train: {args.training_dir}, val: {args.validation_dir}, test: {args.testing_dir}')
    print(f'output model: {args.model_dir}, data: {args.output_data_dir}')

    # configure integrations https://ludwig-ai.github.io/ludwig-docs/user_guide/#integrations
    try:
        import ludwig.contrib
        if args.integration == 'comet':
            ludwig.contrib.use_contrib('comet')
            print(f'using comet integration')
        elif args.integration == 'wandb':
            ludwig.contrib.use_contrib('wandb')
            print(f'using wandb integration')
    except Exception as e:
        print('integration not supported: {}'.format(e))

    # import ludwig after contrib incase we need to hook TF prior to loading
    from ludwig.api import LudwigModel
    
    # loading csv dataframes
    train_df = read_csv(args.training_dir)
    val_df = read_csv(args.validation_dir)
    test_df = read_csv(args.testing_dir)
    
    # train the model based on config yaml file
    ludwig_model = LudwigModel('config.yml')
    train_stats, _, _  = ludwig_model.train(
        experiment_name=args.experiment_name,
        model_name=args.model_name,
        training_set=train_df,
        validation_set=val_df,
        test_set=test_df,
        output_directory = args.output_data_dir, # Save experiment to output data dir
        skip_save_training_statistics=False, # Save training results to file
        skip_save_log=False, # Save tensorboard logs
        skip_save_progress = True, # Don't save progress for now
    )
    
    print('saving model')
    
    # Save the latest model to model_directory
    ludwig_model.save(args.model_dir)
    
    # Save the compiled SavedModel to model directory
    ludwig_model.save_savedmodel(args.model_dir)
    
    print('emmiting metrics')
    
    # enuemrate through the channels and output features to get metrics
    for channel in train_stats:
        for output in ludwig_model.config['output_features']:
            for metric in train_stats['training'][output['name']]:
                # get the metric from last epoch
                value = train_stats['training'][output['name']][metric][-1]
                print('{}_{}={};'.format(channel, metric, value))
    

    print('evaluating test dataset')
        
    # output evaluations based on test
    ludwig_model.evaluate(test_df,
        output_directory=args.output_data_dir,
        skip_save_unprocessed_output=True, # Only save CSV values
        skip_save_predictions=False, # Write predictions to file
        skip_save_eval_stats=False, # Write evaluation stats to file
        collect_predictions=True,
        collect_overall_stats=True,
    )
    
    # Return the model 
    return ludwig_model

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    # reads input channels training and testing from the environment variables
    parser.add_argument("--training-dir", type=str, default=os.environ["SM_CHANNEL_TRAINING"])
    parser.add_argument("--validation-dir", type=str, default=os.environ["SM_CHANNEL_VALIDATION"])
    parser.add_argument("--testing-dir", type=str, default=os.environ["SM_CHANNEL_TESTING"])
    parser.add_argument("--model-dir", type=str, default=os.environ["SM_MODEL_DIR"])
    parser.add_argument("--output-data-dir", type=str, default=os.environ["SM_OUTPUT_DATA_DIR"])
    parser.add_argument("--experiment-name", type=str, default='api_experiment')
    parser.add_argument("--model-name", type=str, default='run')
    parser.add_argument("--integration", type=str, required=False)
    args = parser.parse_args()
    train(args)<jupyter_output><empty_output><jupyter_text>Build the base training container<jupyter_code>training_image_name = 'ludwig-training'
!sh build_and_push.sh $training_image_name Dockerfile-training<jupyter_output><empty_output><jupyter_text>Inference ContainerCreate the `ludwig-inference` dockerfile that inherts from `ludwig-training`, adding support for multi-model-server and Java depdendancy.<jupyter_code>%%writefile container/Dockerfile-inference
FROM ludwig-training:latest

# Install necessary dependencies for MMS and SageMaker Inference Toolkit
RUN apt-get update && \
    apt-get -y install --no-install-recommends \
    build-essential \
    ca-certificates \
    openjdk-8-jdk-headless

# install the SageMaker Inference & Training Toolkits
RUN pip --no-cache-dir install multi-model-server retrying sagemaker-inference

# copy the training script and configuration
COPY model_handler.py /opt/ml/code/model_handler.py
COPY entrypoint.py /opt/ml/code/entrypoint.py
RUN chmod +x /opt/ml/code/entrypoint.py

# Run the entrypoint 
ENTRYPOINT ["python", "/opt/ml/code/entrypoint.py"]

# Define command to be passed to the entrypoint
CMD ["serve"]<jupyter_output><empty_output><jupyter_text>Create the [inference handler for tensorflow](https://github.com/aws/sagemaker-tensorflow-training-toolkit) implementation<jupyter_code>%%writefile container/model_handler.py
"""
ModelHandler defines an example model handler for load and inference requests for MXNet CPU models
"""
from __future__ import absolute_import

import json
import os
import re

from ludwig.api import LudwigModel
from io import StringIO
import pandas as pd

from sagemaker_inference import content_types, environment, utils, logging

logger = logging.get_logger()

# TODO: Consider how to handle multi-model in future
ENABLE_MULTI_MODEL = os.getenv("SAGEMAKER_MULTI_MODEL", "false") == "true"

class ModelHandler(object):
    """
    Ludwig Model handler implementation.
    """

    def __init__(self):
        self.environment = environment.Environment()
        self.initialized = False
        self.ludwig_model = None
    
    def initialize(self, context):
        """
        Initialize model. This will be called during model loading time
        :param context: Initial context contains model server system properties.
        :return:
        """
        self.initialized = True
        properties = context.system_properties
        logger.debug('properties: {}'.format(properties))
        
        # Contains the url parameter passed to the load request
        model_dir = properties.get("model_dir")
        # model_dir = os.path.join(model_dir, 'api_experiment_run', 'model')
        
        # Load the model from the default 
        logger.info('loading from model_dir: {}'.format(model_dir))
        self.ludwig_model = LudwigModel.load(model_dir)

    def preprocess(self, request, content_type):
        """
        Transform raw input into model input data.
        :param request: list of raw requests
        :return: list of preprocessed model input data
        """
        # Take the input data and read into pandas dataframe

        logger.info('preprocess content type: {}'.format(content_type))
        logger.debug('input', request) # TEMP
        
        if content_type == 'text/csv':
            df_input = pd.read_csv(StringIO(request))
        else:
            df_input = pd.read_json(StringIO(request))            
        
        logger.info('input shape: {}'.format(df_input.shape))
        return df_input

    def inference(self, df_input):
        """
        Internal inference methods
        :param model_input: transformed model input data list
        :return: list of inference output in NDArray
        """
        # Perform inference on this dataframe
        
        (df_pred, output_directory) = self.ludwig_model.predict(df_input)
        return df_pred

    def postprocess(self, df_pred, accept):
        """
        Return predict result in as list.
        :param inference_output: list of inference output
        :return: list of predict results
        """
        # Return a dictionary output formatted as a list for each item
        logger.info('postprocess accept: {}'.format(accept))

        if accept == 'text/csv':
            output = df_pred.to_csv(index=False)
        else:
            output = df_pred.to_json()
        
        logger.info('output length: {}'.format(len(output)))
        return output
        
    def handle(self, data, context):
        """
        Call preprocess, inference and post-process functions
        :param data: input data
        :param context: mms context
        """
        # Return an array of the result
        
        # get input data
        input_data = data[0].get("body")

        # Get the content and accept types
        request_processor = context.request_processor[0]
        request_property = request_processor.get_request_properties()
        content_type = utils.retrieve_content_type_header(request_property)
        accept = request_property.get("Accept") or request_property.get("accept")
        if not accept or accept == content_types.ANY:
            accept = self.environment.default_accept # Should default to JSON
        if content_type in content_types.UTF8_TYPES:
            input_data = input_data.decode("utf-8")                
        
        # Process input to get output
        model_input = self.preprocess(input_data, content_type)
        model_predictions = self.inference(model_input)
        model_result = self.postprocess(model_predictions, accept)
        
        # Set the response content type
        context.set_response_content_type(0, accept)
        return [model_result]

_service = ModelHandler()


def handle(data, context):
    if not _service.initialized:
        _service.initialize(context)

    if data is None:
        return None

    return _service.handle(data, context)<jupyter_output><empty_output><jupyter_text>Create the entrypoint code<jupyter_code>%%writefile container/entrypoint.py
import subprocess
import sys
import shlex
import os
from retrying import retry
from subprocess import CalledProcessError
from sagemaker_inference import model_server

def _retry_if_error(exception):
    return isinstance(exception, CalledProcessError or OSError)

@retry(stop_max_delay=1000 * 50, retry_on_exception=_retry_if_error)
def _start_mms():
    # by default the number of workers per model is 1, but we can configure it through the
    # environment variable below if desired.
    # os.environ['SAGEMAKER_MODEL_SERVER_WORKERS'] = '2'
    model_server.start_model_server(handler_service='/opt/ml/code/model_handler.py:handle')

def main():
    if sys.argv[1] == 'serve':
        _start_mms()
    else:
        subprocess.check_call(shlex.split(' '.join(sys.argv[1:])))

    # prevent docker exit
    subprocess.call(['tail', '-f', '/dev/null'])
    
main()<jupyter_output><empty_output><jupyter_text>Build the inference container<jupyter_code>inference_image_name = 'ludwig-inference'
!sh build_and_push.sh $inference_image_name Dockerfile-inference<jupyter_output><empty_output><jupyter_text>Model containerCreate the `ludwig-reviews` dockerfile that inherts from `ludwig-training`, adding the configuration and optional tracking integration.<jupyter_code>%%writefile container/Dockerfile-reviews
FROM ludwig-training:latest

COPY config.yml /opt/ml/code/config.yml<jupyter_output><empty_output><jupyter_text>Modify this cell to write [ludwig config](https://ludwig-ai.github.io/ludwig-docs/user_guide/configuration).  See the text-features in the [user guide](https://ludwig-ai.github.io/ludwig-docs/user_guide/text-features) for more options such as setting the encoder, or processing.Encoders that you might want to try include:* `bert`* `gpt2`* `xlnet`* `roberta`* `distilbert`Or else you can download any pre-trained model and include the `pretrained_model_name_or_path` in the configuration.👇👇👇<jupyter_code>%%writefile container/config.yml
input_features:
    -   name: review_headline
        type: text
        level: word
        encoder: parallel_cnn # Simple encoder for title
    -   name: review_body
        type: text
        level: word
        encoder: bert
        preprocessing:
            char_sequence_length_limit: 1024
            pretrained_tokenizer_fast: true
        pretrained_model_name_or_path: /opt/ml/code/transformer
output_features:
    -   name: star_rating
        type: category
training:
    epochs: 5
    early_stop: 1
    batch_size: 16 # OOM for bert if we don't keep this small
    learning_rate: 0.001
    decay: true<jupyter_output><empty_output><jupyter_text>Build the model container<jupyter_code>model_image_name = 'ludwig-reviews'
!sh build_and_push.sh $model_image_name Dockerfile-reviews<jupyter_output><empty_output><jupyter_text>Train and EvaluateWe can now create a training job using based on our custom `ludwig-reviews` container and eveluate results.  Create ExperimentDefine functions to create the sagemaker experiment for training.<jupyter_code>import sagemaker
import boto3
from smexperiments.experiment import Experiment
from smexperiments.trial import Trial
from botocore import exceptions as botoexceptions

def create_or_load_experiment(experiment_name, tags=[]):
    try:
        return Experiment.create(
            experiment_name=experiment_name, tags=tags
        )
    except botoexceptions.ClientError as err:
        errcontent = err.response["Error"]
        if (
            errcontent["Code"] == "ValidationException"
            and "must be unique" in errcontent["Message"]
        ):
            print(f"Loading Experiment '{experiment_name}'")
            return Experiment.load(experiment_name)
        raise err


def create_or_load_trial(experiment_name, trial_name, display_name='Training', tags=[]):
    try:
        Trial.create(experiment_name=experiment_name, trial_name=trial_name, tags=tags)
    except botoexceptions.ClientError as err:
        errcontent = err.response["Error"]
        if (
            errcontent["Code"] == "ValidationException"
            and "must be unique" in errcontent["Message"]
        ):
            print(f"Loading Trial '{trial_name}'")
            return Trial.load(trial_name)
        raise err

# Get the session and default bucket
role = sagemaker.get_execution_role()
session = sagemaker.session.Session()
bucket = session.default_bucket()<jupyter_output><empty_output><jupyter_text>Update the trial name to identify the dataset and training job.👇👇👇<jupyter_code>exp_name = 'ludwig-reviews'
trial_name = 'bert-yelp' ## << Update this for new experiment trial
create_or_load_experiment(experiment_name=exp_name)
create_or_load_trial(experiment_name=exp_name, trial_name=trial_name)<jupyter_output><empty_output><jupyter_text>Upload DataUpload data to s3 in bucket prefixed with experiment and trial<jupyter_code># Specify data previx version
prefix = f'ludwig-transformer/{exp_name}/{trial_name}'

s3_train_uri = session.upload_data('train.csv', bucket, prefix + '/data/training')
s3_val_uri = session.upload_data('validation.csv', bucket, prefix + '/data/validation')
s3_test_uri = session.upload_data('test.csv', bucket, prefix + '/data/test')<jupyter_output><empty_output><jupyter_text>Train ModelGet the model container `image_uri` based on current region and account id.<jupyter_code># Get the region and account id
def get_image_uri(image_name):
    region = boto3.session.Session().region_name
    sts = boto3.client('sts')
    account_id = sts.get_caller_identity()["Account"]
    return f"{account_id}.dkr.ecr.{region}.amazonaws.com/{image_name}:latest"
    
training_image_uri = get_image_uri(model_image_name)
print(training_image_uri)<jupyter_output><empty_output><jupyter_text>Run the training job in the cloud or with `local` mode to test.<jupyter_code>def get_metric_definitions(channels, metrics):
    for ch in channels:
        for metric in metrics:
            yield { "Name": "{}_{}".format(ch, metric), "Regex": "{}_{}=(.*?);".format(ch, metric) }

# Get metric definitions
metric_definitions = list(get_metric_definitions(
    ['training', 'validation'], 
    ['loss', 'mean_squared_error', 'mean_absolute_error', 'r2']))

instance_type = "ml.p3.2xlarge"
# instance_type = "local" # << Uncomment to run local training

# set hyperameters, 
hyperparameters = {
    "model-name": exp_name,
    "experiment-name": trial_name, 
    "integration": 'comet', # Specify integration for additional logging
}

# Set the base job name
base_job_name = f"{exp_name}-{trial_name}"

# Create the estimator
ludwig = sagemaker.estimator.Estimator(
    training_image_uri,
    role,
    instance_count=1,
    instance_type=instance_type,
    base_job_name=base_job_name,
    metric_definitions=metric_definitions,
    hyperparameters=hyperparameters
)

# Specify the data source
s3_input_train = sagemaker.inputs.TrainingInput(s3_data=s3_train_uri, content_type="text/csv")
s3_input_val = sagemaker.inputs.TrainingInput(s3_data=s3_val_uri, content_type="text/csv")
s3_input_test = sagemaker.inputs.TrainingInput(s3_data=s3_test_uri, content_type="text/csv")
data = {"training": s3_input_train, "validation": s3_input_val,"testing": s3_input_test}

# Fit the model
ludwig.fit(inputs=data, experiment_config={
    "ExperimentName": hyperparameters['model-name'], # group by mode
    "TrialName": hyperparameters['experiment-name'], # trail by experiment
})<jupyter_output><empty_output><jupyter_text>Estimated Training Time / PriceOn 15k records* ml.p3.2xlarge: 24 mins (0.4 hr) = \$ 1.53Optionally attach to an existing model below if you are restarting your notebook.<jupyter_code># import sagemaker
# ludwig = sagemaker.estimator.Estimator.attach('ludwig-reviews-bert-yelp-2020-11-18-04-06-18-496')<jupyter_output><empty_output><jupyter_text>Download model and output artifactsGet the model artifaction s3 location<jupyter_code>model_uri = ludwig.model_data
output_uri = model_uri.replace('model.tar.gz', 'output.tar.gz')<jupyter_output><empty_output><jupyter_text>Download model artifacts and extract to `model` folder.<jupyter_code>!rm -Rf model && mkdir -p model
!aws s3 cp $model_uri .
!tar xvzf model.tar.gz -C ./model<jupyter_output><empty_output><jupyter_text>Download output data and extract to `results` folder.<jupyter_code>!rm -Rf output && mkdir -p results
!aws s3 cp $output_uri .
!tar xvzf output.tar.gz -C ./results<jupyter_output><empty_output><jupyter_text>Evaluate modelLoad back the test set and predictions<jupyter_code>import pandas as pd

# Load test dataset
test_df = pd.read_csv('test.csv')
test_df['score'] = test_df['star_rating']

# Load predictions
pred_df = pd.read_csv('./results/star_rating_predictions.csv', header=None, names=['score_predictions'])
prob_df = pd.read_csv('./results/star_rating_probability.csv', header=None, names=['score_probabilty'])

# Join with the test dataset
pred_df = test_df.join(pred_df).join(prob_df)
print(pred_df.shape)
pred_df.head()
from sklearn.metrics import confusion_matrix

cf_matrix = confusion_matrix(pred_df['score'], pred_df['score_predictions'])
print(cf_matrix)<jupyter_output><empty_output><jupyter_text>Visualize the confusion matrix as a heatmap<jupyter_code>import seaborn as sns
import numpy as np

classes = ['1','2','3','4','5'] # Define classes as strings
cf_percent = cf_matrix / cf_matrix.astype(np.float).sum(axis=1)
sns.heatmap(cf_percent, annot=True, xticklabels=classes, yticklabels=classes, fmt=".0%")<jupyter_output><empty_output><jupyter_text>Return the overall accuracy<jupyter_code>from sklearn.metrics import accuracy_score

print('Accuracy: {}'.format(accuracy_score(pred_df['score_predictions'], test_df['score'])))<jupyter_output><empty_output><jupyter_text>Plot the ROC curve for each of the classes by loading back the probabilites matrix aligned to the score columns<jupyter_code>import json

# Load the score column order
with open('./model/training_set_metadata.json', 'r') as f:
    metadata = json.load(f)
score_cols = metadata['star_rating']['idx2str']

# Load the probabilities matrix first UNK col
prob_df = pd.read_csv('./results/star_rating_probabilities.csv', header=None, names=score_cols).drop('<UNK>', axis=1)
prob_df = prob_df[classes] # re-order cols
prob_df.head()
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import label_binarize
import numpy as np
import matplotlib.pyplot as plt
from itertools import cycle

# Convert string classes to int for binarize of predictions
n_classes = len(classes)
y_test = label_binarize(test_df['score'], classes=[int(c) for c in classes])
y_score = prob_df.values
assert y_test.shape == y_score.shape

# Compute ROC curve and ROC area for each class
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

# Compute micro-average ROC curve and ROC area
fpr["micro"], tpr["micro"], _ = roc_curve(y_test.ravel(), y_score.ravel())
roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])

# First aggregate all false positive rates
all_fpr = np.unique(np.concatenate([fpr[i] for i in range(n_classes)]))

# Then interpolate all ROC curves at this points
mean_tpr = np.zeros_like(all_fpr)
for i in range(n_classes):
    mean_tpr += np.interp(all_fpr, fpr[i], tpr[i])

# Finally average it and compute AUC
mean_tpr /= n_classes

fpr["macro"] = all_fpr
tpr["macro"] = mean_tpr
roc_auc["macro"] = auc(fpr["macro"], tpr["macro"])

# Plot all ROC curves
plt.figure()
plt.plot(fpr["micro"], tpr["micro"],
         label='micro-average ROC curve (area = {0:0.2f})'.format(roc_auc["micro"]),
         color='deeppink', linestyle=':', linewidth=4)

plt.plot(fpr["macro"], tpr["macro"],
         label='macro-average ROC curve (area = {0:0.2f})'.format(roc_auc["macro"]),
         color='navy', linestyle=':', linewidth=4)

lw=2
colors = cycle(['aqua', 'darkorange', 'cornflowerblue'])
for i, color in zip(range(n_classes), colors):
    plt.plot(fpr[i], tpr[i], color=color, lw=lw,
             label='ROC curve of class {0} (area = {1:0.2f})'.format(classes[i], roc_auc[i]))

plt.plot([0, 1], [0, 1], 'k--', lw=lw)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic for multi-class')
plt.legend(loc="lower right")
plt.show()<jupyter_output><empty_output><jupyter_text>Test locallyDeploy the model locally using the built container<jupyter_code>!docker run --rm --name "ludwig-inference" \
    -p 8080:8080 \
    -v "$PWD/model:/opt/ml/model" \
    -v "$PWD:/opt/ml/input" ludwig-inference:latest serve<jupyter_output><empty_output><jupyter_text>> While the above cell is running, click here [TEST NOTEBOOK](test-ludwig.ipynb) to run some tests.>> After you finish the tests, press **STOP**  DeployCreate the model specifying the inference container, and then deploy endpoint.<jupyter_code>inference_image_uri = get_image_uri(inference_image_name)
print(inference_image_uri)<jupyter_output><empty_output><jupyter_text>Deploy the endpoint passing the `inference_image_uri` which will be passed down to create model.<jupyter_code>predictor = ludwig.deploy(initial_instance_count=1, 
                          instance_type='ml.g4dn.xlarge',
                          image_uri=inference_image_uri)<jupyter_output><empty_output><jupyter_text>Optionally attach a predictor<jupyter_code># from sagemaker.predictor import Predictor
# predictor = Predictor('ludwig-reviews-bert-yelp-2020-11-17-23-48-01-555')<jupyter_output><empty_output><jupyter_text>PredictLoad a sample dataset and predict against the endpoint<jupyter_code>import pandas as pd

test_df = pd.read_csv('test.csv')
payload = test_df.to_csv(index=False)
%%time

from sagemaker.serializers import CSVSerializer
from sagemaker.deserializers import CSVDeserializer

# Make a prediction against the payload
predictor.serializer = CSVSerializer()
predictor.deserializer = CSVDeserializer()
result = predictor.predict(data=payload)

# Load the result as array where first element is the column names
pred_df = pd.DataFrame(result[1:], columns=result[0])
print(pred_df.shape)
pred_df.head()<jupyter_output><empty_output><jupyter_text>Save results to file so for analysis.<jupyter_code>pred_df.to_csv('./results/score_predictions.csv', header=False)
pred_df.head()<jupyter_output><empty_output><jupyter_text>CleanupDelete endpoint<jupyter_code>predictor.delete_endpoint()<jupyter_output><empty_output><jupyter_text>Appendix: Ludwig CLIIf you have a powerful enough notebook instance, you can execute the train and evaluate directly in the notebook with the [ludwig CLI](https://ludwig-ai.github.io/ludwig-docs/user_guide/command-line-interface)<jupyter_code>%%capture
!pip uninstall -y horovod # uninstall horovod to get ludwig to work correctly in notebook
!pip install -U ludwig[text]<jupyter_output><empty_output><jupyter_text>Verify that we have transformers and installed as part of ludwig<jupyter_code>import ludwig
import transformers
print('lugwig: {}, transformers: {}'.format(ludwig.__version__, transformers.__version__))<jupyter_output><empty_output><jupyter_text>TrainRun training in the shell<jupyter_code>!sudo mkdir -p /opt/ml/code
!sudo chmod 777 /opt/ml/code
!ln -s $(pwd)/container/transformer /opt/ml/code/transformer
!ludwig train --config_file container/config.yml --experiment_name=exp1 --training_set train.csv --validation_set validation.csv --test_set test.csv<jupyter_output><empty_output><jupyter_text>EvaluateEvalate the model<jupyter_code>!ludwig evaluate --dataset test.csv --model_path results/exp1_run/model/<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10628482972136223, 'fraction_numerical': 0.007244582043343654, 'mean_word_length': 4.448886639676114, 'pattern_counts': {'":': 11, '<': 183, '<?xml version=': 0, '>': 185, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '44', 'dir': 'jupyter-structured-clean-dedup', 'id': '352869', 'license': 'MIT-0', 'n_tokens_mistral': 10501, 'n_tokens_neox': 10130, 'n_words': 2541, 'path': 'train-ludwig.ipynb', 'repo_name': 'aws-samples/amazon-sagemaker-ludwig-transformer', 'size': '32300'}