<jupyter_start><jupyter_text>Finetuning PyTorch BERT with NGCThe BERT family of models are a powerful set of natural language understanding models based on the transformer architecture originating from the paper Attention Is All You Need, which you can find here:  https://arxiv.org/abs/1706.03762These models work by running unsupervised pre-training on massive sets of text data. This process requires an enormous amount of time and compute. Luckily for us, BERT models are built for transfer learning. BERT models are able to be finetuned to perform many different NLU tasks like question answering, sentiment analysis, document summarization, and more.For this tutorial, we are going to download a BERT base model and finetune this model on the Stanford Question Answering Dataset and walk through the steps necessary to deploy it to a Sagemaker endpoint.<jupyter_code>!wget https://api.ngc.nvidia.com/v2/models/nvidia/bert_base_pyt_amp_ckpt_pretraining_lamb/versions/1/files/bert_base.pt -O bert_base.pt
import collections
import math
import random
import torch
import os, tarfile, json
import time, datetime
from io import StringIO
import numpy as np
import boto3
import sagemaker
from sagemaker.pytorch import estimator, PyTorchModel, PyTorchPredictor, PyTorch
from sagemaker.utils import name_from_base
from model_utils.file_utils import PYTORCH_PRETRAINED_BERT_CACHE
from model_utils.modeling import BertForQuestionAnswering, BertConfig, WEIGHTS_NAME, CONFIG_NAME
from model_utils.tokenization import (BasicTokenizer, BertTokenizer, whitespace_tokenize)
from types import SimpleNamespace
from helper_funcs import *

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
bucket = sagemaker_session.default_bucket() # can replace with your own S3 bucket 'privisaa-bucket-virginia'
prefix = 'bert_pytorch_ngc'
runtime_client = boto3.client('runtime.sagemaker')<jupyter_output><empty_output><jupyter_text>Create our training docker containerNow we are going to create a custom docker container based on the NGC Bert container and push it to AWS Elastic Container Registry (ECR). In order to perform this operation, our sagemaker execution role needs to have access to ECR, this can be configured in IAM.<jupyter_code>%%sh

# The name of our algorithm
algorithm_name=bert-ngc-torch-train

chmod +x train
chmod +x serve

account=$(aws sts get-caller-identity --query Account --output text)

# Get the region defined in the current configuration (default to us-west-2 if none defined)
region=$(aws configure get region)
region=${region:-us-east-1}

fullname="${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest"

# If the repository doesn't exist in ECR, create it.

aws ecr describe-repositories --repository-names "${algorithm_name}" > /dev/null 2>&1

if [ $? -ne 0 ]
then
    aws ecr create-repository --repository-name "${algorithm_name}" > /dev/null
fi

# Get the login command from ECR and execute it directly
$(aws ecr get-login --region ${region} --no-include-email)

# Build the docker image locally with the image name and then push it to ECR
# with the full name.

docker build  -t ${algorithm_name} .
docker tag ${algorithm_name} ${fullname}

# some kind of security auth issue with pushing this to ecr, not authorized to perform ecr:InitiateLayerUpload
docker push ${fullname}<jupyter_output><empty_output><jupyter_text>Download and inspect the dataTo get an idea of what the Stanford Question Answering Dataset contains, let's download it locally and look at it.<jupyter_code>!cd data/squad/ && bash squad_download.sh

# load the v2.0 dev set

with open('data/squad/v2.0/dev-v2.0.json', 'r') as f:
    squad_data = json.load(f)<jupyter_output><empty_output><jupyter_text>Now that we've loaded some of the data, you can use the below block to look at a random context, question, and answer<jupyter_code>ind = random.randint(0,34)
sq = squad_data['data'][ind]
print('Paragraph title: ',sq['title'], '\n')
print(sq['paragraphs'][0]['context'],'\n')
print('Question:', sq['paragraphs'][0]['qas'][0]['question'])
print('Answer:', sq['paragraphs'][0]['qas'][0]['answers'][0]['text'])<jupyter_output><empty_output><jupyter_text>View BERT inputBERT needs us to transform our text data into a numeric representation known as tokens. There are a variety of tokenizers available, we are going to use a tokenizer specially designed for BERT that we will instantiate with our vocabulary file. Let's take a look at our transformed question and context we will be supplying BERT for inference.<jupyter_code>doc_tokens = sq['paragraphs'][0]['context'].split()
tokenizer = BertTokenizer('vocab', do_lower_case=True, max_len=512)
query_tokens = tokenizer.tokenize(sq['paragraphs'][0]['qas'][0]['question'])

feature = preprocess_tokenized_text(doc_tokens, 
                                    query_tokens, 
                                    tokenizer, 
                                    max_seq_length=384, 
                                    max_query_length=64)

tensors_for_inference, tokens_for_postprocessing = feature
tokens_for_postprocessing
tensors_for_inference<jupyter_output><empty_output><jupyter_text>Send data to s3<jupyter_code>!aws s3 cp --recursive data/squad s3://{bucket}/{prefix}
s3train = f's3://{bucket}/{prefix}/v1.1/train-v1.1.json'

train = sagemaker.session.s3_input(s3train, distribution='FullyReplicated', 
                        content_type=None, s3_data_type='S3Prefix')

data_channels = {'train': train}<jupyter_output>'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.<jupyter_text>Instantiate the modelNow we are going to instantiate our model, here we are going to specify our hyperparameters for training as well as the number of GPUs we are going to use. The ml.p3dn.24xlarge instances contain 8 V100 volta GPUs, making them ideal for heavy duty deep learning training. Once we have set our hyperparameters, we will instantiate a Sagemaker Estimator that we will use to run our training job. We specify the Docker image we just pushed to ECR as well as an entrypoint giving instructions for what operations our container should perform when it starts up. Our Docker container has two commands, train and serve. When we instantiate a training job, behind the scenes Sagemaker is running our Docker container and telling it to run the train command.<jupyter_code>account=!aws sts get-caller-identity --query Account --output text

# Get the region defined in the current configuration (default to us-west-2 if none defined)
region=!aws configure get region

algoname = 'bert-ngc-torch-train'

fullname="{}.dkr.ecr.{}.amazonaws.com/{}".format(account[0], region[0], algoname)

fullname
# set our hyperparameters
hyperparameters = {'bert_model': 'bert-base-uncased',  'num_train_epochs': 2, 
                   'vocab_file': '/workspace/bert/data/bert_vocab.txt',
                   'config_file':'/workspace/bert/bert_config.json', 
                  'output_dir': '/opt/ml/model',
                  'train_file': '/opt/ml/input/data/train/train-v1.1.json', #'/workspace/bert/data/squad/v1.1/train-v1.1.json',
                  'num_gpus':8, 'train_batch_size':32, 'max_seq_length':512, 'doc_stride':128, 'seed':1,
                  'learning_rate':3e-5,
                  'save_to_s3':bucket}

# instantiate model
torch_model = PyTorch( role=role,
                      train_instance_count=1,
                      train_instance_type='ml.p3dn.24xlarge',
                      entry_point='transform_script.py',
                      image_name=fullname,
                      framework_version='1.5.0',
                      hyperparameters=hyperparameters
                     )<jupyter_output>Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.<jupyter_text>Fine-tune the modelIf you use an instance with 4 GPUs and a batch size of 4 this process will take ~15 minutes to complete for this particular finetuning task with 2 epochs. Each additional epoch will add another 7 or so minutes. It's recommended to at minimum use a training instance with 4 GPUs, although you will likely get better performance with one of the ml.p3.16xlarge or ml.p3dn.24xlarge instances.<jupyter_code>torch_model.fit(data_channels)<jupyter_output><empty_output><jupyter_text>Deploy our trained modelNow that we've finetuned our base BERT model, what now? Let's deploy our trained model to an endpoint and ask it some questions!<jupyter_code>endpoint_name = 'bert-endpoint-byoc-150'

# if deploying from a model you trained in the same session 
#     bert_end = torch_model.deploy(instance_type='ml.g4dn.4xlarge', initial_instance_count=1, 
#                           endpoint_name=endpoint_name)

model_data = f's3://{bucket}/model.tar.gz'

# We are going to use a SageMaker serving container
torch_model = PyTorchModel(model_data=model_data,
                       role=role,
                      entry_point='transform_script.py',
                      framework_version='1.5.0')
bert_end = torch_model.deploy(instance_type='ml.g4dn.4xlarge', initial_instance_count=1, 
                          endpoint_name=endpoint_name)
context = """
NGC Containers are designed to enable a software platform centered around minimal OS requirements, 
Docker and driver installation on the server or workstation, and provisioning of all application and SDK software 
in the NGC containers through the NGC container registry. NGC manages a catalog of fully integrated and optimized 
deep learning framework containers that take full advantage of NVIDIA GPUs in both single GPU and 
multi-GPU configurations. 
"""

amzn_context = """
Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning (ML) models quickly. SageMaker removes the heavy lifting from each step of the machine learning process to make it easier to develop high quality models.
Traditional ML development is a complex, expensive, iterative process made even harder because there are no integrated 
tools for the entire machine learning workflow. You need to stitch together tools and workflows, which is time-consuming
and error-prone. SageMaker solves this challenge by providing all of the components used for machine learning in a single 
toolset so models get to production faster with much less effort and at lower cost.
"""

question = "What configurations can NGC containers work with?"
question = "What do NGC containers take advantage of?"

amzn_question = "How does SageMaker solve traditional ML Development?"<jupyter_output><empty_output><jupyter_text>Now that our endpoint has been deployed, let's send it some requests!<jupyter_code>%%time

# context='Danielle is a girl who really loves her cat, Steve. Steve is a large cat with a very furry belly. He gets very excited by the prospect of eating chicken covered in gravy.'
# question='who loves Steve?'  # 'What kind of food does Steve like?'

pass_in_data = {'context':context, 'question':question}
json_data = json.dumps(pass_in_data)


# if model_data:
response = runtime_client.invoke_endpoint(EndpointName=endpoint_name,
                                       ContentType='application/json',
                                       Body=json_data)
response = eval(response['Body'].read().decode('utf-8'))
doc_tokens = context.split()
tokenizer = BertTokenizer('vocab', do_lower_case=True, max_len=512)
query_tokens = tokenizer.tokenize(question)
feature = preprocess_tokenized_text(doc_tokens, 
                                    query_tokens, 
                                    tokenizer, 
                                    max_seq_length=384, 
                                    max_query_length=64)
tensors_for_inference, tokens_for_postprocessing = feature
response = get_predictions(doc_tokens, tokens_for_postprocessing, 
                         response[0], response[1], n_best_size=1, 
                         max_answer_length=64, do_lower_case=True, 
                         can_give_negative_answer=True, 
                         null_score_diff_threshold=-11.0)

#response = bert_end.predict(json.dumps(pass_in_data), initial_args={'ContentType':'application/json'}) 

# print result
print('-------------------------------------------------------------------------------------')
print(f'{question} : {response[0]["text"]}')
print('-------------------------------------------------------------------------------------')<jupyter_output>-------------------------------------------------------------------------------------
In what country is Normandy located? : France
-------------------------------------------------------------------------------------
CPU times: user 74.4 ms, sys: 3.91 ms, total: 78.3 ms
Wall time: 176 ms<jupyter_text>Cleanup<jupyter_code>!rm bert_base.pt
!rm s3_bucket.txt
bert_end.delete_endpoint()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10625774473358117, 'fraction_numerical': 0.012004337050805452, 'mean_word_length': 3.4961699164345403, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '663230', 'license': 'MIT-0', 'n_tokens_mistral': 3685, 'n_tokens_neox': 3520, 'n_words': 1239, 'path': 'PyTorch_BYOC_BERT_Finetuning/bert_pytorch_ngc_finetuning.ipynb', 'repo_name': 'Jiahong-Nvidia/amazon-sagemaker-nvidia-ngc-examples', 'size': '12912'}