<jupyter_start><jupyter_text>Detecting Data Drift in NLP using SageMaker Custom Model MonitorThis Example is an extension of [Fine-tuning a PyTorch BERT model and deploying it with Amazon Elastic Inference on Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/fine-tuning-a-pytorch-bert-model-and-deploying-it-with-amazon-elastic-inference-on-amazon-sagemaker/) aws blog post. We will use the dataset and model thats outlined in this blog and extend it to demo custom model monitoring capability using [SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html)Detecting data drift in NLP is a challenging task. Model monitoring becomes an important aspect in MLOPs because the change in data distribution during inference time can cause Model decay. ML models are probabilistic and trained on certain corpus of historical data. Drift is distribution change between the training and deployment data, which is concerning if model performance changes.We will begin with creating PyTorch Model using previously trained model artifacts. We will deploy the model to a SageMaker real time endpoint. To establish a baseline of training data distribution we will caluclate BERT sentence embedding and use that in the custom model monitoring scripts to compare the real time inferece traffic to compare a distance metrics to determine the deviation fron training distribution ![title](images/nlp-data-drift-bert.png)  SetupTo start, we import some Python libraries and initialize a SageMaker session, S3 bucket and prefix, and IAM role.<jupyter_code># need torch 1.3.1 for elastic inference
!python -m pip install --upgrade pip
!pip install torch==1.3.1 --quiet
!pip install transformers --quiet
import os
import numpy as np
import pandas as pd
import sagemaker

sagemaker_session = sagemaker.Session()

bucket = sagemaker_session.default_bucket()
model_prefix = "sagemaker/nlp-data-drift-bert-model"

role = sagemaker.get_execution_role()<jupyter_output><empty_output><jupyter_text>Prepare training dataWe use Corpus of Linguistic Acceptability (CoLA) (https://nyu-mll.github.io/CoLA/), a dataset of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. We download and unzip the data using the following code:  Download data<jupyter_code>if not os.path.exists("./cola_public_1.1.zip"):
    !curl -o ./cola_public_1.1.zip https://nyu-mll.github.io/CoLA/cola_public_1.1.zip
if not os.path.exists("./cola_public/"):
    !unzip cola_public_1.1.zip<jupyter_output><empty_output><jupyter_text>Get sentences and labelsLet us take a quick look at our data. First we read in the training data. The only two columns we need are the sentence itself and its label.<jupyter_code>df = pd.read_csv(
    "./cola_public/raw/in_domain_train.tsv",
    sep="\t",
    header=None,
    usecols=[1, 3],
    names=["label", "sentence"],
)
sentences = df.sentence.values
labels = df.label.values
df.head()<jupyter_output><empty_output><jupyter_text>Printing out a few sentences shows us how sentences are labeled based on their grammatical completeness.<jupyter_code>print(sentences[20:25])
print(labels[20:25])<jupyter_output>['The professor talked us.' 'We yelled ourselves hoarse.'
 'We yelled ourselves.' 'We yelled Harry hoarse.'
 'Harry coughed himself into a fit.']
[0 1 0 0 1]<jupyter_text>We then split the dataset for training and testing.<jupyter_code>from sklearn.model_selection import train_test_split

train, test = train_test_split(df)
train.to_csv("./cola_public/train.csv", index=False)
test.to_csv("./cola_public/test.csv", index=False)<jupyter_output><empty_output><jupyter_text>Next, we upload both to Amazon S3 for use later. The SageMaker Python SDK provides a helpful function for uploading to Amazon S3:<jupyter_code>inputs_train = sagemaker_session.upload_data("./cola_public/train.csv", bucket=bucket, key_prefix=model_prefix)
inputs_test = sagemaker_session.upload_data("./cola_public/test.csv", bucket=bucket, key_prefix=model_prefix)<jupyter_output><empty_output><jupyter_text>Run training  Training scriptWe use the [PyTorch-Transformers library](https://pytorch.org/hub/huggingface_pytorch-transformers), which contains PyTorch implementations and pre-trained model weights for many NLP models, including BERT.Our training script should save model artifacts learned during training to a file path called `model_dir`, as stipulated by the SageMaker PyTorch image. Upon completion of training, model artifacts saved in `model_dir` will be uploaded to S3 by SageMaker and will become available in S3 for deployment.We save this script in a file named `train_deploy.py`, and put the file in a directory named `code/`. The full training script can be viewed under `code/`.<jupyter_code>!pygmentize code/train_deploy.py<jupyter_output>[34mimport[39;49;00m [04m[36margparse[39;49;00m
[34mimport[39;49;00m [04m[36mjson[39;49;00m
[34mimport[39;49;00m [04m[36mlogging[39;49;00m
[34mimport[39;49;00m [04m[36mos[39;49;00m
[34mimport[39;49;00m [04m[36msys[39;49;00m

[34mimport[39;49;00m [04m[36mnumpy[39;49;00m [34mas[39;49;00m [04m[36mnp[39;49;00m
[34mimport[39;49;00m [04m[36mpandas[39;49;00m [34mas[39;49;00m [04m[36mpd[39;49;00m
[34mimport[39;49;00m [04m[36mtorch[39;49;00m
[34mimport[39;49;00m [04m[36mtransformers[39;49;00m
[34mimport[39;49;00m [04m[36mtorch[39;49;00m[04m[36m.[39;49;00m[04m[36mdistributed[39;49;00m [34mas[39;49;00m [04m[36mdist[39;49;00m
[34mimport[39;49;00m [04m[36mtorch[39;49;00m[04m[36m.[39;49;00m[04m[36mutils[39;49;00m[04m[36m.[39;49;00m[04m[36mdata[39;49;00m
[34mimport[39;49;00m [04m[36mtorch[39;49;00m[04m[36m.[39;49;00m[04m[36mutils[39;49;00m[04m[36m.[39;49;00m[04m[36mdata[39;49;00m[04m[...]<jupyter_text>Train on Amazon SageMakerWe use Amazon SageMaker to train and deploy a model using our custom PyTorch code. The Amazon SageMaker Python SDK makes it easier to run a PyTorch script in Amazon SageMaker using its PyTorch estimator. After that, we can use the SageMaker Python SDK to deploy the trained model and run predictions. For more information on how to use this SDK with PyTorch, see [the SageMaker Python SDK documentation](https://sagemaker.readthedocs.io/en/stable/using_pytorch.html).To start, we use the `PyTorch` estimator class to train our model. When creating our estimator, we make sure to specify a few things:* `entry_point`: the name of our PyTorch script. It contains our training script, which loads data from the input channels, configures training with hyperparameters, trains a model, and saves a model. It also contains code to load and run the model during inference.* `source_dir`: the location of our training scripts and requirements.txt file. "requirements.txt" lists packages you want to use with your script.* `framework_version`: the PyTorch version we want to useThe PyTorch estimator supports multi-machine, distributed PyTorch training. To use this, we just set train_instance_count to be greater than one. Our training script supports distributed training for only GPU instances. After creating the estimator, we then call fit(), which launches a training job. We use the Amazon S3 URIs where we uploaded the training data earlier.<jupyter_code>from sagemaker.pytorch import PyTorch

# place to save model artifact
output_path = f"s3://{bucket}/{model_prefix}"

estimator = PyTorch(
    entry_point="train_deploy.py",
    source_dir="code",
    role=role,
    framework_version="1.3.1",
    py_version="py3",
    instance_count=1,  # this script only support distributed training for GPU instances.
    instance_type="ml.p3.2xlarge",
    output_path=output_path,
    hyperparameters={
        "epochs": 1,
        "num_labels": 2,
        "backend": "gloo",
    },
    disable_profiler=True, # disable debugger
)
estimator.fit({"training": inputs_train, "testing": inputs_test})<jupyter_output>2021-08-12 21:14:46 Starting - Starting the training job...
2021-08-12 21:14:49 Starting - Launching requested ML instances......
2021-08-12 21:16:04 Starting - Preparing the instances for training............
2021-08-12 21:17:53 Downloading - Downloading input data...
2021-08-12 21:18:26 Training - Downloading the training image......
2021-08-12 21:19:45 Training - Training image download completed. Training in progress..[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device[0m
[34mbash: no job control in this shell[0m
[34m2021-08-12 21:19:46,535 sagemaker-containers INFO     Imported framework sagemaker_pytorch_container.training[0m
[34m2021-08-12 21:19:46,561 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.[0m
[34m2021-08-12 21:19:49,576 sagemaker_pytorch_container.training INFO     Invoking user training script.[0m
[34m2021-08-12 21:19:49,888 sagemaker-containers INFO     Module default_user_module_name doe[...]<jupyter_text>Host After training our model, we host it on an Amazon SageMaker Endpoint. To make the endpoint load the model and serve predictions, we implement a few methods in `train_deploy.py`.* `model_fn()`: function defined to load the saved model and return a model object that can be used for model serving. The SageMaker PyTorch model server loads our model by invoking model_fn.* `input_fn()`: deserializes and prepares the prediction input. In this example, our request body is first serialized to JSON and then sent to model serving endpoint. Therefore, in `input_fn()`, we first deserialize the JSON-formatted request body and return the input as a `torch.tensor`, as required for BERT.* `predict_fn()`: performs the prediction and returns the result.To deploy our endpoint, we call `deploy()` on our PyTorch estimator object, passing in our desired number of instances and instance type:  Enable Model Monitor Data Capture on the SageMaker Endpoint<jupyter_code>from sagemaker.model_monitor import DataCaptureConfig

#s3_capture_upload_path = f's3://{sagemaker_session.default_bucket()}/{s3_prefix}/endpoint/data_capture'
prefix = "sagemaker/CustomModelMonitor"
data_capture_prefix = "{}/datacapture".format(prefix)
s3_capture_upload_path = "s3://{}/{}".format(bucket, data_capture_prefix)

print(s3_capture_upload_path)

data_capture_config = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri=s3_capture_upload_path
)
predictor = estimator.deploy(endpoint_name='nlp-data-drift-bert-endpoint',
                             initial_instance_count=1, 
                             instance_type="ml.m4.xlarge",
                             data_capture_config=data_capture_config)
endpoint_name = predictor.endpoint_name
print(endpoint_name)<jupyter_output>nlp-data-drift-bert-endpoint<jupyter_text>Predictions using the model<jupyter_code>predictor.serializer = sagemaker.serializers.JSONSerializer()
predictor.deserializer = sagemaker.deserializers.JSONDeserializer()
# batch inference 

print("Sending test traffic to the endpoint {}. \nPlease wait...".format(endpoint_name))

result = predictor.predict([
    "CLI to download the zip file", 
    "Thanks so much for driving me home",
    "construct the sub-embeddings and corresponding baselines",
    "our Bert model and interpret what the model",
    "Bert models using Captum library",
    "case study we focus on a fine-tuned Question Answering model on SQUAD datase",
    "we pretrain the model, we can load ",
    "need to define baselines / references, nu",
    "defines numericalized special tokens ",
    "Thanks so much for cooking dinner. I really appreciate it",
    "let's define the ground truth for prediction's start and en",
    "pre-computation of embeddings for the second option is necessary because",
    "to summarize attributions for each word token in the sequence.",
    "Nice to meet you, Sergio. So, where are you from"
])

print("Done!")
print("Predicted class: ", np.argmax(result, axis=1))<jupyter_output>Predicted class:  [0 1 1 0 1 1 1 0 1 1 0 1 1 1]<jupyter_text>View Captured Data<jupyter_code>#Note: It takes a few minutes for the capture data to appear in S3

import boto3

s3_client = boto3.Session().client('s3')

current_endpoint_capture_prefix = "{}/{}".format(data_capture_prefix, endpoint_name)
result = s3_client.list_objects(Bucket=bucket, Prefix=current_endpoint_capture_prefix)
capture_files = [capture_file.get("Key") for capture_file in result.get("Contents")]
print("Found Capture Files:")
print("\n ".join(capture_files))
def get_obj_body(obj_key):
    return s3_client.get_object(Bucket=bucket, Key=obj_key).get('Body').read().decode("utf-8")

capture_file = get_obj_body(capture_files[-1])
print(capture_file[:2000])
import json

print(json.dumps(json.loads(capture_file.split('\n')[0]), indent=2))<jupyter_output>{
  "captureData": {
    "endpointInput": {
      "observedContentType": "application/json",
      "mode": "INPUT",
      "data": "[\"CLI to download the zip file\", \"Thanks so much for driving me home\", \"construct the sub-embeddings and corresponding baselines\", \"our Bert model and interpret what the model\", \"Bert models using Captum library\", \"case study we focus on a fine-tuned Question Answering model on SQUAD datase\", \"we pretrain the model, we can load \", \"need to define baselines / references, nu\", \"defines numericalized special tokens \", \"Thanks so much for cooking dinner. I really appreciate it\", \"let's define the ground truth for prediction's start and en\", \"pre-computation of embeddings for the second option is necessary because\", \"to summarize attributions for each word token in the sequence.\", \"Nice to meet you, Sergio. So, where are you from\"]",
      "encoding": "JSON"
    },
    "endpointOutput": {
      "observedContentType": "application/json[...]<jupyter_text>Build an Embedding Matrix from Input DatasetBERT developers created two main models:    The BASE: Number of transformer blocks (L): 12, Hidden layer size (H): 768 and Attention heads(A): 12    The LARGE: Number of transformer blocks (L): 24, Hidden layer size (H): 1024 and Attention heads(A): 16In this post, I will be using the BASE model as it is smaller and easier to train.At this point, to make things clearer it is important to understand the special tokens that BERT authors used for fine-tuning and specific task training. These are the following:    [CLS] : The first token of every sequence. A classification token which is normally used in conjunction with a softmax layer for classification tasks. For anything else, it can be safely ignored.    [SEP] : A sequence delimiter token which was used at pre-training for sequence-pair tasks (i.e. Next sentence prediction). Must be used when sequence pair tasks are required. When a single sequence is used it is just appended at the end.    [MASK] : Token used for masked words. Only used for pre-training.    The input layer is simply the vector of the sequence tokens along with the special tokens. The “ing” token in the example above may raise some eyebrows so to clarify, BERT utilizes WordPiece [6] for tokenization which in effect, splits token like “playing” to “play” and “ing”. This is mainly to cover a wider spectrum of Out-Of-Vocabulary (OOV) words.Token embeddings are the vocabulary IDs for each of the tokens.Sentence Embeddings is just a numeric class to distinguish between sentence A and B.And lastly, Transformer positional embeddings indicate the position of each word in the sequence<jupyter_code>from transformers import BertTokenizer, BertModel
import torch

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased',
                                  output_hidden_states = True, # Whether the model returns all hidden-states.
                                  )

# Put the model in "evaluation" mode, meaning feed-forward operation.
model.eval()

# Tokenize all of the sentences and map the tokens to thier word IDs.
input_ids = []
attention_masks = []
sentence_embeddings = []

# For every sentence...
for sent in sentences:
    # `encode_plus` will:
    #   (1) Tokenize the sentence.
    #   (2) Prepend the `[CLS]` token to the start.
    #   (3) Append the `[SEP]` token to the end.
    #   (4) Map tokens to their IDs.
    #   (5) Pad or truncate the sentence to `max_length`
    #   (6) Create attention masks for [PAD] tokens.
    encoded_dict = tokenizer.encode_plus(
                        sent,                      # Sentence to encode.
                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                        max_length = 64,           # Pad & truncate all sentences.
                        pad_to_max_length = True,
                        return_attention_mask = True,   # Construct attn. masks.
                        return_tensors = 'pt',     # Return pytorch tensors.
                   )
    
    with torch.no_grad():
        outputs = model(encoded_dict['input_ids'], encoded_dict['attention_mask'])
        hidden_states = outputs[2]
        token_vecs = hidden_states[-2][0]
        sentence_embedding = torch.mean(token_vecs, dim=0)
        sentence_embeddings.append(sentence_embedding)
sentence_embeddings_list = []

for i in sentence_embeddings:
    sentence_embeddings_list.append(i.numpy())<jupyter_output><empty_output><jupyter_text>Save the embeddingd as .npy file<jupyter_code>np.save('embeddings.npy', sentence_embeddings_list)
sentence_embeddings_list[0]<jupyter_output><empty_output><jupyter_text>Upload the sentence embedding to S3<jupyter_code>!aws s3 cp embeddings.npy s3://{bucket}/{model_prefix}/embeddings/<jupyter_output>upload: ./embeddings.npy to s3://sagemaker-us-east-1-757967535041/sagemaker/nlp-data-drift-bert-model/embeddings/embeddings.npy<jupyter_text>Evaluation ScriptAmazon SageMaker Model Monitor provides a prebuilt container with ability to analyze the data captured from endpoints for tabular datasets. If you would like to bring your own container, Model Monitor provides extension points which you can leverage. Under the hood, when you create a MonitoringSchedule, Model Monitor ultimately kicks off processing jobs. Hence the container needs to be aware of the processing job contract We need to create an evaluation script that is compatible with container contract inputs and outputs[Container Contract Inputs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-byoc-contract-inputs.html)[Container Contract Outputs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-byoc-contract-outputs.html)<jupyter_code>!pygmentize docker/evaluation.py<jupyter_output>[33m"""Custom Model Monitoring script for Detecting Data Drift in NLP using SageMaker Model Monitor[39;49;00m
[33m"""[39;49;00m

[37m# Python Built-Ins:[39;49;00m
[34mfrom[39;49;00m [04m[36mcollections[39;49;00m [34mimport[39;49;00m defaultdict
[34mimport[39;49;00m [04m[36mdatetime[39;49;00m
[34mimport[39;49;00m [04m[36mjson[39;49;00m
[34mimport[39;49;00m [04m[36mos[39;49;00m
[34mimport[39;49;00m [04m[36mtraceback[39;49;00m
[34mfrom[39;49;00m [04m[36mtypes[39;49;00m [34mimport[39;49;00m SimpleNamespace

[37m# External Dependencies:[39;49;00m
[34mimport[39;49;00m [04m[36mnumpy[39;49;00m [34mas[39;49;00m [04m[36mnp[39;49;00m
[34mimport[39;49;00m [04m[36mboto3[39;49;00m
[34mfrom[39;49;00m [04m[36mscipy[39;49;00m[04m[36m.[39;49;00m[04m[36mspatial[39;49;00m[04m[36m.[39;49;00m[04m[36mdistance[39;49;00m [34mimport[39;49;00m cosine
[34mfrom[39;49;00m [04m[36mtransformers[39;49;00m [34mimport[39;49[...]<jupyter_text>Build and Push Image to ECR<jupyter_code>import boto3

account_id = boto3.client('sts').get_caller_identity().get('Account')
ecr_repository = 'nlp-data-drift-bert-v1'
tag = ':latest'

region = boto3.session.Session().region_name

sm = boto3.client('sagemaker')

uri_suffix = 'amazonaws.com'
if region in ['cn-north-1', 'cn-northwest-1']:
    uri_suffix = 'amazonaws.com.cn'
processing_repository_uri = f'{account_id}.dkr.ecr.{region}.{uri_suffix}/{ecr_repository + tag}'
# Creating the ECR repository and pushing the container image

# SageMaker Classic Notebook Instance:
!docker build -t $ecr_repository docker
!$(aws ecr get-login --region $region --registry-ids $account_id --no-include-email)
!aws ecr create-repository --repository-name $ecr_repository
!docker tag {ecr_repository + tag} $processing_repository_uri
!docker push $processing_repository_uri

# SageMaker Studio:
# !cd docker && sm-docker build . --repository $ecr_repository$tag<jupyter_output>Sending build context to Docker daemon  17.41kB
Step 1/9 : FROM python:3.7-slim-buster
 ---> b5900b90787e
Step 2/9 : RUN pip3 install sagemaker
 ---> Using cache
 ---> fccd9f2a7991
Step 3/9 : RUN pip3 install scipy
 ---> Using cache
 ---> c247162d8522
Step 4/9 : RUN pip3 install transformers
 ---> Using cache
 ---> c0fa548d853f
Step 5/9 : RUN pip3 install torch
 ---> Using cache
 ---> 5d959178e287
Step 6/9 : RUN pip3 install s3fs
 ---> Using cache
 ---> c31974fc2b7a
Step 7/9 : ENV PYTHONUNBUFFERED=TRUE
 ---> Using cache
 ---> 9fa91339885b
Step 8/9 : ADD evaluation.py /
 ---> b64cf3d05d5f
Step 9/9 : ENTRYPOINT ["python3", "/evaluation.py"]
 ---> Running in 567a3862a0bd
Removing intermediate container 567a3862a0bd
 ---> cacf6ab9c602
Successfully built cacf6ab9c602
Successfully tagged nlp-data-drift-bert-v1:latest
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
Configure a cre[...]<jupyter_text>Custom Model Monitor for detetcing data drift<jupyter_code>from sagemaker.model_monitor import ModelMonitor

monitor = ModelMonitor(
    base_job_name='nlp-data-drift-bert-v1',
    role=role,
    image_uri=processing_repository_uri,
    instance_count=1,
    instance_type='ml.m5.large',
    env={ 'THRESHOLD':'0.5', 'bucket': bucket },
)
from sagemaker.model_monitor import CronExpressionGenerator, MonitoringOutput
from sagemaker.processing import ProcessingInput, ProcessingOutput

destination = f's3://{sagemaker_session.default_bucket()}/{prefix}/{endpoint_name}/monitoring_schedule'

processing_output = ProcessingOutput(
    output_name='result',
    source='/opt/ml/processing/resultdata',
    destination=destination,
)
output = MonitoringOutput(source=processing_output.source, destination=processing_output.destination)

monitor.create_monitoring_schedule(
    monitor_schedule_name='nlp-data-drift-bert-schedule',
    output=output,
    endpoint_input=predictor.endpoint_name,
    schedule_cron_expression=CronExpressionGenerator.hourly(),
)
monitor.describe_schedule()
jobs = monitor.list_executions()
jobs
if len(jobs) > 0:
    last_execution_desc = monitor.list_executions()[-1].describe()
    print(last_execution_desc)
    print(f'\nExit Message: {last_execution_desc.get("ExitMessage", "None")}')
else:
    print("""No processing job has been executed yet. 
    This means that one hour has not passed yet. 
    You can go to the next code cell and run the processing job manually""")<jupyter_output>No processing job has been executed yet. 
    This means that one hour has not passed yet. 
    You can go to the next code cell and run the processing job manually<jupyter_text>Manually execute the processing job<jupyter_code>from sagemaker.processing import Processor

processor = Processor(
    base_job_name='nlp-data-drift-bert-v1',
    role=role,
    image_uri=processing_repository_uri,
    instance_count=1,
    instance_type='ml.m5.large',
    env={ 'THRESHOLD':'0.5','bucket': bucket },
)
    
processor.run(
    [ProcessingInput(
        input_name='endpointdata',
        source = "s3://{}/{}/{}".format(bucket, data_capture_prefix,endpoint_name),
        #source=f's3://{sagemaker_session.default_bucket()}/{s3_prefix}/endpoint/data_capture',
        destination = '/opt/ml/processing/input/endpoint',
    )],
    [ProcessingOutput(
        output_name='result',
        source='/opt/ml/processing/resultdata',
        destination=destination,
    )],
)<jupyter_output>Job Name:  nlp-data-drift-bert-v1-2021-08-12-21-51-29-927
Inputs:  [{'InputName': 'endpointdata', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-757967535041/sagemaker/CustomModelMonitor/datacapture/nlp-data-drift-bert-endpoint', 'LocalPath': '/opt/ml/processing/input/endpoint', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]
Outputs:  [{'OutputName': 'result', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-757967535041/sagemaker/CustomModelMonitor/nlp-data-drift-bert-endpoint/monitoring_schedule', 'LocalPath': '/opt/ml/processing/resultdata', 'S3UploadMode': 'EndOfJob'}}]
.............................[34mStarting evaluation with config[0m
[34mnamespace(bucket='sagemaker-us-east-1-757967535041', dataset_format=None, dataset_source='/opt/ml/processing/input/endpoint', end_time=None, max_ratio_threshold=0.5, output_path='/opt/ml/processing/resultdata', publish_cloud[...]<jupyter_text>CleanupLastly, please remember to delete the monitoring schedule and Amazon SageMaker endpoint to avoid charges:<jupyter_code>#Delete the monitoring schedule
monitor.delete_monitoring_schedule()
#Delete endpoint
sm.delete_endpoint(EndpointName=endpoint_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11170171187368091, 'fraction_numerical': 0.04705698428828266, 'mean_word_length': 4.925660027790644, 'pattern_counts': {'":': 13, '<': 66, '<?xml version=': 0, '>': 83, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '156786', 'license': 'MIT-0', 'n_tokens_mistral': 8927, 'n_tokens_neox': 7938, 'n_words': 2357, 'path': 'detect-data-drift-nlp-custom-model-monitor.ipynb', 'repo_name': 'aws-samples/detecting-data-drift-in-nlp-using-amazon-sagemaker-custom-model-monitor', 'size': '25586'}