<jupyter_start><jupyter_text>Evaluate Model with Amazon SageMaker Processing Jobs and Scikit-LearnOften, distributed data processing frameworks such as Scikit-Learn are used to pre-process data sets in order to prepare them for training.  In this notebook we'll use Amazon SageMaker Processing, and leverage the power of Scikit-Learn in a managed SageMaker environment to run our processing workload.  NOTE:  THIS NOTEBOOK WILL TAKE A 5-10 MINUTES TO COMPLETE. PLEASE BE PATIENT. ![](img/prepare_dataset_bert.png)![](img/processing.jpg)  Contents1. Setup Environment1. Setup Input Data1. Setup Output Data1. Build a Spark container for running the processing job1. Run the Processing Job using Amazon SageMaker1. Inspect the Processed Output Data  Setup EnvironmentLet's start by specifying:* The S3 bucket and prefixes that you use for training and model data. Use the default bucket specified by the Amazon SageMaker session.* The IAM role ARN used to give processing and training access to the dataset.<jupyter_code>import sagemaker
import boto3

sess = sagemaker.Session()
role = sagemaker.get_execution_role()
bucket = sess.default_bucket()
region = boto3.Session().region_name

sm = boto3.Session().client(service_name="sagemaker", region_name=region)
%store -r training_job_name
try:
    training_job_name
    print("[OK]")
except NameError:
    print("+++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the previous TRAIN section before you continue.")
    print("+++++++++++++++++++++++++++++++")
# training_job_name='tensorflow-training-2021-01-02-06-07-04-440'
print(training_job_name)
%store -r raw_input_data_s3_uri
try:
    raw_input_data_s3_uri
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(raw_input_data_s3_uri)
%store -r max_seq_length
try:
    max_seq_length
    print("[OK]")
except NameError:
    print("+++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the previous TRAIN section before you continue.")
    print("+++++++++++++++++++++++++++++++")
print(max_seq_length)
%store -r experiment_name
try:
    experiment_name
    print("[OK]")
except NameError:
    print("+++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the previous TRAIN section before you continue.")
    print("+++++++++++++++++++++++++++++++")
print(experiment_name)
%store -r trial_name
try:
    trial_name
    print("[OK]")
except NameError:
    print("+++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the previous TRAIN section before you continue.")
    print("+++++++++++++++++++++++++++++++")
print(trial_name)
print(training_job_name)
from sagemaker.tensorflow.estimator import TensorFlow

describe_training_job_response = sm.describe_training_job(TrainingJobName=training_job_name)
print(describe_training_job_response)
model_dir_s3_uri = describe_training_job_response["ModelArtifacts"]["S3ModelArtifacts"].replace("model.tar.gz", "")
model_dir_s3_uri<jupyter_output><empty_output><jupyter_text>Run the Processing Job using Amazon SageMakerNext, use the Amazon SageMaker Python SDK to submit a processing job using our custom python script.  Create the `Experiment Config`<jupyter_code>experiment_config = {
    "ExperimentName": experiment_name,
    "TrialName": trial_name,
    "TrialComponentDisplayName": "evaluate",
}<jupyter_output><empty_output><jupyter_text>Set the Processing Job Hyper-Parameters<jupyter_code>processing_instance_type = "ml.m5.xlarge"
processing_instance_count = 1<jupyter_output><empty_output><jupyter_text>Choosing a `max_seq_length` for BERTSince a smaller `max_seq_length` leads to faster training and lower resource utilization, we want to find the smallest review length that captures `80%` of our reviews.Remember our distribution of review lengths from a previous section?```mean         51.683405std         107.030844min           1.00000010%           2.00000020%           7.00000030%          19.00000040%          22.00000050%          26.00000060%          32.00000070%          43.00000080%          63.00000090%         110.000000100%       5347.000000max        5347.000000```![](img/review_word_count_distribution.png)Review length `63` represents the `80th` percentile for this dataset.  However, it's best to stick with powers-of-2 when using BERT.  So let's choose `64` as this is the smallest power-of-2 greater than `63`.  Reviews with length > `64` will be truncated to `64`.<jupyter_code>from sagemaker.sklearn.processing import SKLearnProcessor

processor = SKLearnProcessor(
    framework_version="0.23-1",
    role=role,
    instance_type=processing_instance_type,
    instance_count=processing_instance_count,
    max_runtime_in_seconds=7200,
)
from sagemaker.processing import ProcessingInput, ProcessingOutput

processor.run(
    code="evaluate_model_metrics.py",
    inputs=[
        ProcessingInput(
            input_name="model-tar-s3-uri", source=model_dir_s3_uri, destination="/opt/ml/processing/input/model/"
        ),
        ProcessingInput(
            input_name="evaluation-data-s3-uri",
            source=raw_input_data_s3_uri,
            destination="/opt/ml/processing/input/data/",
        ),
    ],
    outputs=[
        ProcessingOutput(s3_upload_mode="EndOfJob", output_name="metrics", source="/opt/ml/processing/output/metrics"),
    ],
    arguments=["--max-seq-length", str(max_seq_length)],
    experiment_config=experiment_config,
    logs=True,
    wait=False,
)
scikit_processing_job_name = processor.jobs[-1].describe()["ProcessingJobName"]
print(scikit_processing_job_name)
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://console.aws.amazon.com/sagemaker/home?region={}#/processing-jobs/{}">Processing Job</a></b>'.format(
            region, scikit_processing_job_name
        )
    )
)
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://console.aws.amazon.com/cloudwatch/home?region={}#logStream:group=/aws/sagemaker/ProcessingJobs;prefix={};streamFilter=typeLogStreamPrefix">CloudWatch Logs</a> After About 5 Minutes</b>'.format(
            region, scikit_processing_job_name
        )
    )
)
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://s3.console.aws.amazon.com/s3/buckets/{}/{}/?region={}&tab=overview">S3 Output Data</a> After The Processing Job Has Completed</b>'.format(
            bucket, scikit_processing_job_name, region
        )
    )
)<jupyter_output><empty_output><jupyter_text>Monitor the Processing Job<jupyter_code>running_processor = sagemaker.processing.ProcessingJob.from_processing_name(
    processing_job_name=scikit_processing_job_name, sagemaker_session=sess
)

processing_job_description = running_processor.describe()

print(processing_job_description)
processing_evaluation_metrics_job_name = processing_job_description["ProcessingJobName"]
print(processing_evaluation_metrics_job_name)
running_processor.wait(logs=False)<jupyter_output><empty_output><jupyter_text>_Please Wait Until the ^^ Processing Job ^^ Completes Above._  Inspect the Processed Output DataTake a look at a few rows of the transformed dataset to make sure the processing was successful.<jupyter_code>processing_job_description = running_processor.describe()

output_config = processing_job_description["ProcessingOutputConfig"]
for output in output_config["Outputs"]:
    if output["OutputName"] == "metrics":
        processed_metrics_s3_uri = output["S3Output"]["S3Uri"]

print(processed_metrics_s3_uri)
!aws s3 ls $processed_metrics_s3_uri/<jupyter_output><empty_output><jupyter_text>Pass Variables to the Next Notebook(s)<jupyter_code>%store processing_evaluation_metrics_job_name
%store processed_metrics_s3_uri
%store<jupyter_output><empty_output><jupyter_text>Show the Experiment Tracking Lineage<jupyter_code>from sagemaker.analytics import ExperimentAnalytics

import pandas as pd

pd.set_option("max_colwidth", 500)
# pd.set_option("max_rows", 100)

experiment_analytics = ExperimentAnalytics(
    sagemaker_session=sess, experiment_name=experiment_name, sort_by="CreationTime", sort_order="Descending"
)

experiment_analytics_df = experiment_analytics.dataframe()
experiment_analytics_df
trial_component_name = experiment_analytics_df.TrialComponentName[0]
print(trial_component_name)
trial_component_description = sm.describe_trial_component(TrialComponentName=trial_component_name)
trial_component_description
from sagemaker.lineage.visualizer import LineageTableVisualizer

lineage_table_viz = LineageTableVisualizer(sess)
lineage_table_viz_df = lineage_table_viz.show(processing_job_name=processing_evaluation_metrics_job_name)
lineage_table_viz_df<jupyter_output><empty_output><jupyter_text>Release Resources<jupyter_code>%%html

<p><b>Shutting down your kernel for this notebook to release resources.</b></p>
<button class="sm-command-button" data-commandlinker-command="kernelmenu:shutdown" style="display:none;">Shutdown Kernel</button>
        
<script>
try {
    els = document.getElementsByClassName("sm-command-button");
    els[0].click();
}
catch(err) {
    // NoOp
}    
</script>
%%javascript

try {
    Jupyter.notebook.save_checkpoint();
    Jupyter.notebook.session.delete();
}
catch(err) {
    // NoOp
}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13558091286307053, 'fraction_numerical': 0.023132780082987553, 'mean_word_length': 5.5274204468517265, 'pattern_counts': {'":': 4, '<': 57, '<?xml version=': 0, '>': 58, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '298291', 'license': 'Apache-2.0', 'n_tokens_mistral': 3135, 'n_tokens_neox': 2875, 'n_words': 647, 'path': '07_train/04_Evaluate_Model_Metrics.ipynb', 'repo_name': 'NRauschmayr/workshop', 'size': '9640'}