<jupyter_start><jupyter_text>NOTE:  THIS NOTEBOOK WILL TAKE ABOUT 30 MINUTES TO COMPLETE. PLEASE BE PATIENT.  Fine-Tuning a BERT Model and Create a Text ClassifierIn the previous section, we've already performed the Feature Engineering to create BERT embeddings from the `reviews_body` text using the pre-trained BERT model, and split the dataset into train, validation and test files. To optimize for Tensorflow training, we saved the files in TFRecord format. Now, let’s fine-tune the BERT model to our Customer Reviews Dataset and add a new classification layer to predict the `star_rating` for a given `review_body`.![BERT Training](img/bert_training.png)As mentioned earlier, BERT’s attention mechanism is called a Transformer. This is, not coincidentally, the name of the popular BERT Python library, “Transformers,” maintained by a company called HuggingFace. We will use a variant of BERT called [DistilBert](https://arxiv.org/pdf/1910.01108.pdf) which requires less memory and compute, but maintains very good accuracy on our dataset.<jupyter_code>import boto3
import sagemaker
import pandas as pd

sess = sagemaker.Session()
bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
region = boto3.Session().region_name

sm = boto3.Session().client(service_name="sagemaker", region_name=region)<jupyter_output><empty_output><jupyter_text>_PRE-REQUISITE: You need to have succesfully run the notebooks in the `PREPARE` section before you continue with this notebook._<jupyter_code>%store -r processed_train_data_s3_uri
try:
    processed_train_data_s3_uri
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(processed_train_data_s3_uri)
%store -r processed_validation_data_s3_uri
try:
    processed_validation_data_s3_uri
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(processed_validation_data_s3_uri)
%store -r processed_test_data_s3_uri
try:
    processed_test_data_s3_uri
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(processed_test_data_s3_uri)
%store -r max_seq_length
try:
    max_seq_length
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(max_seq_length)
%store -r experiment_name
try:
    experiment_name
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(experiment_name)
%store -r trial_name
try:
    trial_name
except NameError:
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("[ERROR] Please run the notebooks in the PREPARE section before you continue.")
    print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(trial_name)<jupyter_output><empty_output><jupyter_text>Specify the Dataset in S3We are using the train, validation, and test splits created in the previous section.<jupyter_code>print(processed_train_data_s3_uri)

!aws s3 ls $processed_train_data_s3_uri/
print(processed_validation_data_s3_uri)

!aws s3 ls $processed_validation_data_s3_uri/
print(processed_test_data_s3_uri)

!aws s3 ls $processed_test_data_s3_uri/<jupyter_output><empty_output><jupyter_text>Specify S3 `Distribution Strategy`<jupyter_code>from sagemaker.inputs import TrainingInput

s3_input_train_data = TrainingInput(s3_data=processed_train_data_s3_uri, distribution="ShardedByS3Key")
s3_input_validation_data = TrainingInput(s3_data=processed_validation_data_s3_uri, distribution="ShardedByS3Key")
s3_input_test_data = TrainingInput(s3_data=processed_test_data_s3_uri, distribution="ShardedByS3Key")

print(s3_input_train_data.config)
print(s3_input_validation_data.config)
print(s3_input_test_data.config)<jupyter_output><empty_output><jupyter_text>Setup Hyper-Parameters for Classification Layer<jupyter_code>print(max_seq_length)
epochs = 1
learning_rate = 0.00001
epsilon = 0.00000001
train_batch_size = 128
validation_batch_size = 128
test_batch_size = 128
train_steps_per_epoch = 1
validation_steps = 1
test_steps = 1
train_instance_count = 1
train_instance_type = "ml.c5.9xlarge"
train_volume_size = 1024
use_xla = True
use_amp = True
freeze_bert_layer = False
enable_sagemaker_debugger = True
enable_checkpointing = False
enable_tensorboard = True
input_mode = "File"
run_validation = True
run_test = True
run_sample_predictions = True<jupyter_output><empty_output><jupyter_text>Setup Metrics To Track Model PerformanceThese sample log lines...```45/50 [=====>..] - ETA: 3s - loss: 0.425 - accuracy: 0.88150/50 [=======>] - ETA: 0s - val_loss: 0.407 - val_accuracy: 0.885```...will produce the following 4 metrics in CloudWatch:`loss` = 0.425`accuracy` = 0.881`val_loss` = 0.407`val_accuracy` = 0.885<jupyter_code>metrics_definitions = [
    {"Name": "train:loss", "Regex": "loss: ([0-9\\.]+)"},
    {"Name": "train:accuracy", "Regex": "accuracy: ([0-9\\.]+)"},
    {"Name": "validation:loss", "Regex": "val_loss: ([0-9\\.]+)"},
    {"Name": "validation:accuracy", "Regex": "val_accuracy: ([0-9\\.]+)"},
]<jupyter_output><empty_output><jupyter_text>Setup SageMaker DebuggerDefine Debugger Rules as deccribed here:  https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-built-in-rules.html<jupyter_code>from sagemaker.debugger import Rule
from sagemaker.debugger import rule_configs
from sagemaker.debugger import ProfilerRule
from sagemaker.debugger import CollectionConfig
from sagemaker.debugger import DebuggerHookConfig

actions = rule_configs.ActionList(
    #    rule_configs.StopTraining(),
    #    rule_configs.Email("")
)

rules = [
    Rule.sagemaker(
        base_config=rule_configs.loss_not_decreasing(),
        rule_parameters={
            "collection_names": "losses,metrics",
            "use_losses_collection": "true",
            "num_steps": "10",
            "diff_percent": "50",
        },
        collections_to_save=[
            CollectionConfig(
                name="losses",
                parameters={
                    "save_interval": "10",
                },
            ),
            CollectionConfig(
                name="metrics",
                parameters={
                    "save_interval": "10",
                },
            ),
        ],
        actions=actions,
    ),
    Rule.sagemaker(
        base_config=rule_configs.overtraining(),
        rule_parameters={
            "collection_names": "losses,metrics",
            "patience_train": "10",
            "patience_validation": "10",
            "delta": "0.5",
        },
        collections_to_save=[
            CollectionConfig(
                name="losses",
                parameters={
                    "save_interval": "10",
                },
            ),
            CollectionConfig(
                name="metrics",
                parameters={
                    "save_interval": "10",
                },
            ),
        ],
        actions=actions,
    ),
    ProfilerRule.sagemaker(rule_configs.ProfilerReport()),
    ProfilerRule.sagemaker(rule_configs.BatchSize()),
    ProfilerRule.sagemaker(rule_configs.CPUBottleneck()),
    ProfilerRule.sagemaker(rule_configs.GPUMemoryIncrease()),
    ProfilerRule.sagemaker(rule_configs.IOBottleneck()),
    ProfilerRule.sagemaker(rule_configs.LoadBalancing()),
    ProfilerRule.sagemaker(rule_configs.LowGPUUtilization()),
    ProfilerRule.sagemaker(rule_configs.OverallSystemUsage()),
    #        ProfilerRule.sagemaker(rule_configs.OverallFrameworkMetrics()),
    ProfilerRule.sagemaker(rule_configs.StepOutlier()),
]

hook_config = DebuggerHookConfig(
    hook_parameters={
        "save_interval": "10",  # number of steps
        "export_tensorboard": "true",
        "tensorboard_dir": "hook_tensorboard/",
    }
)<jupyter_output><empty_output><jupyter_text>Specify a Debugger profiler configurationThe following configuration will capture system metrics at 500 milliseconds. The system metrics include utilization per CPU, GPU, memory utilization per CPU, GPU as well I/O and network.Debugger will capture detailed profiling information from step 5 to step 15. This information includes Horovod metrics, dataloading, preprocessing, operators running on CPU and GPU.<jupyter_code>from sagemaker.debugger import ProfilerConfig, FrameworkProfile

profiler_config = ProfilerConfig(
    system_monitor_interval_millis=500,
    framework_profile_params=FrameworkProfile(local_path="/opt/ml/output/profiler/", start_step=5, num_steps=10),
)<jupyter_output><empty_output><jupyter_text>Specify Checkpoint S3 LocationThis is used for Spot Instances Training.  If nodes are replaced, the new node will start training from the latest checkpoint.<jupyter_code>import uuid

checkpoint_s3_prefix = "checkpoints/{}".format(str(uuid.uuid4()))
checkpoint_s3_uri = "s3://{}/{}/".format(bucket, checkpoint_s3_prefix)

print(checkpoint_s3_uri)<jupyter_output><empty_output><jupyter_text>Setup Our BERT + TensorFlow Script to Run on SageMakerPrepare our TensorFlow model to run on the managed SageMaker service<jupyter_code>!pygmentize src/tf_bert_reviews.py
from sagemaker.tensorflow import TensorFlow

estimator = TensorFlow(
    entry_point="tf_bert_reviews.py",
    source_dir="src",
    role=role,
    instance_count=train_instance_count,
    instance_type=train_instance_type,
    volume_size=train_volume_size,
    #                        use_spot_instances=True,
    #                        max_wait=7200, # Seconds to wait for spot instances to become available
    checkpoint_s3_uri=checkpoint_s3_uri,
    py_version="py37",
    framework_version="2.3.1",
    hyperparameters={
        "epochs": epochs,
        "learning_rate": learning_rate,
        "epsilon": epsilon,
        "train_batch_size": train_batch_size,
        "validation_batch_size": validation_batch_size,
        "test_batch_size": test_batch_size,
        "train_steps_per_epoch": train_steps_per_epoch,
        "validation_steps": validation_steps,
        "test_steps": test_steps,
        "use_xla": use_xla,
        "use_amp": use_amp,
        "max_seq_length": max_seq_length,
        "freeze_bert_layer": freeze_bert_layer,
        "enable_sagemaker_debugger": enable_sagemaker_debugger,
        "enable_checkpointing": enable_checkpointing,
        "enable_tensorboard": enable_tensorboard,
        "run_validation": run_validation,
        "run_test": run_test,
        "run_sample_predictions": run_sample_predictions,
    },
    input_mode=input_mode,
    metric_definitions=metrics_definitions,
    rules=rules,
    debugger_hook_config=hook_config,
    profiler_config=profiler_config,
    #                       max_run=7200, # number of seconds
)<jupyter_output><empty_output><jupyter_text>Create the `Experiment Config`<jupyter_code>experiment_config = {"ExperimentName": experiment_name, "TrialName": trial_name, "TrialComponentDisplayName": "train"}<jupyter_output><empty_output><jupyter_text>Train the Model on SageMaker<jupyter_code>estimator.fit(
    inputs={"train": s3_input_train_data, "validation": s3_input_validation_data, "test": s3_input_test_data},
    experiment_config=experiment_config,
    wait=False,
)
training_job_name = estimator.latest_training_job.name
print("Training Job Name:  {}".format(training_job_name))
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/{}">Training Job</a> After About 5 Minutes</b>'.format(
            region, training_job_name
        )
    )
)
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://console.aws.amazon.com/cloudwatch/home?region={}#logStream:group=/aws/sagemaker/TrainingJobs;prefix={};streamFilter=typeLogStreamPrefix">CloudWatch Logs</a> After About 5 Minutes</b>'.format(
            region, training_job_name
        )
    )
)
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://s3.console.aws.amazon.com/s3/buckets/{}/{}/?region={}&tab=overview">S3 Output Data</a> After The Training Job Has Completed</b>'.format(
            bucket, training_job_name, region
        )
    )
)
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://s3.console.aws.amazon.com/s3/buckets/{}/{}/?region={}&tab=overview">S3 Checkpoint Data</a> After The Training Job Has Completed</b>'.format(
            bucket, checkpoint_s3_prefix, region
        )
    )
)
%%time

estimator.latest_training_job.wait(logs=False)<jupyter_output><empty_output><jupyter_text>Wait Until the ^^ Training Job ^^ Completes Above!  [INFO] _Feel free to continue to the next workshop section while this notebook is running._<jupyter_code>%store training_job_name
!aws s3 cp s3://$bucket/$training_job_name/output/model.tar.gz ./model.tar.gz
!mkdir -p ./model/
!tar -xvzf ./model.tar.gz -C ./model/
!saved_model_cli show --all --dir ./model/tensorflow/saved_model/0/
!saved_model_cli run --dir ./model/tensorflow/saved_model/0/ --tag_set serve --signature_def serving_default \
    --input_exprs 'input_ids=np.zeros((1,64));input_mask=np.zeros((1,64))'<jupyter_output><empty_output><jupyter_text>![Confusion Matrix](./model/metrics/confusion_matrix.png)  Analyze Debugger Rules<jupyter_code>estimator.latest_training_job.rule_job_summary()
%store<jupyter_output><empty_output><jupyter_text>Show the Experiment Tracking Lineage<jupyter_code>from sagemaker.analytics import ExperimentAnalytics

import pandas as pd

pd.set_option("max_colwidth", 500)

experiment_analytics = ExperimentAnalytics(
    sagemaker_session=sess,
    experiment_name=experiment_name,
    metric_names=["validation:accuracy"],
    sort_by="CreationTime",
    sort_order="Descending",
)

experiment_analytics_df = experiment_analytics.dataframe()
experiment_analytics_df
from sagemaker.lineage.visualizer import LineageTableVisualizer

lineage_table_viz = LineageTableVisualizer(sess)
lineage_table_viz_df = lineage_table_viz.show(training_job_name=training_job_name)
lineage_table_viz_df<jupyter_output><empty_output><jupyter_text>Release Resources<jupyter_code>%%html

<p><b>Shutting down your kernel for this notebook to release resources.</b></p>
<button class="sm-command-button" data-commandlinker-command="kernelmenu:shutdown" style="display:none;">Shutdown Kernel</button>
        
<script>
try {
    els = document.getElementsByClassName("sm-command-button");
    els[0].click();
}
catch(err) {
    // NoOp
}    
</script>
%%javascript

try {
    Jupyter.notebook.save_checkpoint();
    Jupyter.notebook.session.delete();
}
catch(err) {
    // NoOp
}
# Internal - DO NOT RUN

# step_prefix = '07_train'
# !aws s3 cp s3://$bucket/$training_job_name/output/model.tar.gz s3://dsoaws/$step_prefix/tensorflow/ --acl public-read-write --acl bucket-owner-full-control
# !aws s3 ls --recursive s3://dsoaws/$step_prefix/<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15908379748419801, 'fraction_numerical': 0.013893234870767882, 'mean_word_length': 4.701034605779522, 'pattern_counts': {'":': 48, '<': 89, '<?xml version=': 0, '>': 91, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '78387', 'license': 'Apache-2.0', 'n_tokens_mistral': 5210, 'n_tokens_neox': 4900, 'n_words': 996, 'path': '07_train/02_Train_Reviews_BERT_Transformers_TensorFlow_ScriptMode.ipynb', 'repo_name': 'NRauschmayr/workshop', 'size': '15979'}