<jupyter_start><jupyter_text>Batch Transform for ExplanationsIn this notebook, we'll use Amazon SageMaker Batch Transform to obtainexplanations for our complete dataset.   We start by importing a variety of packages that will be used throughoutthe notebook. One of the most important packages used throughout thissolution is the Amazon SageMaker Python SDK (i.e. `import sagemaker`). Wealso import modules from our own custom package that can be found at`./package`.<jupyter_code>import boto3
from pathlib import Path
import sagemaker
from sagemaker.transformer import Transformer

from package import config, utils<jupyter_output><empty_output><jupyter_text>Up next, we define the current folder, a sagemaker session and asagemaker client (from `boto3`).<jupyter_code>current_folder = utils.get_current_folder(globals())
sagemaker_session = sagemaker.Session()
sagemaker_client = boto3.client('sagemaker')<jupyter_output><empty_output><jupyter_text>We define a function below to retrieve the same model that was created inlast stage. Model refers to the package of model assets and deploymentcode. We could have created another model here (using the same model datafrom the training stage) but let's use the same model to avoidduplication.<jupyter_code>def get_latest_model(name_contains):
    response = sagemaker_client.list_models(
        NameContains=name_contains
    )
    models = response['Models']
    assert len(models) > 0, "Couldn't find any models with '{}' in name.".format(name_contains)
    latest_model = models[0]['ModelName']
    return latest_model
latest_model = get_latest_model(config.RESOURCE_NAME)
job_name = latest_model<jupyter_output><empty_output><jupyter_text>Unlike the last stage, where we deployed an endpoint, we define a`Transformer` to perform the batch computation. We specify the instancetype that should be used for the computation (i.e. `ml.c5.xlarge`) and anumber of other parameters. `strategy='SingleRecord'` means that recordswill be processed by the explainer one at a time. And `output_path`defines where the Batch Transform output should be saved.<jupyter_code>batch_explainer = Transformer(
    model_name=latest_model,
    instance_count=1,
    instance_type='ml.c5.xlarge',
    strategy='SingleRecord',
    assemble_with='Line',
    output_path='s3://' + str(Path(config.S3_BUCKET, 'explanations', job_name)) + '/',
    accept='application/json',
    base_transform_job_name=config.RESOURCE_NAME,
    sagemaker_session=sagemaker_session,
    tags=[{'Key': config.TAG_KEY, 'Value': config.RESOURCE_NAME}]
)<jupyter_output><empty_output><jupyter_text>We haven't yet started the Batch Transform Job. Calling `.transform` doesthat below. We also specify the `content_type` at this stage, which givesus control over what type of entities we want to return from theexplainer. As an example, we have requested SHAP interaction valuesduring this batch job.<jupyter_code>entities = [
    'data',
    'features',
    'prediction',
    'explanation_shap_values',
    'explanation_shap_interaction_values'
]
batch_explainer.transform(
    data='s3://' + str(Path(config.S3_BUCKET, config.DATASETS_S3_PREFIX, 'data_test')) + '/',
    content_type="application/json; entities={}".format(",".join(entities)),
    split_type='Line',
    wait=True
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08901629450814726, 'fraction_numerical': 0.003922751961375981, 'mean_word_length': 5.793032786885246, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '93840', 'license': 'Apache-2.0', 'n_tokens_mistral': 993, 'n_tokens_neox': 976, 'n_words': 304, 'path': 'sagemaker/notebooks/4_batch_transform.ipynb', 'repo_name': 'NihalHarish/sagemaker-explaining-credit-decisions', 'size': '3314'}