<jupyter_start><jupyter_text>Deploy and monitor a machine learning workflow for Image Classification  Setting up this notebookNotes about the instance size and kernel setup: this notebook has been tested on1. The `Python 3 (Data Science)` kernel2. The `ml.t3.medium` instance Data StagingWe'll use a sample dataset called CIFAR to simulate the challenges Scones Unlimited are facing in Image Classification. In order to start working with CIFAR we'll need to:1. Extract the data from a hosting service2. Transform it into a usable shape and format3. Load it into a production systemIn other words, we're going to do some simple ETL! 1. Extract the data from the hosting serviceIn the cell below, define a function `extract_cifar_data` that extracts python version of the CIFAR-100 dataset. The CIFAR dataaset is open source and generously hosted by the University of Toronto at: https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz<jupyter_code>import requests

def extract_cifar_data(url, filename="cifar.tar.gz"):
    """A function for extracting the CIFAR-100 dataset and storing it as a gzipped file
    
    Arguments:
    url      -- the URL where the dataset is hosted
    filename -- the full path where the dataset will be written
    
    """
    
    # Todo: request the data from the data url
    # Hint: use `requests.get` method
    r = requests.get(url)
    with open(filename, "wb") as file_context:
        file_context.write(r.content)
    return<jupyter_output><empty_output><jupyter_text>Let's test it out! Run the following cell and check whether a new file `cifar.tar.gz` is created in the file explorer.<jupyter_code>extract_cifar_data("https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz")<jupyter_output><empty_output><jupyter_text>2. Transform the data into a usable shape and formatClearly, distributing the data as a gzipped archive makes sense for the hosting service! It saves on bandwidth, storage, and it's a widely-used archive format. In fact, it's so widely used that the Python community ships a utility for working with them, `tarfile`, as part of its Standard Library. Execute the following cell to decompress your extracted dataset:<jupyter_code>import tarfile

with tarfile.open("cifar.tar.gz", "r:gz") as tar:
    tar.extractall()<jupyter_output><empty_output><jupyter_text>A new folder `cifar-100-python` should be created, containing `meta`, `test`, and `train` files. These files are `pickles` and the [CIFAR homepage](https://www.cs.toronto.edu/~kriz/cifar.html) provides a simple script that can be used to load them. We've adapted the script below for you to run:<jupyter_code>import pickle

with open("./cifar-100-python/meta", "rb") as f:
    dataset_meta = pickle.load(f, encoding='bytes')

with open("./cifar-100-python/test", "rb") as f:
    dataset_test = pickle.load(f, encoding='bytes')

with open("./cifar-100-python/train", "rb") as f:
    dataset_train = pickle.load(f, encoding='bytes')
# Feel free to explore the datasets

dataset_train.keys()<jupyter_output><empty_output><jupyter_text>As documented on the homepage, `b'data'` contains rows of 3073 unsigned integers, representing three channels (red, green, and blue) for one 32x32 pixel image per row.<jupyter_code>32*32*3<jupyter_output><empty_output><jupyter_text>For a simple gut-check, let's transform one of our images. Each 1024 items in a row is a channel (red, green, then blue). Each 32 items in the channel are a row in the 32x32 image. Using python, we can stack these channels into a 32x32x3 array, and save it as a PNG file:<jupyter_code>import numpy as np

# Each 1024 in a row is a channel (red, green, then blue)
row = dataset_train[b'data'][0]
red, green, blue = row[0:1024], row[1024:2048], row[2048:]

# Each 32 items in the channel are a row in the 32x32 image
red = red.reshape(32,32)
green = green.reshape(32,32)
blue = blue.reshape(32,32)

# Combine the channels into a 32x32x3 image!
combined = np.dstack((red,green,blue))<jupyter_output><empty_output><jupyter_text>For a more concise version, consider the following:<jupyter_code># All in one:
test_image = np.dstack((
    row[0:1024].reshape(32,32),
    row[1024:2048].reshape(32,32),
    row[2048:].reshape(32,32)
))
import matplotlib.pyplot as plt
plt.imshow(test_image);<jupyter_output><empty_output><jupyter_text>Looks like a cow! Let's check the label. `dataset_meta` contains label names in order, and `dataset_train` has a list of labels for each row.<jupyter_code>dataset_train[b'fine_labels'][0]<jupyter_output><empty_output><jupyter_text>Our image has a label of `19`, so let's see what the 19th item is in the list of label names.<jupyter_code>print(dataset_meta[b'fine_label_names'][19])<jupyter_output>b'cattle'<jupyter_text>Ok! 'cattle' sounds about right. By the way, using the previous two lines we can do:<jupyter_code>n = 0
print(dataset_meta[b'fine_label_names'][dataset_train[b'fine_labels'][n]])<jupyter_output>b'cattle'<jupyter_text>Now we know how to check labels, is there a way that we can also check file names? `dataset_train` also contains a `b'filenames'` key. Let's see what we have here:<jupyter_code>print(dataset_train[b'filenames'][0])<jupyter_output>b'bos_taurus_s_000507.png'<jupyter_text>"Taurus" is the name of a subspecies of cattle, so this looks like a pretty reasonable filename. To save an image we can also do:<jupyter_code>plt.imsave("file.png", test_image)<jupyter_output><empty_output><jupyter_text>Your new PNG file should now appear in the file explorer -- go ahead and pop it open to see!Now that you know how to reshape the images, save them as files, and capture their filenames and labels, let's just capture all the bicycles and motorcycles and save them. Scones Unlimited can use a model that tells these apart to route delivery drivers automatically.In the following cell, identify the label numbers for Bicycles and Motorcycles:<jupyter_code>import pandas as pd

# Todo: Filter the dataset_train and dataset_meta objects to find the label numbers for Bicycle and Motorcycles
for  index, label_name in enumerate(dataset_meta[b'fine_label_names']):
    if label_name in [b'bicycle',b'motorcycle']:
        print( f"Label Name: {label_name},  Label Number: {index}")<jupyter_output>Label Name: b'bicycle',  Label Number: 8
Label Name: b'motorcycle',  Label Number: 48<jupyter_text>Good job! We only need objects with label 8 and 48 -- this drastically simplifies our handling of the data! Below we construct a dataframe for you, and you can safely drop the rows that don't contain observations about bicycles and motorcycles. Fill in the missing lines below to drop all other rows:<jupyter_code>#Construct the dataframe
df_train = pd.DataFrame({
    "filenames": dataset_train[b'filenames'],
    "labels": dataset_train[b'fine_labels'],
    "row": range(len(dataset_train[b'filenames']))
})

# Drop all rows from df_train where label is not 8 or 48
df_train =df_train.loc[df_train["labels"].isin([8,48])]

# Decode df_train.filenames so they are regular strings
df_train["filenames"] = df_train["filenames"].apply(
    lambda x: x.decode("utf-8")
)


df_test = pd.DataFrame({
    "filenames": dataset_test[b'filenames'],
    "labels": dataset_test[b'fine_labels'],
    "row": range(len(dataset_test[b'filenames']))
})

# Drop all rows from df_test where label is not 8 or 48
df_test =df_test.loc[df_test["labels"].isin([8,48])]

# Decode df_test.filenames so they are regular strings
df_test["filenames"] = df_test["filenames"].apply(
    lambda x: x.decode("utf-8")
)<jupyter_output><empty_output><jupyter_text>Now that the data is filtered for just our classes, we can save all our images.<jupyter_code>!mkdir ./train
!mkdir ./test<jupyter_output>mkdir: cannot create directory ‘./train’: File exists
mkdir: cannot create directory ‘./test’: File exists<jupyter_text>In the previous sections we introduced you to several key snippets of code:1. Grabbing the image data:```pythondataset_train[b'data'][0]```2. A simple idiom for stacking the image data into the right shape```pythonimport numpy as npnp.dstack((    row[0:1024].reshape(32,32),    row[1024:2048].reshape(32,32),    row[2048:].reshape(32,32)))```3. A simple `matplotlib` utility for saving images```pythonplt.imsave(path+row['filenames'], target)```Compose these together into a function that saves all the images into the `./test` and `./train` directories. Use the comments in the body of the `save_images` function below to guide your construction of the function:<jupyter_code>import os

def save_images(row,file_name, target_folder_path, img_dataset):
   
    #Grab the image data in row-major form
    img =  img_dataset[b'data'][row]
    
    # Consolidated stacking/reshaping from earlier
    target = np.dstack((
       img[0:1024].reshape(32,32),
        img[1024:2048].reshape(32,32),
        img[2048:].reshape(32,32)
    ))
    
    # Save the image
    try:
        img_file_path = os.path.join( target_folder_path, file_name)
        plt.imsave(img_file_path, target)
    except e:
        return f"Error Saving {file_name} to folder {target_folder_path} \n Error: {e}  "
    # Return any signal data you want for debugging
    return f"Successfully saved {file_name} to folder {target_folder_path}."

    ## TODO: save ALL images using the save_images function

for df_row in df_train.itertuples():
    print(save_images(df_row.row, df_row.filenames,"./train", dataset_train))


for df_row in df_test.itertuples():
    print(save_images(df_row.row, df_row.filenames,"./test", dataset_test))<jupyter_output>Successfully saved bike_s_000682.png to folder ./train.
Successfully saved bike_s_000127.png to folder ./train.
Successfully saved cycle_s_002598.png to folder ./train.
Successfully saved minibike_s_000824.png to folder ./train.
Successfully saved motorcycle_s_001856.png to folder ./train.
Successfully saved bicycle_s_000537.png to folder ./train.
Successfully saved moped_s_000169.png to folder ./train.
Successfully saved motorcycle_s_001453.png to folder ./train.
Successfully saved safety_bike_s_000482.png to folder ./train.
Successfully saved dirt_bike_s_000124.png to folder ./train.
Successfully saved minibike_s_001863.png to folder ./train.
Successfully saved minibike_s_001177.png to folder ./train.
Successfully saved motorcycle_s_001792.png to folder ./train.
Successfully saved minibike_s_001969.png to folder ./train.
Successfully saved velocipede_s_001585.png to folder ./train.
Successfully saved moped_s_000035.png to folder ./train.
Successfully saved bike_s_000256.png to folder[...]<jupyter_text>3. Load the dataNow we can load the data into S3.Using the sagemaker SDK grab the current region, execution role, and bucket.<jupyter_code>import sagemaker

session=sagemaker.Session()
bucket= session.default_bucket()
print("Default Bucket: {}".format(bucket))

region = session.boto_region_name
print("AWS Region: {}".format(region))

role = sagemaker.get_execution_role()
print("RoleArn: {}".format(role))<jupyter_output>Default Bucket: sagemaker-us-east-1-647975508946
AWS Region: us-east-1
RoleArn: arn:aws:iam::647975508946:role/service-role/AmazonSageMaker-ExecutionRole-20220115T122340<jupyter_text>With this data we can easily sync your data up into S3!<jupyter_code>import os

os.environ["DEFAULT_S3_BUCKET"] = bucket
!aws s3 sync ./train s3://${DEFAULT_S3_BUCKET}/train/
!aws s3 sync ./test s3://${DEFAULT_S3_BUCKET}/test/<jupyter_output>upload: train/bicycle_s_000017.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000017.png
upload: train/bicycle_s_000038.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000038.png
upload: train/bicycle_s_000021.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000021.png
upload: train/bicycle_s_000124.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000124.png
upload: train/bicycle_s_000137.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000137.png
upload: train/bicycle_s_000147.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000147.png
upload: train/bicycle_s_000039.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000039.png
upload: train/bicycle_s_000066.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000066.png
upload: train/bicycle_s_000043.png to s3://sagemaker-us-east-1-647975508946/train/bicycle_s_000043.png
upload: train/bicycle_s_000099.png to s3://sagemaker-us-east-1-6479755089[...]<jupyter_text>And that's it! You can check the bucket and verify that the items were uploaded. Model TrainingFor Image Classification, Sagemaker [also expects metadata](https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html) e.g. in the form of TSV files with labels and filepaths. We can generate these using our Pandas DataFrames from earlier:<jupyter_code>def to_metadata_file(df, prefix):
    df["s3_path"] = df["filenames"]
    df["labels"] = df["labels"].apply(lambda x: 0 if x==8 else 1)
    return df[["row", "labels", "s3_path"]].to_csv(
        f"{prefix}.lst", sep="\t", index=False, header=False
    )
    
to_metadata_file(df_train.copy(), "train")
to_metadata_file(df_test.copy(), "test")<jupyter_output><empty_output><jupyter_text>We can also upload our manifest files:<jupyter_code>import boto3

# Upload files
boto3.Session().resource('s3').Bucket(
    bucket).Object('train.lst').upload_file('./train.lst')
boto3.Session().resource('s3').Bucket(
    bucket).Object('test.lst').upload_file('./test.lst')<jupyter_output><empty_output><jupyter_text>Using the `bucket` and `region` info we can get the latest prebuilt container to run our training job, and define an output location on our s3 bucket for the model. Use the `image_uris` function from the SageMaker SDK to retrieve the latest `image-classification` image below:<jupyter_code># Use the image_uris function to retrieve the latest 'image-classification' image 
algo_image = sagemaker.image_uris.retrieve("image-classification", region, "latest")
s3_output_location = f"s3://{bucket}/models/image_model"<jupyter_output>Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: latest.<jupyter_text>We're ready to create an estimator! Create an estimator `img_classifier_model` that uses one instance of `ml.p2.xlarge`. Ensure that y ou use the output location we defined above - we'll be referring to that later!<jupyter_code>img_classifier_model=sagemaker.estimator.Estimator(
    image_uri=algo_image,
    role=role,
    instance_count=1,
    instance_type='ml.p2.xlarge',
    output_path = s3_output_location,
    sagemaker_session = sagemaker.Session()    
)<jupyter_output><empty_output><jupyter_text>We can also set a few key hyperparameters and define the inputs for our model:<jupyter_code>img_classifier_model.set_hyperparameters(
    image_shape='3,32,32', 
    num_classes=2, 
    num_training_samples=df_train.shape[0]
)<jupyter_output><empty_output><jupyter_text>The `image-classification` image uses four input channels with very specific input parameters. For convenience, we've provided them below:<jupyter_code>from sagemaker.debugger import Rule, rule_configs
from sagemaker.session import TrainingInput
model_inputs = {
        "train": sagemaker.inputs.TrainingInput(
            s3_data=f"s3://{bucket}/train/",
            content_type="application/x-image"
        ),
        "validation": sagemaker.inputs.TrainingInput(
            s3_data=f"s3://{bucket}/test/",
            content_type="application/x-image"
        ),
        "train_lst": sagemaker.inputs.TrainingInput(
            s3_data=f"s3://{bucket}/train.lst",
            content_type="application/x-image"
        ),
        "validation_lst": sagemaker.inputs.TrainingInput(
            s3_data=f"s3://{bucket}/test.lst",
            content_type="application/x-image"
        )
}<jupyter_output><empty_output><jupyter_text>Great, now we can train the model using the model_inputs. In the cell below, call the `fit` method on our model,:<jupyter_code>img_classifier_model.fit(model_inputs)<jupyter_output>2022-01-15 07:25:58 Starting - Starting the training job...
2022-01-15 07:26:21 Starting - Launching requested ML instancesProfilerReport-1642231558: InProgress
......
2022-01-15 07:27:25 Starting - Preparing the instances for training............
2022-01-15 07:29:22 Downloading - Downloading input data......
2022-01-15 07:30:22 Training - Downloading the training image......
2022-01-15 07:31:27 Training - Training image download completed. Training in progress..[34mDocker entrypoint called with argument(s): train[0m
[34m[01/15/2022 07:31:33 INFO 140673250826048] Reading default configuration from /opt/amazon/lib/python3.7/site-packages/image_classification/default-input.json: {'use_pretrained_model': 0, 'num_layers': 152, 'epochs': 30, 'learning_rate': 0.1, 'lr_scheduler_factor': 0.1, 'optimizer': 'sgd', 'momentum': 0, 'weight_decay': 0.0001, 'beta_1': 0.9, 'beta_2': 0.999, 'eps': 1e-08, 'gamma': 0.9, 'mini_batch_size': 32, 'image_shape': '3,224,224', 'precision_dtype': 'float32'}[...]<jupyter_text>If all goes well, you'll end up with a model topping out above `.8` validation accuracy. With only 1000 training samples in the CIFAR dataset, that's pretty good. We could definitely pursue data augmentation & gathering more samples to help us improve further, but for now let's proceed to deploy our model. Getting ready to deployTo begin with, let's configure Model Monitor to track our deployment. We'll define a `DataCaptureConfig` below:<jupyter_code>from sagemaker.model_monitor import DataCaptureConfig

data_capture_config = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri=f"s3://{bucket}/data_capture"
)<jupyter_output><empty_output><jupyter_text>Note the `destination_s3_uri` parameter: At the end of the project, we can explore the `data_capture` directory in S3 to find crucial data about the inputs and outputs Model Monitor has observed on our model endpoint over time.With that done, deploy your model on a single `ml.m5.xlarge` instance with the data capture config attached:<jupyter_code>deployment = img_classifier_model.deploy(
    initial_instance_count=1,
    instance_type='ml.m5.xlarge',
    data_capture_config=data_capture_config
    )

endpoint = deployment.endpoint_name
print(endpoint)<jupyter_output>---------------!image-classification-2022-01-15-07-43-27-313<jupyter_text>Note the endpoint name for later as well.Next, instantiate a Predictor:<jupyter_code>from sagemaker.predictor import Predictor
predictor = Predictor(endpoint)<jupyter_output><empty_output><jupyter_text>In the code snippet below we are going to prepare one of your saved images for prediction. Use the predictor to process the `payload`.<jupyter_code>from sagemaker.serializers import IdentitySerializer
import base64

predictor.serializer = IdentitySerializer("image/png")
with open("./test/bicycle_s_001789.png", "rb") as f:
    payload = f.read()

    
inference = predictor.predict(payload)<jupyter_output><empty_output><jupyter_text>Your `inference` object is an array of two values, the predicted probability value for each of your classes (bicycle and motorcycle respectively.) So, for example, a value of `b'[0.91, 0.09]'` indicates the probability of being a bike is 91% and being a motorcycle is 9%.<jupyter_code>print(inference)<jupyter_output>b'[0.9110357761383057, 0.08896422386169434]'<jupyter_text>Draft Lambdas and Step Function WorkflowYour operations team uses Step Functions to orchestrate serverless workflows. One of the nice things about Step Functions is that [workflows can call other workflows](https://docs.aws.amazon.com/step-functions/latest/dg/connect-stepfunctions.html), so the team can easily plug your workflow into the broader production architecture for Scones Unlimited.In this next stage you're going to write and deploy three Lambda functions, and then use the Step Functions visual editor to chain them together! Our functions are going to work with a simple data object:```python{    "inferences": [],  Output of predictor.predict    "s3_key": "",  Source data S3 key    "s3_bucket": "",  Source data S3 bucket    "image_data": ""   base64 encoded string containing the image data}```A good test object that you can use for Lambda tests and Step Function executions, throughout the next section, might look like this:```python{  "image_data": "",  "s3_bucket": MY_BUCKET_NAME,  Fill in with your bucket  "s3_key": "test/bicycle_s_000513.png"}```Using these fields, your functions can read and write the necessary data to execute your workflow. Let's start with the first function. Your first Lambda function will copy an object from S3, base64 encode it, and then return it to the step function as `image_data` in an event.Go to the Lambda dashboard and create a new Lambda function with a descriptive name like "serializeImageData" and select thr 'Python 3.8' runtime. Add the same permissions as the SageMaker role you created earlier. (Reminder: you do this in the Configuration tab under "Permissions"). Once you're ready, use the starter code below to craft your Lambda handler:```pythonimport jsonimport boto3import base64s3 = boto3.client('s3')def lambda_handler(event, context):    """A function to serialize target data from S3"""         Get the s3 address from the Step Function event input    key =  TODO: fill in    bucket =  TODO: fill in         Download the data from s3 to /tmp/image.png     TODO: fill in         We read the data from a file    with open("/tmp/image.png", "rb") as f:        image_data = base64.b64encode(f.read())     Pass the data back to the Step Function    print("Event:", event.keys())    return {        'statusCode': 200,        'body': {            "image_data": image_data,            "s3_bucket": bucket,            "s3_key": key,            "inferences": []        }    }```The next function is responsible for the classification part - we're going to take the image output from the previous function, decode it, and then pass inferences back to the the Step Function.Because this Lambda will have runtime dependencies (i.e. the SageMaker SDK) you'll need to package them in your function. *Key reading:* https://docs.aws.amazon.com/lambda/latest/dg/python-package-create.htmlpython-package-create-with-dependencyCreate a new Lambda function with the same rights and a descriptive name, then fill in the starter code below for your classifier Lambda.```pythonimport jsonimport sagemakerimport base64from sagemaker.serializers import IdentitySerializer Fill this in with the name of your deployed modelENDPOINT =  TODO: fill indef lambda_handler(event, context):     Decode the image data    image = base64.b64decode( TODO: fill in)     Instantiate a Predictor    predictor =  TODO: fill in     For this model the IdentitySerializer needs to be "image/png"    predictor.serializer = IdentitySerializer("image/png")         Make a prediction:    inferences =  TODO: fill in         We return the data back to the Step Function        event["inferences"] = inferences.decode('utf-8')    return {        'statusCode': 200,        'body': json.dumps(event)    }```Finally, we need to filter low-confidence inferences. Define a threshold between 1.00 and 0.000 for your model: what is reasonble for you? If the model predicts at `.70` for it's highest confidence label, do we want to pass that inference along to downstream systems? Make one last Lambda function and tee up the same permissions:```pythonimport jsonTHRESHOLD = .93def lambda_handler(event, context):         Grab the inferences from the event    inferences =  TODO: fill in         Check if any values in our inferences are above THRESHOLD    meets_threshold =  TODO: fill in         If our threshold is met, pass our data back out of the     Step Function, else, end the Step Function with an error    if meets_threshold:        pass    else:        raise("THRESHOLD_CONFIDENCE_NOT_MET")    return {        'statusCode': 200,        'body': json.dumps(event)    }```Once you have tested the lambda functions, save the code for each lambda function in a python script called 'lambda.py'.With your lambdas in place, you can use the Step Functions visual editor to construct a workflow that chains them together. In the Step Functions console you'll have the option to author a Standard step function *Visually*.When the visual editor opens, you'll have many options to add transitions in your workflow. We're going to keep it simple and have just one: to invoke Lambda functions. Add three of them chained together. For each one, you'll be able to select the Lambda functions you just created in the proper order, filter inputs and outputs, and give them descriptive names.Make sure that you:1. Are properly filtering the inputs and outputs of your invokations (e.g. `$.body`)2. Take care to remove the error handling from the last function - it's supposed to "fail loudly" for your operations colleagues!Take a screenshot of your working step function in action and export the step function as JSON for your submission package. Great! Now you can use the files in `./test` as test files for our workflow. Depending on our threshold, our workflow should reliably pass predictions about images from `./test` on to downstream systems, while erroring out for inferences below our confidence threshold! Testing and EvaluationDo several step function invokations using data from the `./test` folder. This process should give you confidence that the workflow both *succeeds* AND *fails* as expected. In addition, SageMaker Model Monitor will generate recordings of your data and inferences which we can visualize.Here's a function that can help you generate test inputs for your invokations:<jupyter_code>import random
import boto3
import json


def generate_test_case():
    # Setup s3 in boto3
    s3 = boto3.resource('s3')
    
    # Randomly pick from sfn or test folders in our bucket
    objects = s3.Bucket(bucket).objects.filter(Prefix="test")
    
    # Grab any random object key from that folder!
    obj = random.choice([x.key for x in objects])
    
    return json.dumps({
        "image_data": "",
        "s3_bucket": bucket,
        "s3_key": obj
    })
generate_test_case()<jupyter_output><empty_output><jupyter_text>In the Step Function dashboard for your new function, you can create new executions and copy in the generated test cases. Do several executions so that you can generate data you can evaluate and visualize.Once you've done several executions, let's visualize the record of our inferences. Pull in the JSONLines data from your inferences like so:<jupyter_code>from sagemaker.s3 import S3Downloader

# In S3 your data will be saved to a datetime-aware path
# Find a path related to a datetime you're interested in

data_path="s3://sagemaker-us-east-1-647975508946/data_capture/image-classification-2022-01-15-07-43-27-313/AllTraffic/2022/01/15/07/"

S3Downloader.download(data_path, "captured_data")






# Feel free to repeat this multiple times and pull in more data<jupyter_output><empty_output><jupyter_text>The data are in JSONLines format, where multiple valid JSON objects are stacked on top of eachother in a single `jsonl` file. We'll import an open-source library, `jsonlines` that was purpose built for parsing this format.<jupyter_code>!pip install jsonlines
import jsonlines<jupyter_output>/opt/conda/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/opt/conda/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
Collecting jsonlines
  Using cached jsonlines-3.0.0-py3-none-any.whl (8.5 kB)
Requirement already satisfied: attrs>=19.2.0 in /opt/conda/lib/python3.7/site-packages (from jsonlines) (19.3.0)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from jsonlines) (4.0.1)
Installing collected packages: jsonlines
Successfully installed jsonlines-3.0.0
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.[...]<jupyter_text>Now we can extract the data from each of the source files:<jupyter_code>import os



# List the file names we downloaded
file_handles = os.listdir("./captured_data")

# Dump all the data into an array
json_data = []
for jsonl in file_handles:
    with jsonlines.open(f"./captured_data/{jsonl}") as f:
        json_data.append(f.read())
json_data<jupyter_output><empty_output><jupyter_text>The data should now be a list of dictionaries, with significant nesting. We'll give you an example of some code that grabs data out of the objects and visualizes it:<jupyter_code># Define how we'll get our data
def simple_getter(obj):
    inferences = obj["captureData"]["endpointOutput"]["data"]
    timestamp = obj["eventMetadata"]["inferenceTime"]
    return json.loads(inferences), timestamp

simple_getter(json_data[0])<jupyter_output><empty_output><jupyter_text>Finally, here's an example of a visualization you can build with this data. In this last part, you will take some time and build your own - the captured data has the input images, the resulting inferences, and the timestamps.<jupyter_code>import matplotlib.pyplot as plt
# Populate the data for the x and y axis
x = []
y = []
for obj in json_data:
    inference, timestamp = simple_getter(obj)
    
    y.append(max(inference))
    x.append(timestamp)

# Todo: here is an visualization example, take some time to build another visual that helps monitor the result
# Plot the data
plt.scatter(x, y, c=['r' if k<.80 else 'b' for k in y ])
plt.axhline(y=0.80, color='g', linestyle='--')
plt.ylim(bottom=.60)

# Add labels
plt.ylabel("Confidence")
plt.suptitle("Observed Recent Inferences", size=14)
plt.title("Pictured with confidence threshold for production use", size=10)

# Give it some pizzaz!
plt.style.use("Solarize_Light2")
plt.gcf().autofmt_xdate()<jupyter_output><empty_output><jupyter_text>Todo: build your own visualization<jupyter_code># Plot the histogram data
plt.figure(figsize=(10,5))
plt.hist(y, bins=75)
plt.axvline(x=0.75, color='green', linestyle='--')

# Add labels
plt.xlabel("Inference Confidence")
plt.suptitle("Histogram of Observed Recent Inferences", size=20)
plt.style.use("fivethirtyeight")
plt.figure(figsize=(13,5))
plt.plot(x, y, lw = 2, color = 'purple', alpha = 1)    
plt.suptitle("Observed Recent Inferences", size=14)
plt.xlabel("Number of Inference")
plt.ylabel("Confidence")
plt.style.use( 'seaborn-dark')
plt.gcf().autofmt_xdate()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09700936396332177, 'fraction_numerical': 0.03577098791433107, 'mean_word_length': 4.893450448730189, 'pattern_counts': {'":': 24, '<': 137, '<?xml version=': 0, '>': 137, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '153373', 'license': 'MIT', 'n_tokens_mistral': 10273, 'n_tokens_neox': 9391, 'n_words': 3215, 'path': 'starter.ipynb', 'repo_name': 'iragnis/ML-Workflow-Scones-Unlimited', 'size': '30863'}