<jupyter_start><jupyter_text>ReInvent 2019 Amazon Personalize Workshop![MainDiagram](static/imgs/image.png) AgendaAt a high level working with Amazon Personalize follows the steps in the diagram below:![FlowDiagram](static/imgs/personalize_process.png)The specific process that you will be following is:1. Imports and Setup1. Preparing Your Data1. Importing Your Data1. Selecting a Recipe1. Training a Solution1. Deploying a Campaign1. Getting Recommendations1. Real-Time Interactions1. Conclusion1. Bonus: Bulk Export of RecommendationsIn each of these steps you'll see and execute code snippets written in Python using our Boto3 SDK, these snippets can be modified to be components of your production integration with Personalize. This notebook will walk you through the steps to build a recommendation model for movies that are tailored to specific users, the goal is to recommend movies based on that users' history of positive interactions with movies.The data is provided via the [MovieLens Project](https://movielens.org), you can read more about it later if you are interested.The content below has been written to guide you through the process within the timeline of a workshop session. You can deploy the CloudFormation template you found on GitHub later to your own account and can walk through notebooks 1-3 to do the same exercise again. Additionally you can update the content in the cells to reflect your own data and it is an effective path to building a custom recommendaiton model for your use case. How to Use the NotebookCode is broken up into cells like the one below. There's a triangular `Run` button at the top of this page you can click to execute each cell and move onto the next, or you can press `Shift` + `Enter` or `Shift` + `Return`(Mac Users) while in the cell to execute it and move onto the next one.As a cell is executing you'll notice a line to the side showcase an `*` while the cell is running or it will update to a number to indicate the last cell that completed executing after it has finished exectuting all the code within a cell.Simply follow the instructions below and execute the cells to get started with Amazon Personalize.  Imports and Setup Python ships with a broad collection of libraries and we need to import those as well as the ones installed to help us like boto3(The AWS SDK) and Pandas/Numpy which are core data science tools. The cell below will import them for use here. It will also update the boto3 library to the latest version. You may see a warning(yellow text) or an error(red text), this is perfectly fine, just continue running the next cell.<jupyter_code># Update boto3
!pip install --upgrade boto3
# Imports
import boto3
import json
import numpy as np
import pandas as pd
import time
import datetime
import uuid<jupyter_output><empty_output><jupyter_text>Next you will want to validate that your environment can communicate successfully with Amazon Personalize, the lines below do just that.<jupyter_code># Configure the SDK to Personalize:
personalize = boto3.client('personalize')
personalize_runtime = boto3.client('personalize-runtime')
personalize_events = boto3.client(service_name='personalize-events')<jupyter_output><empty_output><jupyter_text>The last bit of setup is to determine the region you are running this workshop in. The cell below will do that and assign it to the `region` variable.<jupyter_code>with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:
    data = json.load(notebook_info)
    resource_arn = data['ResourceArn']
    region = resource_arn.split(':')[3]<jupyter_output><empty_output><jupyter_text>Preparing Your DataTo begin you will need a collection of data points where users have interacted with content in some way. Amazon Personalize assumes that if an interaction is recorded it is a positive one, you will see how we use that later. The cell below will download the data you need, extract the content from a Zip file, and then display a small portion of it to your screen.<jupyter_code>!wget -N http://files.grouplens.org/datasets/movielens/ml-100k.zip
!unzip -o ml-100k.zip
data = pd.read_csv('./ml-100k/u.data', sep='\t', names=['USER_ID', 'ITEM_ID', 'RATING', 'TIMESTAMP'], engine='python')
pd.set_option('display.max_rows', 5)
data<jupyter_output><empty_output><jupyter_text>As you can see the data contains a UserID, ItemID, Rating, and Timestamp.We do not need the `Rating` column so it will be removed before we save the file.Once the data has been prepared you will save it to a CSV file locally with the last line.<jupyter_code>data = data[['USER_ID', 'ITEM_ID', 'TIMESTAMP']] # select columns that match the columns in the schema below
filename = "movie-lens-ml-100k-prepared.csv"
data.to_csv(filename, index=False)<jupyter_output><empty_output><jupyter_text>Before you can upload the data into S3 you will need to create a bucket, the cell below will do that.<jupyter_code>print(region)
s3 = boto3.client('s3')
account_id = boto3.client('sts').get_caller_identity().get('Account')
bucket_name = account_id + "reinventpersonalizeworkshop"
print(bucket_name)
if region != "us-east-1":
    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': region})
else:
    s3.create_bucket(Bucket=bucket_name)<jupyter_output><empty_output><jupyter_text>Now you can upload the file to S3 and it will be ready to import into Amazon Personalize.<jupyter_code>boto3.Session().resource('s3').Bucket(bucket_name).Object(filename).upload_file(filename)<jupyter_output><empty_output><jupyter_text>Importing Your DataThe steps for importing your data are:1. Create a Dataset Group.1. Determine the schema for your dataset.1. Create a Dataset using your schema.1. Run an ImportJob to load the data for use with Personalize.In Amazon Personalize a Dataset Group is how your information is isolated from any other experiment. No information is shared between these groups at all. A Dataset Group can contain your interaction data, item metadata, user metadata, event trackers, solutions, and campaigns. To learn more about them: https://docs.aws.amazon.com/personalize/latest/dg/API_DatasetGroup.html Begin by creating a Dataset Group  Create Dataset Group<jupyter_code>create_dataset_group_response = personalize.create_dataset_group(
    name = "personalize-RI-demo"
)

dataset_group_arn = create_dataset_group_response['datasetGroupArn']
print(json.dumps(create_dataset_group_response, indent=2))<jupyter_output><empty_output><jupyter_text>Dataset Group Create LoopIt takes a few seconds to create a DatasetGroup so the cell below will poll until the DatasetGroup is active and you can continue with the workshop. Once it says `ACTIVE` move to the `Create Schema` step.<jupyter_code>max_time = time.time() + 3*60*60 # 3 hours
while time.time() < max_time:
    describe_dataset_group_response = personalize.describe_dataset_group(
        datasetGroupArn = dataset_group_arn
    )
    status = describe_dataset_group_response["datasetGroup"]["status"]
    print("DatasetGroup: {}".format(status))
    
    if status == "ACTIVE" or status == "CREATE FAILED":
        break
        
    time.sleep(5)<jupyter_output><empty_output><jupyter_text>Create Schema Now you can determine the Schema for your Dataset. In this workshop time is limited so you will only be utilizing user-item-interaction data or interaction data for short. The cell below contains the required items for this dataset and maps to the structure of the CSV that you uploaded earlier.<jupyter_code>schema = {
    "type": "record",
    "name": "Interactions",
    "namespace": "com.amazonaws.personalize.schema",
    "fields": [
        {
            "name": "USER_ID",
            "type": "string"
        },
        {
            "name": "ITEM_ID",
            "type": "string"
        },
        {
            "name": "TIMESTAMP",
            "type": "long"
        }
    ],
    "version": "1.0"
}<jupyter_output><empty_output><jupyter_text>The cell below will create a schema inside Amazon Personalize that can be connected to a Dataset this is how Personalize understands the content within your CSV file.<jupyter_code>create_schema_response = personalize.create_schema(
    name = "personalize-ri-demo-schema",
    schema = json.dumps(schema)
)

schema_arn = create_schema_response['schemaArn']
print(json.dumps(create_schema_response, indent=2))<jupyter_output><empty_output><jupyter_text>Create Dataset Next create the Dataset for interactions and assign it the Schema provided and assign it to the Dataset Group you created earlier with this code:<jupyter_code>dataset_type = "INTERACTIONS"
create_dataset_response = personalize.create_dataset(
    name = "personalize-ri-interactions",
    datasetType = dataset_type,
    datasetGroupArn = dataset_group_arn,
    schemaArn = schema_arn
)

dataset_arn = create_dataset_response['datasetArn']
print(json.dumps(create_dataset_response, indent=2))<jupyter_output><empty_output><jupyter_text>Attach Policy to S3 Bucket Before you can execute the import job for your data you will need to attach a bucket policy to your S3 bucket to allow Personalize to communicate with it, as well as an IAM role for the service to use within this AWS account.<jupyter_code>s3 = boto3.client("s3")

policy = {
    "Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": "personalize.amazonaws.com"
            },
            "Action": [
                "s3:*Object",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::{}".format(bucket_name),
                "arn:aws:s3:::{}/*".format(bucket_name)
            ]
        }
    ]
}

s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy))<jupyter_output><empty_output><jupyter_text>Just a note, IAM changes in the cell below take just a few seconds to be confirmed so this section will take 1 minute to complete. Execute the cell below next.  Create Personalize Role<jupyter_code>iam = boto3.client("iam")

role_name = "PersonalizeRoleRIDemo"
assume_role_policy_document = {
    "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "personalize.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
    ]
}

create_role_response = iam.create_role(
    RoleName = role_name,
    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)
)

# AmazonPersonalizeFullAccess provides access to any S3 bucket with a name that includes "personalize" or "Personalize" 
# if you would like to use a bucket with a different name, please consider creating and attaching a new policy
# that provides read access to your bucket or attaching the AmazonS3ReadOnlyAccess policy to the role
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess"
iam.attach_role_policy(
    RoleName = role_name,
    PolicyArn = policy_arn
)

# Now add S3 support
iam.attach_role_policy(
    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess',
    RoleName=role_name
)
time.sleep(60) # wait for a minute to allow IAM role policy attachment to propagate

role_arn = create_role_response["Role"]["Arn"]
print(role_arn)<jupyter_output><empty_output><jupyter_text>At last you are ready to import the data into Personalize, the first cell below will start the process and the second contains a while loop that will poll the service to determine when the import job has completed. During this workshop you are importing a relatively small dataset and it may seem like this takes a while for such a small file. The leading use of time here is provisioning dedicated resources that will actually run the task. This allows Personalize to provide HIPAA compliance for example, it also means that just because your files are larger that it won't take an extremely long time, once the resources are up and running the import is pretty quick.It may take up to 20 minutes for the import process to complete.   Create Dataset Import Job<jupyter_code>create_dataset_import_job_response = personalize.create_dataset_import_job(
    jobName = "personalize-ri-import",
    datasetArn = dataset_arn,
    dataSource = {
        "dataLocation": "s3://{}/{}".format(bucket_name, filename)
    },
    roleArn = role_arn
)

dataset_import_job_arn = create_dataset_import_job_response['datasetImportJobArn']
print(json.dumps(create_dataset_import_job_response, indent=2))<jupyter_output><empty_output><jupyter_text>Dataset Import LoopThe cell below will take a bit longer to complete, it is polling to learn when your Dataset has been fully imported into Personalize. This should take around 20 minutes to complete. Most of the time is spent provisioning infrastructure behind the scenes. This means that much larger files do not take much longer to import in most cases.Run the cell below, then move on when it reaches an `ACTIVE` state.<jupyter_code>current_time = datetime.datetime.now()
print("Import Started on: ", current_time.strftime("%I:%M:%S %p"))

max_time = time.time() + 3*60*60 # 3 hours
while time.time() < max_time:
    describe_dataset_import_job_response = personalize.describe_dataset_import_job(
        datasetImportJobArn = dataset_import_job_arn
    )
    status = describe_dataset_import_job_response["datasetImportJob"]['status']
    print("DatasetImportJob: {}".format(status))
    
    if status == "ACTIVE" or status == "CREATE FAILED":
        break
        
    time.sleep(60)
    
current_time = datetime.datetime.now()
print("Import Completed on: ", current_time.strftime("%I:%M:%S %p"))<jupyter_output><empty_output><jupyter_text>Selecting a RecipeInside Personalize the algorithms available are called recipes, the code below will give you a full list of them. They each have various use cases and detailed information can be found here: https://docs.aws.amazon.com/personalize/latest/dg/working-with-predefined-recipes.htmlIn this workshop you will be using HRNN(Hierarchical Recurrent Neural Network) quoting the docs:```HRRN is a hierarchical recurrent neural network, which can model the user-item interactions across a given timeframe. Use the HRNN recipe when user behavior changes over time, which is referred to the evolving intent problem.To train a model, HRNN uses the Interactions dataset from a dataset group. A dataset group is a set of related datasets, which can include the Users, Items, and Interactions datasets.```A paper explaining it more in detail as it relates to recommendations: https://openreview.net/pdf?id=ByzxsrrkJ4 It is used here as an example of a recommendation system built using deep neural networks and can be trained using only our interactions data. This is a great place to start when running your own experiments later as well.<jupyter_code>personalize.list_recipes()<jupyter_output><empty_output><jupyter_text>The cell below selects the `HRNN` recipe.<jupyter_code>recipe_arn = "arn:aws:personalize:::recipe/aws-hrnn"<jupyter_output><empty_output><jupyter_text>Training a SolutionWithin Amazon Personalize a model trained on customer data is called a Solution, these are versioned and so you will first create a Solution, then a Solution Version. The versions are utilized to track model improvement over time based on newer available data. Creating a Solution itself is nearly instantaneous however the actual training to create a version can take a bit of time. This is usually the longest waiting period in the process. If you are doing this outside of a workshop it is a great time to check your emails, grab a coffee, etc.  Create Solution<jupyter_code>create_solution_response = personalize.create_solution(
    name = "personalize-ri-soln-hrnn",
    datasetGroupArn = dataset_group_arn,
    recipeArn = recipe_arn
)

solution_arn = create_solution_response['solutionArn']
print(json.dumps(create_solution_response, indent=2))<jupyter_output><empty_output><jupyter_text>Create Solution Version<jupyter_code>create_solution_version_response = personalize.create_solution_version(
    solutionArn = solution_arn
)

solution_version_arn = create_solution_version_response['solutionVersionArn']
print(json.dumps(create_solution_version_response, indent=2))<jupyter_output><empty_output><jupyter_text>Create Solution Version LoopTraining a model can take a bit of time, this will often take at least 20 minutes to complete. Run the cell below, again waiting for it to reach an `ACTIVE` state before moving to the next bit.<jupyter_code>current_time = datetime.datetime.now()
print("Training Started on: ", current_time.strftime("%I:%M:%S %p"))

max_time = time.time() + 3*60*60 # 3 hours
while time.time() < max_time:
    describe_solution_version_response = personalize.describe_solution_version(
        solutionVersionArn = solution_version_arn
    )
    status = describe_solution_version_response["solutionVersion"]["status"]
    print("SolutionVersion: {}".format(status))
    
    if status == "ACTIVE" or status == "CREATE FAILED":
        break
        
    time.sleep(60)
    
current_time = datetime.datetime.now()
print("Training Completed on: ", current_time.strftime("%I:%M:%S %p"))<jupyter_output><empty_output><jupyter_text>Get Metrics of Solution VersionNow that your solution and version exists, you can obtain the metrics for it to judge its performance.<jupyter_code>get_solution_metrics_response = personalize.get_solution_metrics(
    solutionVersionArn = solution_version_arn
)

print(json.dumps(get_solution_metrics_response, indent=2))<jupyter_output><empty_output><jupyter_text>Create and Wait for the CampaignNow that you have a working solution version you will need to create a campaign to use it with your applications. A campaign is simply a hosted copy of your model. Again there will be a short wait so after executing you can take a quick break while the infrastructure is being provisioned.  Create Campaign<jupyter_code>create_campaign_response = personalize.create_campaign(
    name = "personalize-ri-camp",
    solutionVersionArn = solution_version_arn,
    minProvisionedTPS = 1
)

campaign_arn = create_campaign_response['campaignArn']
print(json.dumps(create_campaign_response, indent=2))<jupyter_output><empty_output><jupyter_text>Wait for Campaign to Have ACTIVE Status Create a Campaign LoopIn this section Personalize is deploying your model and that takes a few minutes to complete. Execute the cell below to agian poll until the task has completed. It should take 10 to 15 minutes. Once the cell has reached an `ACTIVE` state, continue on.<jupyter_code>current_time = datetime.datetime.now()
print("Deploying Started on: ", current_time.strftime("%I:%M:%S %p"))

max_time = time.time() + 3*60*60 # 3 hours
while time.time() < max_time:
    describe_campaign_response = personalize.describe_campaign(
        campaignArn = campaign_arn
    )
    status = describe_campaign_response["campaign"]["status"]
    print("Campaign: {}".format(status))
    
    if status == "ACTIVE" or status == "CREATE FAILED":
        break
        
    time.sleep(60)
    
current_time = datetime.datetime.now()
print("Deploying Completed on: ", current_time.strftime("%I:%M:%S %p"))<jupyter_output><empty_output><jupyter_text>Get Sample RecommendationsAfter the campaign is active you are ready to get recommendations. First we need to select a random user from the collection. Then we will create a few helper functions for getting movie information to show for recommendations instead of just IDs.<jupyter_code># Getting a random user:
user_id, item_id, _ = data.sample().values[0]
print("USER: {}".format(user_id))
items = pd.read_csv('./ml-100k/u.item', sep='|', usecols=[0,1], encoding='latin-1', names=['ITEM_ID', 'TITLE'], index_col='ITEM_ID')

def get_movie_title(movie_id):
    """
    Takes in an ID, returns a title
    """
    movie_id = int(movie_id)-1
    return items.iloc[movie_id]['TITLE']<jupyter_output><empty_output><jupyter_text>Call GetRecommendationsThe code below will get recommendations for the random user selected, it then places the recommendations in a dataframe and renders it.<jupyter_code>get_recommendations_response = personalize_runtime.get_recommendations(
    campaignArn = campaign_arn,
    userId = str(user_id),
)
# Update DF rendering
pd.set_option('display.max_rows', 30)

print("Recommendations for user: ", user_id)

item_list = get_recommendations_response['itemList']

recommendation_list = []

for item in item_list:
    title = get_movie_title(item['itemId'])
    recommendation_list.append(title)
    
recommendations_df = pd.DataFrame(recommendation_list, columns = ['Original Recommendations'])
recommendations_df<jupyter_output><empty_output><jupyter_text>Creating an Event TrackerBefore your recommendation system can respond to real time events you will need an event tracker, the code below will generate one and can be used going forward with this lab. Feel free to name it something more clever.<jupyter_code>response = personalize.create_event_tracker(
    name='MovieClickTrackerRI',
    datasetGroupArn=dataset_group_arn
)
print(response['eventTrackerArn'])
print(response['trackingId'])
TRACKING_ID = response['trackingId']
event_tracker_arn = response['eventTrackerArn']<jupyter_output><empty_output><jupyter_text>Simulating User BehaviorThe lines below provide a code sample that simulates a user interacting with a particular item, you will then get recommendations that differ from those when you started.<jupyter_code>session_dict = {}
def send_movie_click(USER_ID, ITEM_ID):
    """
    Simulates a click as an envent
    to send an event to Amazon Personalize's Event Tracker
    """
    # Configure Session
    try:
        session_ID = session_dict[USER_ID]
    except:
        session_dict[USER_ID] = str(uuid.uuid1())
        session_ID = session_dict[USER_ID]
        
    # Configure Properties:
    event = {
    "itemId": str(ITEM_ID),
    }
    event_json = json.dumps(event)
        
    # Make Call
    personalize_events.put_events(
    trackingId = TRACKING_ID,
    userId= USER_ID,
    sessionId = session_ID,
    eventList = [{
        'sentAt': int(time.time()),
        'eventType': 'EVENT_TYPE',
        'properties': event_json
        }]
    )

def get_new_recommendations_df(recommendations_df, movie_ID):
    # Get the title of the movie for the header of the column
    movie_title_clicked = get_movie_title(movie_to_click)
    # Interact with the movie
    send_movie_click(USER_ID=str(user_id), ITEM_ID=movie_to_click)
    # Sleep for 2 seconds
    time.sleep(2)
    # Get new recommendations
    get_recommendations_response = personalize_runtime.get_recommendations(
        campaignArn = campaign_arn,
        userId = str(user_id),
    )
    # Build a new dataframe of recommendations
    item_list = get_recommendations_response['itemList']
    recommendation_list = []
    for item in item_list:
        title = get_movie_title(item['itemId'])
        recommendation_list.append(title)
    new_rec_DF = pd.DataFrame(recommendation_list, columns = [movie_title_clicked])
    # Add this dataframe to the old one
    recommendations_df = recommendations_df.join(new_rec_DF)
    return recommendations_df<jupyter_output><empty_output><jupyter_text>The 3 cells below will simulate adding a movie ot the dataset and then rendering the results after each interaction. The first column is the original recommendations, the subsequent columns are for each movie that your user has now interacted with. The column header is the name of the movie that the user interacted with, and you can then see how the recommendations are altered.<jupyter_code>movie_to_click = 180
recommendations_df = get_new_recommendations_df(recommendations_df, movie_to_click)
recommendations_df
movie_to_click = 210
recommendations_df = get_new_recommendations_df(recommendations_df, movie_to_click)
recommendations_df
movie_to_click = 415
recommendations_df = get_new_recommendations_df(recommendations_df, movie_to_click)
recommendations_df<jupyter_output><empty_output><jupyter_text>ConclusionIn this workshop you successfully created a dataset group, a dataset based on interactions, trained a recommendation model based on the data, deployed a campaign to generate recommendations, evaluated the initial recommendations, leveraged real time event tracking for even better recommendations, and below is bonus material to show how to export recommendations in  This content will stay public on GitHub and can be used within your organization in order to build custom recommendation models, take a look at the various other notebooks in the future.Thank you, and good luck in your next Machine Learning project!BONUS MATERIAL BELOW:  Batch RecommendationsSo far you have seen how to generate recommendations via an API call and interact with event trackers. That works well for many applications but you may find yourself wanting to cache all recommendations for users locally or even to study the recommendations for new ideas. To support that Amazon Personalize supports batch exporting of your recommendations to a file as well. The cells below will walk you through sending recommendations to a file in S3 and then will show its contents. The file can be downloaded from S3 to your local computer as well. First you will need to create a JSON file of user IDS<jupyter_code>user_IDs = ['561', '233', '579']

json_input_filename = "json_input.json"
with open(json_input_filename, 'w') as json_input:
    for user_id in user_IDs:
        json_input.write('{"userId": "' + user_id + '"}\n')<jupyter_output><empty_output><jupyter_text>Now upload this file to S3:<jupyter_code>boto3.Session().resource('s3').Bucket(bucket_name).Object(json_input_filename).upload_file(json_input_filename)
s3_input_path = "s3://" + bucket_name + "/" + json_input_filename
print(s3_input_path)<jupyter_output><empty_output><jupyter_text>Now that the input file is available in S3, you need to define where the output will go, and create a batch inference job.<jupyter_code>s3_output_path = "s3://" + bucket_name + "/"
print(s3_output_path)<jupyter_output><empty_output><jupyter_text>The cell below creates the batch job.<jupyter_code>personalize_rec = boto3.client(service_name='personalize')
batchInferenceJobArn = personalize_rec.create_batch_inference_job (
    solutionVersionArn = solution_version_arn,
    jobName = "RI-Workshop-Batch-Inference-Job",
    roleArn = role_arn,
    jobInput = 
     {"s3DataSource": {"path": s3_input_path}},
    jobOutput = 
     {"s3DataDestination":{"path": s3_output_path}}
)
batchInferenceJobArn = batchInferenceJobArn['batchInferenceJobArn']<jupyter_output><empty_output><jupyter_text>The next cell will poll until the export has completed. This is the final waiting loop of the workshop! This will again poll until it reaches an `ACTIVE` state, you can continue on once this is reached.<jupyter_code>current_time = datetime.datetime.now()
print("Import Started on: ", current_time.strftime("%I:%M:%S %p"))

max_time = time.time() + 3*60*60 # 3 hours
while time.time() < max_time:
    describe_dataset_inference_job_response = personalize_rec.describe_batch_inference_job(
        batchInferenceJobArn = batchInferenceJobArn
    )
    status = describe_dataset_inference_job_response["batchInferenceJob"]['status']
    print("DatasetInferenceJob: {}".format(status))
    
    if status == "ACTIVE" or status == "CREATE FAILED":
        break
        
    time.sleep(60)
    
current_time = datetime.datetime.now()
print("Import Completed on: ", current_time.strftime("%I:%M:%S %p"))<jupyter_output><empty_output><jupyter_text>With the data successfully exported, grab the file and parse it.<jupyter_code>s3 = boto3.client('s3')
export_name = json_input_filename + ".out"
s3.download_file(bucket_name, export_name, export_name)

# Update DF rendering
pd.set_option('display.max_rows', 30)
with open(export_name) as json_file:
    # Get the first line and parse it
    line = json.loads(json_file.readline())
    # Do the same for the other lines
    while line:
        # extract the user ID 
        col_header = "User: " + line['input']['userId']
        # Create a list for all the movies
        recommendation_list = []
        # Add all the entries
        for item in line['output']['recommendedItems']:
            title = get_movie_title(item)
            recommendation_list.append(title)
        if 'bulk_recommendations_df' in locals():
            new_rec_DF = pd.DataFrame(recommendation_list, columns = [col_header])
            bulk_recommendations_df = bulk_recommendations_df.join(new_rec_DF)
        else:
            bulk_recommendations_df = pd.DataFrame(recommendation_list, columns=[col_header])
        try:
            line = json.loads(json_file.readline())
        except:
            line = None
bulk_recommendations_df<jupyter_output><empty_output><jupyter_text>In the cell above you can see the various recommendations for the users provided, in a real scenario the list of users could be your entire userbase allowing you to quickly reference and compare results between them.  Extra Bonus: CleanupThe cells below are totally optional if you are using Event Engine as your account is deleted after the workshop. If you'd like to use this in your own account later, the cells below will delete the resources created to prevent continous charges.<jupyter_code># Delete the campaign:
personalize.delete_campaign(campaignArn=campaign_arn)
time.sleep(60)
# Delete the solution
personalize.delete_solution(solutionArn=solution_arn)
time.sleep(60)
# Delete the event tracker
personalize.delete_event_tracker(eventTrackerArn=event_tracker_arn)
time.sleep(60)
# Delete the interaction dataset
personalize.delete_dataset(datasetArn=dataset_arn)
time.sleep(60)
# Delete the event dataset
event_interactions_dataset_arn = dataset_arn
event_interactions_dataset_arn = event_interactions_dataset_arn.replace("INTERACTIONS", "EVENT_INTERACTIONS")
personalize.delete_dataset(datasetArn=event_interactions_dataset_arn)
time.sleep(60)
# Delete the schema
personalize.delete_schema(schemaArn=schema_arn)
# Delete the Dataset Group
personalize.delete_dataset_group(datasetGroupArn=dataset_group_arn)
# Empty the S3 Bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
bucket.objects.all().delete()
time.sleep(60)
# Delete the S3 Bucket
bucket.delete()
# IAM policies should also be removed
iam = boto3.client("iam")
iam.detach_role_policy(PolicyArn="arn:aws:iam::aws:policy/AmazonS3FullAccess", RoleName=role_name)
iam.detach_role_policy(PolicyArn="arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess",RoleName=role_name)

iam.delete_role(RoleName=role_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0789806439062348, 'fraction_numerical': 0.007359854748241092, 'mean_word_length': 4.9648037130148905, 'pattern_counts': {'":': 39, '<': 150, '<?xml version=': 0, '>': 145, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '569369', 'license': 'MIT-0', 'n_tokens_mistral': 8997, 'n_tokens_neox': 8668, 'n_words': 3034, 'path': 'next_steps/workshops/Reinvent_2019/ReInvent2019_Workshop.ipynb', 'repo_name': 'mikejgillespie/amazon-personalize-samples', 'size': '30843'}