<jupyter_start><jupyter_text>Interacting with Campaigns In this notebook, you will deploy and interact with campaigns in Amazon Personalize.1. [Introduction](intro)1. [Create campaigns](create)1. [Interact with campaigns](interact)1. [Batch recommendations](batch)1. [Wrap up](wrapup) Introduction [Back to top](top)At this point, you should have several solutions and at least one solution version for each. Once a solution version is created, it is possible to get recommendations from them, and to get a feel for their overall behavior.This notebook starts off by deploying each of the solution versions from the previous notebook into individual campaigns. Once they are active, there are resources for querying the recommendations, and helper functions to digest the output into something more human-readable. As you with your customer on Amazon Personalize, you can modify the helper functions to fit the structure of their data input files to keep the additional rendering working.To get started, once again, we need to import libraries, load values from previous notebooks, and load the SDK.<jupyter_code>import time
#import datetime
from time import sleep
import json
from datetime import datetime
import uuid
import random

import boto3, botocore
import pandas as pd
import numpy as np
%store -r
personalize = boto3.client('personalize')
personalize_runtime = boto3.client('personalize-runtime')

# Establish a connection to Personalize's event streaming
personalize_events = boto3.client(service_name='personalize-events')<jupyter_output><empty_output><jupyter_text>Add New items to Catalog [Back to top](top)Now that all campaigns are deployed and active, we will need to add content to our catalog as it is released. Given that this content is completely new, it will not have interactions in the dataset for the user personalization solution to train on. Due to this we will use the exploration capability in the user personalization campaign. First we will get the current time, which we will then use as the CREATION_TIMESTAMP for our new items. Amazon Personalize uses creation timestamp data (in UNIX epoch time format, in seconds) to calculate the age of an item and adjust recommendations accordingly.If creation timestamp data is missing for one or more items, Amazon Personalize infers this information from interaction data, if any, and uses the timestamp of the item’s oldest interaction data as the item's creation date. If an item has no interaction data, its creation date is set as the timestamp of the latest interaction in the training set and is considered a new item. The creation timestamp data will be added to the user personalization model (as long as exploration value is above 0 and the creation timestamp is within the exploration_item_age_cut_off.<jupyter_code>currenttime = round(time.time()) - 1000
currentyear = datetime.utcfromtimestamp(currenttime).strftime('%Y')
new_titles = {'ITEM_ID': [itemmetadata_df.last_valid_index() + 1, itemmetadata_df.last_valid_index() + 2, itemmetadata_df.last_valid_index() + 3, itemmetadata_df.last_valid_index() + 4], 'title': ['New Title 1' + " (" + str(currentyear) + ")", 'New Title 2' + " (" + str(currentyear) + ")", 'New Title 3' + " (" + str(currentyear) + ")", 'New Title 4' + " (" + str(currentyear) + ")"],'GENRE': ['Action|Comedy', 'Sci-Fi|Fantasy', 'Drama|Thriller', 'Documentary|IMAX'],'YEAR': [currentyear, currentyear, currentyear, currentyear],'CREATION_TIMESTAMP': [round(time.time()) - 1000, round(time.time()) - 2000, round(time.time()) - 3000, round(time.time())]}
new_titles_df = pd.DataFrame(new_titles)
new_titles_df=new_titles_df.astype(dtype= {"ITEM_ID":"int64", "title":"object", "GENRE":"object","YEAR":"object","CREATION_TIMESTAMP":"int64"})
new_titles_df=new_titles_df.set_index('ITEM_ID')
new_titles_df
items_df
itemmetadata_df = itemmetadata_df.append(new_titles_df, ignore_index=False)
itemmetadata_df = itemmetadata_df[['GENRE','YEAR','CREATION_TIMESTAMP']]
itemmetadata_df.tail(10)<jupyter_output><empty_output><jupyter_text>We will also add the new titles to the look up table from the previous notebook<jupyter_code>new_items_df = new_titles_df[['title']]
new_items_df = new_items_df.rename_axis('movieId')
items_df = items_df.append(new_items_df, ignore_index=False)
items_df
items_df<jupyter_output><empty_output><jupyter_text>now we will add the titles to item metadata via a batch import process. Note, you can also add these in realtime using the putitems api (see code example below) ```pythondef add_title(item_id, genre, year, creationtimestamp):       itemId= str(item_id)    print(itemId)    itemdata = {        "genre": genre,        "year": year,        "creationTimestamp": creationtimestamp    } Make Call    personalize_events.put_items(    datasetArn=items_dataset_arn,    items=[        {            'itemId': str(itemId),            'properties': json.dumps(itemdata)        },    ]    )```and then execute by calling the function```pythonfor index, row in new_titles_df.iterrows():    print("Adding Item " + str(index) + " " + row['title'] + " with genres " + row['genres'])    add_title(item_id=index,genre=row['genres'],year=row['year'],creationtimestamp=row['CREATION_TIMESTAMP'])``` Similar to 02_Validating_and_Importing_Item_Metadata.ipynb we will store the updated item_metadata_df to a csv file and upload it to S3<jupyter_code>itemmetadata_filename = "item-meta.csv"
itemmetadata_df.to_csv((data_dir+"/"+itemmetadata_filename), index=True, float_format='%.0f')
itemmetadata_file_path = data_dir + "/" + itemmetadata_filename
boto3.Session().resource('s3').Bucket(bucket_name).Object(itemmetadata_filename).upload_file(itemmetadata_file_path)
interactions_s3DataPath = "s3://"+bucket_name+"/"+itemmetadata_filename<jupyter_output><empty_output><jupyter_text>Once the updated file is in S3, we will start a dataset import job<jupyter_code>create_dataset_import_job_response = personalize.create_dataset_import_job(
    jobName = "personalize-poc-item-import" + str(currenttime),
    datasetArn = items_dataset_arn,
    dataSource = {
        "dataLocation": "s3://{}/{}".format(bucket_name, itemmetadata_filename)
    },
    roleArn = role_arn
)

dataset_import_job_arn = create_dataset_import_job_response['datasetImportJobArn']
print(json.dumps(create_dataset_import_job_response, indent=2))<jupyter_output><empty_output><jupyter_text>We will check the status of the import.<jupyter_code>%%time

max_time = time.time() + 6*60*60 # 6 hours
while time.time() < max_time:
    describe_dataset_import_job_response = personalize.describe_dataset_import_job(
        datasetImportJobArn = dataset_import_job_arn
    )
    status = describe_dataset_import_job_response["datasetImportJob"]['status']
    print("DatasetImportJob: {}".format(status))
    
    if status == "ACTIVE" or status == "CREATE FAILED":
        break
        
    time.sleep(60)<jupyter_output><empty_output><jupyter_text>To update the model(solutionVersion), we ran the createSolutionVersion with trainingMode set to UPDATE. This updates the model with the latest item information and adjusts the exploration according to implicit feedback from the users. This is not equivalent to training a model, which you can do by setting trainingMode to FULL. You should perform full training less frequently, typically once every 1–5 days. When the new updated solutionVersion is created, you can update the campaign to get recommendations using it. NOTE: This happens automatically every 2 hours, for the purposes of this workshop we are running it on demand.<jupyter_code>user_personalization_update_solution_response = personalize.create_solution_version(
    solutionArn = user_personalization_solution_arn, 
    trainingMode='UPDATE')
new_user_personalization_solution_version_arn = user_personalization_update_solution_response['solutionVersionArn']
print("Creating solution version: {}".format(new_user_personalization_solution_version_arn))

status = None
max_time = time.time() + 60*60 # 1 hour
while time.time() < max_time:
    describe_solution_version_response = personalize.describe_solution_version(
    solutionVersionArn = new_user_personalization_solution_version_arn
    )
    status = describe_solution_version_response["solutionVersion"]["status"]
    print("SolutionVersion: {}".format(status))

    if status == "ACTIVE" or status == "CREATE FAILED":
         break
    time.sleep(60)<jupyter_output><empty_output><jupyter_text>Now that the solution is updated with the new solution version, including item-metadata information about our new titles, we can update the campaign to use this version.<jupyter_code>userpersonalization_update_campaign_response = personalize.update_campaign(campaignArn=userpersonalization_campaign_arn, solutionVersionArn=new_user_personalization_solution_version_arn)
userpersonalization_campaign_arn = userpersonalization_update_campaign_response['campaignArn']
print(json.dumps(userpersonalization_update_campaign_response, indent=2))<jupyter_output><empty_output><jupyter_text>Now lets check the status of the solution version update.<jupyter_code>describe_campaign_update_response = personalize.describe_campaign(
            campaignArn = userpersonalization_campaign_arn)
status = describe_campaign_update_response["campaign"]["status"]
print(status)<jupyter_output><empty_output><jupyter_text>Note that the campaign itself is ACTIVE, however the latestCampaignUpdate is either in CREATE PENDING or CREATE IN_PROGRESS, the campaign will remain active with the previous solution version until the new solution version is active at which point it will become active in the current campaign. We will watch for the latestCampaignUpdate to become ACTIVE. Our new items will become available once the campaign is active with the new solution version.<jupyter_code>status = None
max_time = time.time() + 60*60 # 1 hour
while time.time() < max_time:
    describe_campaign_update_response = personalize.describe_campaign(
            campaignArn = userpersonalization_campaign_arn
        )
    status = describe_campaign_update_response["campaign"]["latestCampaignUpdate"]["status"]
    print("CampaignUpdate: {}".format(status))

    if status == "ACTIVE" or status == "CREATE FAILED":
         break
    time.sleep(60)<jupyter_output><empty_output><jupyter_text>We will create a movie lookup function as in the '05_Interacting_with_Campaigns_and_Filters.ipynb' notebook<jupyter_code>def get_movie_by_id(movie_id, movie_df=items_df):
    """
    This takes in an movie_id from Personalize so it will be a string,
    converts it to an int, and then does a lookup in a default or specified
    dataframe.
    
    A really broad try/except clause was added in case anything goes wrong.
    
    Feel free to add more debugging or filtering here to improve results if
    you hit an error.
    """
    try:
        return movie_df.loc[int(movie_id)]['title']
    except:
        return "Error obtaining title"<jupyter_output><empty_output><jupyter_text>Now let's test a few simple values to check our error catching.<jupyter_code># A known good id (NewTitle4)
print(get_movie_by_id(movie_id=str(items_df.last_valid_index())))
# A bad type of value
print(get_movie_by_id(movie_id='987.9393939'))
# Really bad values
print(get_movie_by_id(movie_id='Steve'))<jupyter_output><empty_output><jupyter_text>Great! Now we have a way of rendering results.<jupyter_code># Update DF rendering
pd.set_option('display.max_rows', 30)

def get_new_recommendations_df(recommendations_df, movie_ID):
    # Get the movie name
    movie_name = get_movie_by_id(movie_ID)
    # Get the recommendations
    get_recommendations_response = personalize_runtime.get_recommendations(
        campaignArn = sims_campaign_arn,
        itemId = str(movie_ID),
    )
    # Build a new dataframe of recommendations
    item_list = get_recommendations_response['itemList']
    recommendation_list = []
    for item in item_list:
        movie = get_movie_by_id(item['itemId'])
        recommendation_list.append(movie)
    new_rec_DF = pd.DataFrame(recommendation_list, columns = [movie_name])
    # Add this dataframe to the old one
    recommendations_df = pd.concat([recommendations_df, new_rec_DF], axis=1)
    return recommendations_df<jupyter_output><empty_output><jupyter_text>User PersonalizationHRNN is one of the more advanced algorithms provided by Amazon Personalize, and is the underlying algorithm in the user personalization recipe. It supports personalization of the items for a specific user based on their past behavior and can intake real time events in order to alter recommendations for a user without retraining. Since HRNN relies on having a sampling of users, let's load the data we need for that and select 3 random users. Since Movielens does not include user data, we will select 3 random numbers from the range of user id's in the dataset.<jupyter_code>if not USE_FULL_MOVIELENS:
    users = random.sample(range(1, 600), 10)
else:
    users = random.sample(range(1, 162000), 10)
users<jupyter_output><empty_output><jupyter_text>Now we render the recommendations for our 3 random users from above. After that, we will explore real-time interactions before moving on to Personalized Ranking.Again, we create a helper function to render the results in a nice dataframe. API call results<jupyter_code># Update DF rendering
pd.set_option('display.max_rows', 30)

def get_new_recommendations_df_users(recommendations_df, user_id):
    # Get the movie name
    #movie_name = get_movie_by_id(artist_ID)
    # Get the recommendations
    get_recommendations_response = personalize_runtime.get_recommendations(
        campaignArn = userpersonalization_campaign_arn,
        userId = str(user_id),
    )
    # Build a new dataframe of recommendations
    item_list = get_recommendations_response['itemList']
    recommendation_list = []
    for item in item_list:
        movie = get_movie_by_id(item['itemId'])
        recommendation_list.append(movie)
    #print(recommendation_list)
    new_rec_DF = pd.DataFrame(recommendation_list, columns = [user_id])
    # Add this dataframe to the old one
    recommendations_df = pd.concat([recommendations_df, new_rec_DF], axis=1)
    return recommendations_df
recommendations_df_users = pd.DataFrame()
#users = users_df.sample(3).index.tolist()

for user in users:
    recommendations_df_users = get_new_recommendations_df_users(recommendations_df_users, user)

recommendations_df_users<jupyter_output><empty_output><jupyter_text>Here we can see the cold items are being recommended based on the exploration weight set when we created the campaign.Now lets send in some interactions from the sample users to simulate viewers engaging with this cold start content.<jupyter_code>session_dict = {}

def send_movie_click(USER_ID, ITEM_ID, EVENT_TYPE):
    """
    Simulates a click as an envent
    to send an event to Amazon Personalize's Event Tracker
    """
    # Configure Session
    try:
        session_ID = session_dict[str(USER_ID)]
    except:
        session_dict[str(USER_ID)] = str(uuid.uuid1())
        session_ID = session_dict[str(USER_ID)]
        
    # Configure Properties:
    event = {
    "itemId": str(ITEM_ID),
    }
    event_json = json.dumps(event)
        
    # Make Call
    
    response = personalize_events.put_events(
    trackingId = TRACKING_ID,
    userId= str(USER_ID),
    sessionId = session_ID,
    eventList = [{
        'sentAt': int(time.time()),
        'eventType': str(EVENT_TYPE),
        'properties': event_json
        }]
    )
   
def get_new_recommendations_df_users_real_time(recommendations_df, user_id, item_id, event_type):
    # Get the artist name (header of column)
    movie_name = get_movie_by_id(item_id)
    # Interact with different movies
    print('sending event ' + event_type + ' for ' + get_movie_by_id(item_id))
    send_movie_click(USER_ID=user_id, ITEM_ID=item_id, EVENT_TYPE=event_type)
    # Get the recommendations (note you should have a base recommendation DF created before)
    get_recommendations_response = personalize_runtime.get_recommendations(
        campaignArn = userpersonalization_campaign_arn,
        userId = str(user_id),
    )
    # Build a new dataframe of recommendations
    item_list = get_recommendations_response['itemList']
    recommendation_list = []
    for item in item_list:
        artist = get_movie_by_id(item['itemId'])
        recommendation_list.append(artist)
    new_rec_DF = pd.DataFrame(recommendation_list, columns = [movie_name])
    # Add this dataframe to the old one
    #recommendations_df = recommendations_df.join(new_rec_DF)
    recommendations_df = pd.concat([recommendations_df, new_rec_DF], axis=1)
    return recommendations_df<jupyter_output><empty_output><jupyter_text>We will iterate through all of the sammple users and both click on and watch the 4 new titles.<jupyter_code># Note this will take about 20 seconds to complete due to the sleeps
for user in users:
    print ('user ' + str(user))
    for index, row in new_titles_df.iterrows():
        user_id=user
        print('sending event click for ' + get_movie_by_id(index))
        send_movie_click(user_id, index,'click')
        print('sending event watch for ' + get_movie_by_id(index))
        send_movie_click(user_id, index,'watch')
        sleep(2)<jupyter_output><empty_output><jupyter_text>Now lets apply item filters to see recommendations for one of these users within a genre<jupyter_code>if not USE_FULL_MOVIELENS:
    users = random.sample(range(1, 600), 3)
else:
    users = random.sample(range(1, 162000), 3)
users
recommendations_df_users = pd.DataFrame()
#users = users_df.sample(3).index.tolist()

for user in users:
    recommendations_df_users = get_new_recommendations_df_users(recommendations_df_users, user)

recommendations_df_users<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08244651058586311, 'fraction_numerical': 0.007561330794219783, 'mean_word_length': 5.313649222065064, 'pattern_counts': {'":': 13, '<': 76, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '486031', 'license': 'MIT-0', 'n_tokens_mistral': 5377, 'n_tokens_neox': 5182, 'n_words': 1576, 'path': 'next_steps/workshops/POC_in_a_box/completed/06_Adding_New_Items_and_Updating_Model.ipynb', 'repo_name': 'lmorri/amazon-personalize-samples', 'size': '17854'}