<jupyter_start><jupyter_text>Recommender Systems 2020/21 Practice 3 - Content Based recommenders Load the data you saw last time:<jupyter_code>from urllib.request import urlretrieve
import zipfile, os

# If file exists, skip the download
data_file_path = "data/Movielens_10M/"
data_file_name = data_file_path + "movielens_10m.zip"

# If directory does not exist, create
if not os.path.exists(data_file_path):
    os.makedirs(data_file_path)

if not os.path.exists(data_file_name):
    urlretrieve ("http://files.grouplens.org/datasets/movielens/ml-10m.zip", data_file_name)
    
dataFile = zipfile.ZipFile(data_file_name)
URM_path = dataFile.extract("ml-10M100K/ratings.dat", path="data/Movielens_10M")

import pandas as pd

URM_all_dataframe = pd.read_csv(filepath_or_buffer=URM_path, 
                                sep="::", 
                                header=None, 
                                dtype={0:int, 1:int, 2:float, 3:int})

URM_all_dataframe.columns = ["UserID", "ItemID", "Interaction", "Timestamp"]
URM_all_dataframe.head(n=10)<jupyter_output><empty_output><jupyter_text>And now load the content informations in the same way: In this case we are using tags. The ICM also contains the user that added the tag in column 0 (but we don't use that information).<jupyter_code>ICM_path = dataFile.extract("ml-10M100K/tags.dat", path = "data/Movielens_10M")

ICM_dataframe = pd.read_csv(filepath_or_buffer=ICM_path, 
                            sep="::", 
                            header=None, 
                            dtype={0:int, 1:int, 2:str, 3:int})

ICM_dataframe.columns = ["UserID", "ItemID", "FeatureID", "Timestamp"]
ICM_dataframe.head(n=10)<jupyter_output><empty_output><jupyter_text>We can see that most users and items have no data associated to them<jupyter_code>n_features = len(ICM_dataframe["FeatureID"].unique())

print ("Number of tags\t {}, Number of item-tag tuples {}".format(n_features, len(ICM_dataframe)))<jupyter_output>Number of tags	 16529, Number of item-tag tuples 95580<jupyter_text>We now build the sparse URM and ICM matrices The tags are strings, we should traslate them into numbers so we can use them as indices in the ICM. We should also ensure that the item and user indices we use in ICM and URM are consistent. To do so we use the same mapper, first we populate it with the URM and then we add the new ids that appear only in the ICM<jupyter_code>user_original_ID_to_index_dict = {}

for user_id in URM_all_dataframe["UserID"].unique():
    user_original_ID_to_index_dict[user_id] = len(user_original_ID_to_index_dict)  

print("Unique user_id in the URM are {}".format(len(user_original_ID_to_index_dict)))
    
for user_id in ICM_dataframe["UserID"].unique():
    if user_id not in user_original_ID_to_index_dict:
        user_original_ID_to_index_dict[user_id] = len(user_original_ID_to_index_dict)
        
print("Unique user_id in the URM and ICM are {}".format(len(user_original_ID_to_index_dict)))
item_original_ID_to_index_dict = {}

for item_id in URM_all_dataframe["ItemID"].unique():
    item_original_ID_to_index_dict[item_id] = len(item_original_ID_to_index_dict)

print("Unique item_id in the URM are {}".format(len(item_original_ID_to_index_dict)))
    
for item_id in ICM_dataframe["ItemID"].unique():
    if item_id not in item_original_ID_to_index_dict:
        item_original_ID_to_index_dict[item_id] = len(item_original_ID_to_index_dict)
        
print("Unique item_id in the URM and ICM are {}".format(len(item_original_ID_to_index_dict)))
feature_original_ID_to_index_dict = {}

for feature_id in ICM_dataframe["FeatureID"].unique():
    feature_original_ID_to_index_dict[feature_id] = len(feature_original_ID_to_index_dict)

print("Unique feature_id in the URM are {}".format(len(feature_original_ID_to_index_dict)))
original_feature_ID = "star wars"
print("New index for feature '{}' is {}".format(original_feature_ID, feature_original_ID_to_index_dict[original_feature_ID]))<jupyter_output>New index for feature 'star wars' is 9<jupyter_text>We can now build the URM and ICM using the new indices<jupyter_code>URM_all_dataframe["UserID"] = [user_original_ID_to_index_dict[user_original] for user_original in
                                      URM_all_dataframe["UserID"].values]

URM_all_dataframe["ItemID"] = [item_original_ID_to_index_dict[item_original] for item_original in 
                                      URM_all_dataframe["ItemID"].values]
URM_all_dataframe.head(n=10)
ICM_dataframe["UserID"] = [user_original_ID_to_index_dict[user_original] for user_original in
                                      ICM_dataframe["UserID"].values]

ICM_dataframe["ItemID"] = [item_original_ID_to_index_dict[item_original] for item_original in 
                                      ICM_dataframe["ItemID"].values]

ICM_dataframe["FeatureID"] = [feature_original_ID_to_index_dict[feature_original] for feature_original in 
                                      ICM_dataframe["FeatureID"].values]
ICM_dataframe.head(n=10)<jupyter_output><empty_output><jupyter_text>When we create the matrices we need to make sure they have the same shape. As we have seen some items and users only appear in one of the two matrices and are missing in the other (if they have no tags or no interactions)<jupyter_code>import scipy.sparse as sps
import numpy as np

n_users = len(user_original_ID_to_index_dict)
n_items = len(item_original_ID_to_index_dict)
n_features = len(feature_original_ID_to_index_dict)
URM_all = sps.csr_matrix((URM_all_dataframe["Interaction"].values, 
                          (URM_all_dataframe["UserID"].values, URM_all_dataframe["ItemID"].values)),
                        shape = (n_users, n_items))

URM_all
ICM_all = sps.csr_matrix((np.ones(len(ICM_dataframe["ItemID"].values)), 
                          (ICM_dataframe["ItemID"].values, ICM_dataframe["FeatureID"].values)),
                        shape = (n_items, n_features))

ICM_all.data = np.ones_like(ICM_all.data)

ICM_all<jupyter_output><empty_output><jupyter_text>Let's take a look at the ICM We leverage CSR and CSC indptr data structure to compute the number of cells that have values for that row or column<jupyter_code>ICM_all = sps.csr_matrix(ICM_all)
features_per_item = np.ediff1d(ICM_all.indptr)

ICM_all = sps.csc_matrix(ICM_all)
items_per_feature = np.ediff1d(ICM_all.indptr)

ICM_all = sps.csr_matrix(ICM_all)
print(features_per_item.shape)
print(items_per_feature.shape)
features_per_item = np.sort(features_per_item)
items_per_feature = np.sort(items_per_feature)
import matplotlib.pyplot as pyplot
%matplotlib inline  

pyplot.plot(features_per_item, 'ro')
pyplot.ylabel('Num features ')
pyplot.xlabel('Sorted items')
pyplot.show()
pyplot.plot(items_per_feature, 'ro')
pyplot.ylabel('Num items ')
pyplot.xlabel('Sorted features')
pyplot.show()<jupyter_output><empty_output><jupyter_text>We can now build the recommender algorithm, but first we need the train/test split and the evaluation function:<jupyter_code>from Base.Evaluation.Evaluator import EvaluatorHoldout
from Data_manager.split_functions.split_train_validation_random_holdout import split_train_in_two_percentage_global_sample

URM_train, URM_test = split_train_in_two_percentage_global_sample(URM_all, train_percentage = 0.80)
URM_train, URM_validation = split_train_in_two_percentage_global_sample(URM_train, train_percentage = 0.80)

evaluator_validation = EvaluatorHoldout(URM_validation, cutoff_list=[10])
evaluator_test = EvaluatorHoldout(URM_test, cutoff_list=[10])

from Base.Similarity.Compute_Similarity_Python import Compute_Similarity_Python

class ItemKNNCBFRecommender(object):
    
    def __init__(self, URM, ICM):
        self.URM = URM
        self.ICM = ICM
        
            
    def fit(self, topK=50, shrink=100, normalize = True, similarity = "cosine"):
        
        similarity_object = Compute_Similarity_Python(self.ICM.T, shrink=shrink, 
                                                  topK=topK, normalize=normalize, 
                                                  similarity = similarity)
        
        self.W_sparse = similarity_object.compute_similarity()

        
    def recommend(self, user_id, at=None, exclude_seen=True):
        # compute the scores using the dot product
        user_profile = self.URM[user_id]
        scores = user_profile.dot(self.W_sparse).toarray().ravel()

        if exclude_seen:
            scores = self.filter_seen(user_id, scores)

        # rank items
        ranking = scores.argsort()[::-1]
            
        return ranking[:at]
    
    
    def filter_seen(self, user_id, scores):

        start_pos = self.URM.indptr[user_id]
        end_pos = self.URM.indptr[user_id+1]

        user_profile = self.URM.indices[start_pos:end_pos]
        
        scores[user_profile] = -np.inf

        return scores<jupyter_output><empty_output><jupyter_text>We need to define Cosine similarity... Let's look at the attached source code  See also a [list of commonly used KNN similarity heuristics](https://github.com/MaurizioFD/RecSys_Course_2018/blob/master/slides/List_of_KNN_similarity_heuristics.pdf)  A KNN is built with the following steps:* Compute the similarity of an item with all others* Select the k-highest similarities<jupyter_code>item_id = 80
shrink = 10<jupyter_output><empty_output><jupyter_text>The numerator is the dot product of the item features times the whole ICM data transposed<jupyter_code>numerator_vector = ICM_all[item_id].dot(ICM_all.T).toarray().ravel()
numerator_vector
item_norms = np.sqrt(np.array(ICM_all.T.power(2).sum(axis=0))).ravel()
item_norms<jupyter_output><empty_output><jupyter_text>The denominator will be the product of norms plus the srink term and a small value which prevents the denominator to be zero (only for non-negative data)<jupyter_code>denominator_vector = item_norms[item_id] * item_norms + shrink + 1e-6
denominator_vector
similarity_vector = numerator_vector/denominator_vector<jupyter_output><empty_output><jupyter_text>Let's sort the similarity from the highest to the lowest<jupyter_code>sorted_item_indices = np.argsort(-similarity_vector)
sorted_item_indices
pyplot.plot(similarity_vector[sorted_item_indices], 'ro')
pyplot.ylabel('Similarity')
pyplot.xlabel('Item')
pyplot.show()<jupyter_output><empty_output><jupyter_text>Now we select the k most similar items<jupyter_code>k = 100
pyplot.plot(similarity_vector[sorted_item_indices[0:k]], 'ro')
pyplot.ylabel('Similarity')
pyplot.xlabel('Item')
pyplot.show()<jupyter_output><empty_output><jupyter_text>The operation is performed for all items.A simple strategy to perform it efficiently is to vectorize the most computationally intensive part, the dot product, on a group of items. The speedup can be of a factor of 10-100.This strategy is limited by the fact that the result of the dot product is a huge item-item dense similarity which likely does not fit in memory.<jupyter_code>import traceback

try:
    numerator_matrix = ICM_all.dot(ICM_all.T).toarray()
    
except Exception as e:
    traceback.print_exc()<jupyter_output><empty_output><jupyter_text>The solution is:* Compute the numerator a block of items at a time leveraging vectorization while not running out of memory* Extract the k-nn on those items* Built incrementally the sparse similarity matrix<jupyter_code>block_size = 100

numerator_block = ICM_all[0:block_size].dot(ICM_all.T).toarray()
numerator_block.shape<jupyter_output><empty_output><jupyter_text>Let's compare the speed to compute the dot product on the whole similarity of the two strategies<jupyter_code>import time

n_items = ICM_all.shape[0]

start_time = time.time()

for n_item in range(n_items):
    numerator_vector = ICM_all[item_id].dot(ICM_all.T).toarray().ravel()
    
end_time = time.time()

print("Computing the similarity one item at a time runs at {:.2f} items/sec".format(n_items/(end_time-start_time)))


n_items = ICM_all.shape[0]

start_time = time.time()

block_size = 100
blocks_start_positions = range(0, n_items, block_size)

for start_pos in blocks_start_positions:
    end_pos = min(start_pos + block_size, n_items)
    
    numerator_block = ICM_all[start_pos:end_pos].dot(ICM_all.T).toarray()
    
end_time = time.time()

print("Computing the similarity in blocks of 100 items at a time runs at {:.2f} items/sec".format(n_items/(end_time-start_time)))<jupyter_output>Computing the similarity in blocks of 100 items at a time runs at 24707.12 items/sec<jupyter_text>In this case the vectorized implementation runs >50 times faster! Usually most of the speed gain comes with blocks of 100 or so items, depending on the system. Much higher than that tends to not be beneficial while requiring increasingly more memory. Now an example of something you should *never* do, nested loops to compute the similarity of each item without vectorization<jupyter_code>n_items = 100

start_time = time.time()

for n_item in range(n_items):
    for second_item in range(n_items):
        numerator_vector = ICM_all[item_id].dot(ICM_all[second_item].T)
    
end_time = time.time()

print("Computing the similarity with nested loops runs at {:.2f} items/sec".format(n_items/(end_time-start_time)))<jupyter_output>Computing the similarity with nested loops runs at 25.64 items/sec<jupyter_text>You see how incredibly slow nested loops are compared to a well vectorized implementation  Test our CBF recommender:<jupyter_code>recommender = ItemKNNCBFRecommender(URM_train, ICM_all)
recommender.fit(shrink=0.0, topK=50)
for user_id in range(10):
    print(recommender.recommend(user_id, at=5))<jupyter_output>[4759  554  851   38 3256]
[ 402  133    7 8913  192]
[ 798 5138 5018 9277 5660]
[8913  423  468 6833  175]
[ 176  403 1184 1314    7]
[ 37  19  24 798  60]
[ 144 1139 1184  150 2207]
[3736 1234 2065 2952 2093]
[ 101  616 4584 1994  788]
[1310 6578 7118 6791   26]<jupyter_text>Let's talk about speed Time to compute recommendations for a fixed group of users<jupyter_code>import time

n_users_to_test = 1000

start_time = time.time()

for user_id in range(n_users_to_test):
    recommender.recommend(user_id, at=5)
    
end_time = time.time()

print("Reasonable implementation speed is {:.2f} usr/sec".format(n_users_to_test/(end_time-start_time)))<jupyter_output>Reasonable implementation speed is 533.75 usr/sec<jupyter_text>Let's add a common mistake.... a CSC URM<jupyter_code>URM_train_csc = URM_train.tocsc()

recommender = ItemKNNCBFRecommender(URM_train_csc, ICM_all)
recommender.fit(shrink=0.0, topK=50)
import time, traceback

try:

    n_users_to_test = 1000

    start_time = time.time()

    for user_id in range(n_users_to_test):
        recommender.recommend(user_id, at=5)

    end_time = time.time()

    print("Wrong implementation speed is {:.2f} usr/sec".format(n_users_to_test/(end_time-start_time)))

    
except Exception as e:
        
    print("Exception {}".format(str(e)))
    traceback.print_exc()<jupyter_output>Exception index 10691 is out of bounds for axis 0 with size 10681<jupyter_text>Hyperparameter tuning Once we have built our model we can play with its hyperparameters* Number of neighbors* Shrinkage* Similarity type  Number of neighbors<jupyter_code>from KNN.ItemKNNCBFRecommender import ItemKNNCBFRecommender

x_tick = [10, 50, 100, 200, 500]
MAP_per_k = []

for topK in x_tick:
    
    recommender = ItemKNNCBFRecommender(URM_train, ICM_all)
    recommender.fit(shrink=0.0, topK=topK)
    
    result_dict, _ = evaluator_test.evaluateRecommender(recommender)
    
    MAP_per_k.append(result_dict[10]["MAP"])
pyplot.plot(x_tick, MAP_per_k)
pyplot.ylabel('MAP')
pyplot.xlabel('TopK')
pyplot.show()<jupyter_output><empty_output><jupyter_text>On this dataset the number of neighbors does not have a great impact on MAP. Higher values of TopK might work even better Different datasets will behave in different ways.  Shrinkage<jupyter_code>x_tick = [0, 10, 50, 100, 200, 500]
MAP_per_shrinkage = []

for shrink in x_tick:
    
    recommender = ItemKNNCBFRecommender(URM_train, ICM_all)
    recommender.fit(shrink=shrink, topK=100)
    
    result_dict, _ = evaluator_test.evaluateRecommender(recommender)
    
    MAP_per_shrinkage.append(result_dict[10]["MAP"])
pyplot.plot(x_tick, MAP_per_shrinkage)
pyplot.ylabel('MAP')
pyplot.xlabel('Shrinkage')
pyplot.show()<jupyter_output><empty_output><jupyter_text>The shrinkage value (i.e. support) have a much stronger impact. Combine a parameter search with the two to ensure maximum recommendation quality Be careful, overfitting! While a thorough parameter tuning might result in significantly higher MAP on your validation split, it could have only marginally better or even worse MAP on the test set  Feature weighting  IDF ![title](https://www.link-assistant.com/images/news/tf-idf-tool-for-seo/screen-03.png)<jupyter_code>num_tot_items = ICM_all.shape[0]

# let's count how many items have a certain feature
items_per_feature = np.ediff1d(ICM_all.indptr)

IDF = np.array(np.log(num_tot_items / items_per_feature))

print(ICM_all.shape)
print(IDF.shape)
IDF
pyplot.plot(np.sort(IDF))
pyplot.ylabel('IDF')
pyplot.xlabel('Sorted feature')
pyplot.show()<jupyter_output><empty_output><jupyter_text>Highest ranked features<jupyter_code>feature_index_to_original_ID_dict = {value:key for key,value in feature_original_ID_to_index_dict.items()}
sorted_features = np.argsort(-IDF)

highest_ranked_features = sorted_features[:20]

for feature_index in highest_ranked_features:
    print(feature_index_to_original_ID_dict[feature_index])<jupyter_output>Dance
<NAME>
<NAME>
solidarity
author:<NAME>
utah
transexual
<NAME>
<NAME>
Bonneville Salt Flats
legal system
<NAME> and <NAME> also together in Hidden (a.k.a. Cache) (2005)
score:George Fenton
tear jerker
R:brief language
<NAME>
great war depiction
champagne
based upon a true story
colonial power<jupyter_text>Lowest ranked features<jupyter_code>lowest_ranked_features = sorted_features[-20:]

for feature_index in lowest_ranked_features:
    print(feature_index_to_original_ID_dict[feature_index])    
from scipy.sparse import diags
diags(IDF)
ICM_idf = ICM_all.copy()

ICM_idf = diags(IDF)*ICM_idf
recommender_idf = ItemKNNCBFRecommender(URM_train, ICM_idf)
recommender_idf.fit(shrink=0.0, topK=50)

result_dict, _ = evaluator_test.evaluateRecommender(recommender_idf)
result_dict[10]<jupyter_output>ItemKNNCBFRecommender: URM Detected 1689 (2.36 %) cold users.
ItemKNNCBFRecommender: URM Detected 56 (0.52 %) cold items.
Similarity column 10681 ( 100 % ), 13567.67 column/sec, elapsed time 0.01 min
EvaluatorHoldout: Processed 51000 ( 73.06% ) in 30.14 sec. Users per second: 1692
EvaluatorHoldout: Processed 69804 ( 100.00% ) in 42.06 sec. Users per second: 1660<jupyter_text>There is  a small gain over the non-weighted ICM. Try other feature weighting methods like BM25...  BM25<jupyter_code>from Base.IR_feature_weighting import okapi_BM_25

ICM_BM25 = ICM_all.copy().astype(np.float32)
ICM_BM25 = okapi_BM_25(ICM_BM25)
ICM_BM25 = ICM_BM25.tocsr()
recommender_bm25 = ItemKNNCBFRecommender(URM_train, ICM_BM25)
recommender_bm25.fit(shrink=0.0, topK=50)

result_dict, _ = evaluator_test.evaluateRecommender(recommender_bm25)
result_dict[10]<jupyter_output>ItemKNNCBFRecommender: URM Detected 1689 (2.36 %) cold users.
ItemKNNCBFRecommender: URM Detected 56 (0.52 %) cold items.
Similarity column 10681 ( 100 % ), 11635.44 column/sec, elapsed time 0.02 min
EvaluatorHoldout: Processed 51000 ( 73.06% ) in 30.18 sec. Users per second: 1690
EvaluatorHoldout: Processed 69804 ( 100.00% ) in 40.42 sec. Users per second: 1727<jupyter_text>Another small gain over TF-IDF  Unnormalized similarity matrix<jupyter_code>recommender_dot = ItemKNNCBFRecommender(URM_train, ICM_all)
recommender_dot.W_sparse = ICM_all * ICM_all.T

result_dict, _ = evaluator_test.evaluateRecommender(recommender_dot)
result_dict[10]<jupyter_output>ItemKNNCBFRecommender: URM Detected 1689 (2.36 %) cold users.
ItemKNNCBFRecommender: URM Detected 56 (0.52 %) cold items.
EvaluatorHoldout: Processed 31000 ( 44.41% ) in 30.59 sec. Users per second: 1014
EvaluatorHoldout: Processed 61000 ( 87.39% ) in 1.03 min. Users per second: 991
EvaluatorHoldout: Processed 69804 ( 100.00% ) in 1.19 min. Users per second: 978
{'fraction_non_alphanumeric': 0.09060768012668251, 'fraction_numerical': 0.030532462391132224, 'mean_word_length': 4.914252268071408, 'pattern_counts': {'":': 2, '<': 106, '<?xml version=': 0, '>': 107, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '414557', 'license': 'MIT', 'n_tokens_mistral': 7504, 'n_tokens_neox': 6895, 'n_words': 1658, 'path': 'Practice 03 - Content Based Filtering.ipynb', 'repo_name': 'SamanFekri/BookRecommendation', 'size': '20271'}