<jupyter_start><jupyter_text>Movie RecommendationsIn this case study we will build multiple model for making recommendations, validate and compare them. Identification InformationName: <NAME>MIT xPro Username: aperusseMIT xPro Email: <EMAIL> Data ExporationTo begin we import the data and perform some cursory data exploration.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

from surprise import Dataset, SVD, NormalPredictor, BaselineOnly, KNNBasic, NMF
from surprise.model_selection import cross_validate, KFold
ratingsdata = Dataset.load_builtin('ml-100k')
col_names = ['user_id', 'item_id', 'rating', 'timestamp']
rawdata = pd.read_table(ratingsdata.ratings_file, names=col_names)
rawdata.describe()
sns.distplot(rawdata['rating'], bins=5, kde=False)
plt.title("Rating Distribution")
plt.show()<jupyter_output><empty_output><jupyter_text>Question 1: Data Analysis**Describe the dataset. How many ratings are in the dataset? How would you describe the distribution of ratings? Is there anything else we should observe? Make sure the histogram is visible in the notebook.**In this dataset, we are given 100000 User/Item/Rating pairings, from which we hope to make recommendations.  Ratings are given on a scale from 1 to 5, where 5 is the most positive rating. As you might expect, ratings are distributed such that a 4 is the most likely response followed by a 3 or a 5.  1's or 2's are rarer and thus probably carry a little more information than a 4.   Baseline Model - RandomAs our baseline we will use `NormalPredictor`, which simply predicts a random rating based on the distribution of the training set, which is assumed to be normal.<jupyter_code>model_baseline = NormalPredictor()
results_baseline = cross_validate(model_baseline, ratingsdata, measures=['RMSE'], cv=100, verbose=False)
m, s = results_baseline['test_rmse'].mean(), 2*results_baseline['test_rmse'].std()
print(f"RMSE Over 5 Folds: {m} +/- {s}")<jupyter_output>RMSE Over 5 Folds: 1.5252159899507298 +/- 0.05840238755785964<jupyter_text>User-Based Collaborative FilteringUser-based Collaborative Filtering works on the premise that people who rated things similarly in the past will do so in the future.  It makes predictions by averaging the ratings that similar users would have given the item.<jupyter_code>model_user = KNNBasic(sim_options={'user_based': True}, verbose=False)
results_user = cross_validate(model_user, ratingsdata, measures=['RMSE'], cv=100, verbose=False)
m, s = results_user['test_rmse'].mean(), 2*results_user['test_rmse'].std()
print(f"RMSE Over 100 Folds: {m} +/- {s}")<jupyter_output>RMSE Over 100 Folds: 0.9670531814311453 +/- 0.050096632817287025<jupyter_text>Item-Based Collaborative FilteringSimilarly, item-based collaborative filtering makes predictions about items using items which have receieved similar ratings.<jupyter_code>model_item = KNNBasic(sim_options={'user_based': False}, verbose=False)
results_item = cross_validate(model_item, ratingsdata, measures=['RMSE'], cv=100, verbose=False)
m, s = results_item['test_rmse'].mean(), 2*results_item['test_rmse'].std()
print(f"RMSE Over 100 Folds: {m} +/- {s}")<jupyter_output>RMSE Over 100 Folds: 0.9607477776787791 +/- 0.0458838205226042<jupyter_text>Question 2: Collaborative Filtering Models**Compare the results from the user-user and item-item models. How do they compare to each other? How do they compare to our original "random" model? Can you provide any intuition as to why the results came out the way they did?**Atleast in our run, the results weren't that different with user-based and item-based collaborative filtering with item-based only barely having an advantage, $0.961 < 0.966$. However, both were significantly better than the random model whose RMSE was considerably higher, $0.966 < 1.519$, than either of the collaborative filtering based approahces.  This makes sense of course, because the random results are... random. It gives a nice sense of what would happen if we simply rolled a die to make recommendations as opposed to using the data we have. Model 4: Matrix Factorization<jupyter_code>model_svd = SVD()
folds = 50
results_item = cross_validate(model_svd, ratingsdata, measures=['RMSE'], cv=folds, verbose=False)
m, s = results_item['test_rmse'].mean(), 2*results_item['test_rmse'].std()
print(f"RMSE Over {folds} Folds: {m} +/- {s}")<jupyter_output>RMSE Over 50 Folds: 0.9242514125226332 +/- 0.031770023605667906<jupyter_text>Question 3: Matrix Factorization Model**The matrix factorization model is different from the collaborative filtering models. Briefly describe this difference. Also, compare the RMSE again. Does it improve? Can you offer any reasoning as to why that might be?**Where Collaborative filtering takes advantage of nearest neighbor to try to make recomendations to user based on similar users or items based on similar items, matrix factorization is useful when you believe there are latent factors, that is features not directly represented in the data, that are truly the levers in someones preferences.  For example, perhaps there are certain actors or actressses who appears in movies which cause them to be popular.  Matrix Factorization allows one to model this possiblity and try to derive that matrix of latent factors.  We in fact did see a improvement in performance by using matrix factorication by a small margin, $0.924 < 0.961$. Taking Model Criticism Further: Precision and RecallGoing beyond the RMSE, we can evaluate the precision and recall of our recommender as well in a similar fachion to how one does in a classification problem.<jupyter_code>def precision_recall_at_k(predictions, k=10, threshold=3.5):
    '''Return precision and recall at k metrics for each user.'''

    # First map the predictions to each user.
    user_est_true = dict()
    for uid, _, true_r, est, _ in predictions:
        current = user_est_true.get(uid, list())
        current.append((est, true_r))
        user_est_true[uid] = current

    precisions = dict()
    recalls = dict()
    for uid, user_ratings in user_est_true.items():

        # Sort user ratings by estimated value
        user_ratings.sort(key=lambda x: x[0], reverse=True)

        # Number of relevant items
        n_rel = sum((true_r >= threshold) for (_, true_r) in user_ratings)

        # Number of recommended items in top k
        n_rec_k = sum((est >= threshold) for (est, _) in user_ratings[:k])

        # Number of relevant and recommended items in top k
        n_rel_and_rec_k = sum(((true_r >= threshold) and (est >= threshold))
                              for (est, true_r) in user_ratings[:k])

        # Precision@K: Proportion of recommended items that are relevant
        precisions[uid] = n_rel_and_rec_k / n_rec_k if n_rec_k != 0 else 1

        # Recall@K: Proportion of relevant items that are recommended
        recalls[uid] = n_rel_and_rec_k / n_rel if n_rel != 0 else 1

    return precisions, recalls
algos = {'baseline': model_baseline, 'user': model_user, 'item': model_item, 'SVD': model_svd}

for name, algo in algos.items():
    print(name)
    p_arr = []
    r_arr = []
    for folds in [5, 10]:
        print(f"Over {folds}-folds")
        kf = KFold(n_splits=5)
        fold = 0
        for trainset, testset in kf.split(ratingsdata):
            algo.fit(trainset)
            predictions = algo.test(testset)
            precisions, recalls = precision_recall_at_k(predictions, k=5, threshold=4)
            fold += 1

            # Precision and recall can then be averaged over all users
            #print(f"Fold {fold}")
            p = sum(prec for prec in precisions.values()) / len(precisions)
            r = sum(rec for rec in recalls.values()) / len(recalls)
            p_arr.append(p)
            r_arr.append(r)
        print(f"Average Precision: {np.mean(p_arr)}")
        print(f"Average Recall: {np.mean(r_arr)}")<jupyter_output>baseline
Over 5-folds
Average Precision: 0.6012145022718249
Average Recall: 0.2707458905346869
Over 10-folds
Average Precision: 0.602457203177272
Average Recall: 0.2709060915008587
user
Over 5-folds
Average Precision: 0.8443762436192029
Average Recall: 0.27138036262948206
Over 10-folds
Average Precision: 0.8457641624272216
Average Recall: 0.27219540289037625
item
Over 5-folds
Average Precision: 0.9425407266620075
Average Recall: 0.17057555577493896
Over 10-folds
Average Precision: 0.9428893250490159
Average Recall: 0.17027643050263264
SVD
Over 5-folds
Average Precision: 0.8733167399522237
Average Recall: 0.26416978079592945
Over 10-folds
Average Precision: 0.8723540686805373
Average Recall: 0.2611656280625076<jupyter_text>Question 4: Precision/Recall**Compute the precision and recall, for each of the 4 models, at k = 5 and 10. This is 2 x 2 x 4 = 16 numerical values. Do you note anything interesting about these values? Anything differerent from the RMSE values you computed above?**As you would expect, the random model performs the worst in terms of precision of all of the models, although not worse in Recall.  That brings us to the user and item collaborative filters.  Here we see a class tradeoff in precision and recall.  Item-based collaborative filtering gets the higher precision but at the cost of having the worst recall.  This is likely because the item-based model returns a lot of negative recommendations and few positives for any user.  As a result, there are fewer false positives because there are also fewer positives identified by the model at all.  You might interpret this as more conservative behavior, only recommending what you like.The SVD plays the balance the best, with a higher precision than user-based and with nearly the same recall. From our analysis thus far, I would continue to conlude that matrix factorization seems to be the best model that we've tried.<jupyter_code>def get_top_n(predictions, n=5):
    '''Return the top-N recommendation for each user from a set of predictions.

    Args:
        predictions(list of Prediction objects): The list of predictions, as
            returned by the test method of an algorithm.
        n(int): The number of recommendation to output for each user. Default
            is 10.

    Returns:
    A dict where keys are user (raw) ids and values are lists of tuples:
        [(raw item id, rating estimation), ...] of size n.
    '''

    # First map the predictions to each user.
    top_n = dict()
    for uid, iid, true_r, est, _ in predictions:
        current = top_n.get(uid, [])
        current.append((iid, est))
        top_n[uid] = current

    # Then sort the predictions for each user and retrieve the k highest ones.
    for uid, user_ratings in top_n.items():
        user_ratings.sort(key=lambda x: x[1], reverse=True)
        top_n[uid] = user_ratings[:n]

    return top_n
trainset = ratingsdata.build_full_trainset()
model_baseline.fit(trainset)
predictions = model_baseline.test(trainset.build_anti_testset())
topn = get_top_n(predictions, n=5)
# Predictions for user `1`
topn['1']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07985923118570655, 'fraction_numerical': 0.048817902905612706, 'mean_word_length': 4.356694055099082, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '599446', 'license': 'MIT', 'n_tokens_mistral': 3659, 'n_tokens_neox': 3179, 'n_words': 1283, 'path': '4.1 Movie Recommendations.ipynb', 'repo_name': 'mrklees/Data-Science-And-Big-Data-Analytics', 'size': '11102'}