<jupyter_start><jupyter_text>Recommender Systems with Surprise- **Created by <NAME>**- **Created on May 23, 2019**- **Updated on Jul 22, 2020**  Experiment description- Model built from a plain text file- The algorithm used is: KNNBasic- Model trained using the technique of cross validation (5 folds)- The RMSE and MAE metrics were used to estimate the model error- Type of filtering: User-based collaborative<jupyter_code># Load the Pandas libraries
import os
import io
import numpy as np
from collections import defaultdict
# Load Surprise libraries
from surprise import KNNBasic
from surprise import Reader
from surprise import Dataset
from surprise import accuracy
from surprise.model_selection import cross_validate
# Load Plotting libraries
%matplotlib inline
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>1. Loading data<jupyter_code># Path to dataset file
file_path = os.path.expanduser('../data/u.data')
# Read the data into a Surprise dataset
reader = Reader(line_format = 'user item rating timestamp', sep = '\t', rating_scale = (1, 5))
data = Dataset.load_from_file(file_path, reader = reader)<jupyter_output><empty_output><jupyter_text>2. Train the model and measure its error A basic collaborative filtering algorithm. The prediction $\hat{r}_{ui}$ is set as: $$ \hat{r}_{ui} = \frac{ \sum_{v \in N_{i}^{k}(u)} {sim(u,v) . r_{vi}} }{ \sum_{v \in N_{i}^{k}(u)} {sim(u,v) } } \tag{1} $$<jupyter_code># Use k-NN algorithm with user-based collaborative filtering and cosine similarity
kk = 50
sim_options = {'name': 'cosine', 'user_based': True}
algo = KNNBasic(k = kk, sim_options = sim_options, verbose = True)
# Run 5-fold cross-validation and print results
cv = cross_validate(algo, data, measures = ['RMSE', 'MAE'], cv = 5, verbose = True)
# Get data
rmse = cv['test_rmse']
mae = cv['test_mae']
x = np.arange(len(rmse))

# Set up the matplotlib figure
fig, ax = plt.subplots(figsize = (10, 5))
plt.xticks(np.arange(min(x), max(x) + 1, 1.0))
plt.ylim(0.5, 1.3)
ax.plot(x, rmse, marker='o', label="rmse")
ax.plot(x, mae, marker='o', label="mae")

# Chart setup
plt.title("Model Errors", fontsize = 12)
plt.xlabel("CV", fontsize = 10)
plt.ylabel("Error", fontsize = 10)
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>3. Make some predictions<jupyter_code># Without real rating
p1 = algo.predict(uid = '13', iid = '181', verbose = True)
# With real rating
p2 = algo.predict(uid = '196', iid = '302', r_ui = 4, verbose = True)<jupyter_output>user: 196        item: 302        r_ui = 4.00   est = 4.30   {'actual_k': 50, 'was_impossible': False}<jupyter_text>4. Get the k nearest neighbors of an item<jupyter_code># Return two mappings to convert raw ids into movie names and movie names into raw ids
def read_item_names(file_path):
    rid_to_name = {}
    name_to_rid = {}
    
    with io.open(file_path, 'r', encoding = 'ISO-8859-1') as f:
        for line in f:
            line = line.split('|')
            rid_to_name[line[0]] = line[1]
            name_to_rid[line[1]] = line[0]
    
    return rid_to_name, name_to_rid
# Read the mappings raw id <-> movie name
item_filepath = '../data/u.item'
rid_to_name, name_to_rid = read_item_names(item_filepath)
# Target movie
target_movie = 'Toy Story (1995)'
# Retrieve inner id of the movie Toy Story
toy_story_raw_id = name_to_rid[target_movie]
toy_story_inner_id = algo.trainset.to_inner_iid(toy_story_raw_id)
print(target_movie + ':', toy_story_inner_id)
# Retrieve inner ids of the nearest neighbors of Toy Story
toy_story_neighbors = algo.get_neighbors(toy_story_inner_id, k = 10)
toy_story_neighbors
# The 10 nearest neighbors of Toy Story are:
print("The movies most similar to '" + target_movie + " are:")

for inner_id in toy_story_neighbors:
    raw_id = algo.trainset.to_raw_iid(inner_id)
    movie = rid_to_name[raw_id]
    print(raw_id, '-', movie)<jupyter_output>The movies most similar to 'Toy Story (1995) are:
526 - Ben-Hur (1959)
596 - Hunchback of Notre Dame, The (1996)
237 - <NAME> (1996)
381 - Muriel's Wedding (1994)
240 - Beavis and Butt-head Do America (1996)
623 - Angels in the Outfield (1994)
429 - Day the Earth Stood Still, The (1951)
33 - Desperado (1995)
409 - Jack (1996)
487 - Roman Holiday (1953)<jupyter_text>5. Get the top-N recommendations<jupyter_code># Return the top-N recommendation for each user from a set of predictions.
def get_top_n(predictions, n = 10):
    
    # First map the predictions to each user.
    top_n = defaultdict(list)
    for uid, iid, true_r, est, _ in predictions:
        top_n[uid].append((iid, est))
        
    # Then sort the predictions for each user and retrieve the k highest ones.
    for uid, user_ratings in top_n.items():
        user_ratings.sort(key=lambda x: x[1], reverse=True)
        top_n[uid] = user_ratings[:n]
    
    return top_n
# Create train_set and test_set
train_set = data.build_full_trainset()
test_set = train_set.build_anti_testset()

# First train a KNN algorithm on the whole dataset
algo.fit(train_set)
predictions = algo.test(test_set)

# RMSE should be low as we are biased
accuracy.rmse(predictions, verbose = True)
# Than predict ratings for all pairs (u, i) that are NOT in the training set
top_n = 10
top_pred = get_top_n(predictions, n = top_n)
# User raw Id
uid_list = ['196']

# Print the recommended items for a specific user
for uid, user_ratings in top_pred.items():
    if uid in uid_list:
        for (iid, rating) in user_ratings:
            movie = rid_to_name[iid]
            print('Movie:', iid, '-', movie, ', rating:', str(rating))<jupyter_output>Movie: 1189 - Prefontaine (1997) , rating: 5
Movie: 1500 - Santa with Muscles (1996) , rating: 5
Movie: 814 - Great Day in Harlem, A (1994) , rating: 5
Movie: 1536 - Aiqing wansui (1994) , rating: 5
Movie: 1293 - Star Kid (1997) , rating: 5
Movie: 1599 - Someone Else's America (1995) , rating: 5
Movie: 1653 - Entertaining Angels: The Dorothy Day Story (1996) , rating: 5
Movie: 1467 - Saint of Fort Washington, The (1993) , rating: 5
Movie: 1122 - They Made Me a Criminal (1939) , rating: 5
Movie: 1201 - <NAME>: Shadow and Light (1996)  , rating: 5
{'fraction_non_alphanumeric': 0.11701434159061277, 'fraction_numerical': 0.041720990873533245, 'mean_word_length': 4.3088235294117645, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '172197', 'license': 'MIT', 'n_tokens_mistral': 2341, 'n_tokens_neox': 2088, 'n_words': 700, 'path': 'code/RS_KNN.ipynb', 'repo_name': 'ansegura7/RS_Surprise', 'size': '6167'}