<jupyter_start><jupyter_text>MovieLens_Recommender_Engine Program based upon original work by <NAME>, UCL.MovieLens dataset from <NAME> and <NAME>. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI=http://dx.doi.org/10.1145/2827872<jupyter_code># Import Common Libraries

import numpy as np
import pandas as pd
# Import dataset with pre-defined header

header = ['user_id', 'item_id', 'rating', 'timestamp']
dataset = pd.read_csv('ml-100K/u.data', sep='\t', names=header)
dataset.head()
# Calculate number of unique users, number of unique movies

n_users = dataset.user_id.unique().shape[0]
n_items = dataset.item_id.unique().shape[0]
print ('Number of users = ' + str(n_users) + ' | Number of movies = ' + str(n_items))
# Cross validation and split the dataset into training set and test set

from sklearn import cross_validation as cv
train_data, test_data = cv.train_test_split(dataset, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Memory-Based Collaborative Filtering approaches can be divided into two main sections: user-item filtering and item-item filtering. A user-item filtering takes a particular user, find users that are similar to that user based on similarity of ratings, and recommend items that those similar users liked. In contrast, item-item filtering will take an item, find users who liked that item, and find other items that those users or similar users also liked. It takes items and outputs other items as recommendations.Item-Item Collaborative Filtering: “Users who liked this item also liked …”User-Item Collaborative Filtering: “Users who are similar to you also liked …”<jupyter_code>#Create two user-item matrices, one for training and another for testing

train_data_matrix = np.zeros((n_users, n_items))
for line in train_data.itertuples():
    train_data_matrix[line[1]-1, line[2]-1] = line[3]

test_data_matrix = np.zeros((n_users, n_items))
for line in test_data.itertuples():
    test_data_matrix[line[1]-1, line[2]-1] = line[3]
# Calculate similarity (cosine) between users and items
# The resulting similarity ranges from 1 meaning exactly the same, with 0 indicating orthogonality (decorrelation), and in-between values indicating intermediate similarity or dissimilarity.

from sklearn.metrics.pairwise import pairwise_distances
user_similarity = pairwise_distances(train_data_matrix, metric='cosine')
item_similarity = pairwise_distances(train_data_matrix.T, metric='cosine')
# Similarity prediction function

def predict(ratings, similarity, type='user'):
    if type == 'user':
        mean_user_rating = ratings.mean(axis=1)
        #You use np.newaxis so that mean_user_rating has same format as ratings
        ratings_diff = (ratings - mean_user_rating[:, np.newaxis])
        pred = mean_user_rating[:, np.newaxis] + similarity.dot(ratings_diff) / np.array([np.abs(similarity).sum(axis=1)]).T
    elif type == 'item':
        pred = ratings.dot(similarity) / np.array([np.abs(similarity).sum(axis=1)])
    return pred
# Item and user similarity prediction 

item_prediction = predict(train_data_matrix, item_similarity, type='item')
user_prediction = predict(train_data_matrix, user_similarity, type='user')
# Root-mean-square error function (RMSE)

from sklearn.metrics import mean_squared_error
from math import sqrt

def rmse(prediction, ground_truth):
    prediction = prediction[ground_truth.nonzero()].flatten()
    ground_truth = ground_truth[ground_truth.nonzero()].flatten()
    return sqrt(mean_squared_error(prediction, ground_truth))
# Compute RMSE for in-memory based collaborative filtering
print ('User-based CF RMSE: ' + str(rmse(user_prediction, test_data_matrix)))
print ('Item-based CF RMSE: ' + str(rmse(item_prediction, test_data_matrix)))
# Compute sparsity ()

sparsity=round(1.0-len(dataset)/float(n_users*n_items),3)
print ('The sparsity level of MovieLens100K is ' +  str(sparsity*100) + '%')
# Model based Collaborative Filtering based upon SVD - Singular Value Decomposition

import scipy.sparse as sp
from scipy.sparse.linalg import svds

#get SVD components from train matrix. Choose k.
u, s, vt = svds(train_data_matrix, k = 20)
s_diag_matrix=np.diag(s)
X_pred = np.dot(np.dot(u, s_diag_matrix), vt)
print ('User-based CF RMSE: ' + str(rmse(X_pred, test_data_matrix)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10142630744849446, 'fraction_numerical': 0.013810278469549468, 'mean_word_length': 5.653614457831325, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '19565', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1442, 'n_tokens_neox': 1348, 'n_words': 425, 'path': 'MovieLens_Recommender_Engine.ipynb', 'repo_name': 'timothysharko/Temporary_Repository', 'size': '4452'}