<jupyter_start><jupyter_text>Collaborative Filtering Algorithm Movie Recommemder System using Collaborative Filtering Algorithm This is an implementation of Collaborative Filtering Algorithm from scratch, based on the lecture of <NAME> on the corresponding topic in Coursera. Dataset source: https://www.kaggle.com/grouplens/movielens-20m-dataset<jupyter_code># Import libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
import os
from textwrap import wrap

# Set default fontsize and colors for graphs
SMALL_SIZE, MEDIUM_SIZE, BIG_SIZE = 8, 12, 16
plt.rc('font', size=MEDIUM_SIZE)       
plt.rc('axes', titlesize=BIG_SIZE)     
plt.rc('axes', labelsize=MEDIUM_SIZE)  
plt.rc('xtick', labelsize=MEDIUM_SIZE) 
plt.rc('ytick', labelsize=MEDIUM_SIZE) 
plt.rc('legend', fontsize=SMALL_SIZE)  
plt.rc('figure', titlesize=BIG_SIZE)
my_colors = 'rgbkymc'

# Disable scrolling for long output
from IPython.display import display, Javascript
disable_js = """
IPython.OutputArea.prototype._should_scroll = function(lines) {
    return false;
}
"""
display(Javascript(disable_js))<jupyter_output><empty_output><jupyter_text>(1) Read and Prepare Data  Read "movie" and "rating" dataset<jupyter_code># Read the input training data
input_data_file_movie = "C:\Study\DataSets\movielens-20m-dataset\\movie.csv"
input_data_file_rating = "C:\Study\DataSets\movielens-20m-dataset\\rating.csv"

movie_data_all = pd.read_csv(input_data_file_movie)
rating_data_all = pd.read_csv(input_data_file_rating)
movie_data_all.head(5)
rating_data_all.head(5)
print("Total number of movies =", movie_data_all.shape[0])
print("Total number of unique movies =", len(movie_data_all.movieId.unique()))
print("")
print("Total number of user ratings =", rating_data_all.shape[0])
print("Total number of unique users =", len(rating_data_all.userId.unique()))
# Keep only required columns
movie_data_all = movie_data_all.drop(['genres'], axis=1)
rating_data_all = rating_data_all.drop(['timestamp'], axis=1)
# Test with a subset of data

# Fetch num_movies number of movies
#num_movies = 10
#movie_data = movie_data_all.iloc[:num_movies, :]

# Fetch only ratings corresponding to above movies 
#rating_data = rating_data_all[rating_data_all.movieId.isin(movie_data.movieId)]

#print("Total number of movies to be used to training =", movie_data.shape[0])
#print("Total number of user ratings to be used for training =", rating_data.shape[0])
# Test with a subset of data
#num_movies = 500
#num_ratings = 10000
#movie_data = movie_data.iloc[:num_movies, :]
#rating_data = rating_data.iloc[:num_ratings, :]<jupyter_output><empty_output><jupyter_text>Select few most popular movies, from two distinct genres. In this particular example, we considered movies of genres "Action" and "Romance". The objective is to find if collborative filtering algorithm can successfully learn the features of these movies based on user ratings, such that we can clearly distinguish their genres and recommend accordingly.<jupyter_code># Pick top movies
top_action_movies = ['Dark Knight, The', 'Lord of the Rings: The Return of the King', 
                     'Inception', 'Star Wars: Episode V - The Empire Strikes Back',
                     'Matrix, The']
top_romantic_movies = ['Notting Hill', 'Love Story \(1970\)', 'When Harry Met Sally',
                       'Titanic \(1997\)', 'Pretty Woman']
top_movies = top_action_movies + top_romantic_movies

movie_data = movie_data_all[movie_data_all.title.str.contains('|'.join(top_movies))]
movie_data
# Pick all ratings
#num_ratings = 2000000
rating_data = rating_data_all.iloc[:, :]<jupyter_output><empty_output><jupyter_text>Merge movie and rating dataset based on movieId column<jupyter_code>movie_rating_merged_data = movie_data.merge(rating_data, on='movieId', how='inner')
movie_rating_merged_data.head()
# Mean rating of a movie
movie_rating_merged_data[movie_rating_merged_data.title == 'Pretty Woman (1990)']['rating'].mean()
# Top 10 movies by mean rating
movie_rating_merged_data.groupby(['title'], sort=False)['rating'].mean().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>(2) Build Collaborative Filtering Model  Create a pivot table of movies (on rows) and corresponsing user ratings (on columns). The pivot table will contain the ratings of only selected movies. Thus, rows = movies and columns = users<jupyter_code>movie_rating_merged_pivot = pd.pivot_table(movie_rating_merged_data,
                                           index=['title'],
                                           columns=['userId'],
                                           values=['rating'],
                                           dropna=False,
                                           fill_value=0
                                          )
movie_rating_merged_pivot.shape
Y = movie_rating_merged_pivot<jupyter_output><empty_output><jupyter_text>Create a matrix R, such that, R(i,j) = 1 iff User j has selected a rating for Movie i. R(i,j) = 0 otherwise.<jupyter_code>R = np.ones(Y.shape)
no_rating_idx = np.where(Y == 0.0)
R[no_rating_idx] = 0
R<jupyter_output><empty_output><jupyter_text>Assign n_m (number of movies), n_u (number of users) and n_f (number of features)<jupyter_code>n_u = Y.shape[1]
n_m = Y.shape[0]
n_f = 2<jupyter_output><empty_output><jupyter_text>Assign random initial values to movie and user parameters. X = parameters of movies (each row represent a movie) Theta = parameters of users (each row represent a user)<jupyter_code>Initial_X = np.random.rand(n_m, n_f)
Initial_Theta = np.random.rand(n_u, n_f)
#print("Initial_X =", Initial_X)
#print("Initial_Theta =", Initial_Theta)<jupyter_output><empty_output><jupyter_text>Cost function or Objective function of collborative filtering algorithm<jupyter_code># Cost Function
def collabFilterCostFunction(X, Theta, Y, R, reg_lambda):
    cost = 0
    error = (np.dot(X, Theta.T) - Y) * R
    error_sq = np.power(error, 2)
    cost = np.sum(np.sum(error_sq)) / 2
    cost = cost + (reg_lambda/2) * ( np.sum(np.sum((np.power(X, 2)))) 
                                    + np.sum(np.sum((np.power(Theta, 2)))) )
    return cost<jupyter_output><empty_output><jupyter_text>Computation of Gradient Descent of collaborative filtering algorithm<jupyter_code># Gradient Descent
def collabFilterGradientDescent(X, Theta, Y, R, alpha, reg_lambda, num_iters):
    cost_history = np.zeros([num_iters, 1])
    
    for i in range(num_iters):
        error = (np.dot(X, Theta.T) - Y) * R
        X_grad = np.dot(error, Theta) + reg_lambda * X
        Theta_grad = np.dot(error.T, X) + reg_lambda * Theta
        
        X = X - alpha * X_grad 
        Theta = Theta - alpha * Theta_grad
        
        cost_history[i] = collabFilterCostFunction(X, Theta, Y, R, reg_lambda)
        
    return X, Theta, cost_history<jupyter_output><empty_output><jupyter_text>(3) Train the collborative filtering model<jupyter_code># Tune hyperparameters
alpha = 0.0001
num_iters = 25000
reg_lambda = 0

# Perform gradient descent to find optimal parameters
X, Theta = Initial_X, Initial_Theta
X, Theta, cost_history = collabFilterGradientDescent(X, Theta, Y, R, alpha, reg_lambda, num_iters)
cost = collabFilterCostFunction(X, Theta, Y, R, reg_lambda)
print("Find cost =", cost)<jupyter_output>Find cost = 29810.756341445544<jupyter_text>Plot cost vs number of iterations<jupyter_code>fig, axes = plt.subplots(figsize=(15,6))
axes.plot(cost_history, 'k--')
axes.set_xlabel('# of iterations')
axes.set_ylabel('Cost')
plt.show()<jupyter_output><empty_output><jupyter_text>Since we have considered only 2 genres (and hence 2 features), we plot the learned feature parameters of movies to visualize the pattern. We find below that the algorithm has learnt the features pretty well and hence the movies of same genre and clustered together.  In this particular example, we considered movies of genres "Action" and "Romance". From the visualization, it can be concluded that X-axis represents "Degree of Action" and Y-axis represents "Degree of Romance". As a next step, we can run K-Means clustering to further verify our understanding.<jupyter_code>fig, axes = plt.subplots(figsize=(10,10))
axes.scatter(X[:,0], X[:,1], color='red', marker='D')

for val, movie in zip(X, Y.index):
    axes.text(val[0], val[1], movie)

axes.set_xlabel('Degree of Action')
axes.set_ylabel('Degree of Romance')
axes.set_title('Movie Matrix')

plt.show()<jupyter_output><empty_output><jupyter_text>For a random user, what are her preferred movies, and what is our recommendation for her based on result of collaborative filtering algorithm?<jupyter_code>user_idx = np.random.randint(n_u)
pred_rating = []
print("Original rating of an user:\n", Y.iloc[:,user_idx].sort_values(ascending=False))

predicted_ratings = np.dot(X, Theta.T)
predicted_ratings = sorted(zip(predicted_ratings[:,user_idx], Y.index), reverse=True)
print("\nPredicted rating of the same user:")
_ = [print(rating, movie) for rating, movie in predicted_ratings]<jupyter_output>Original rating of an user:
 title
Titanic (1997)                                           5.0
Pretty Woman (1990)                                      4.5
Dark Knight, The (2008)                                  4.5
Matrix, The (1999)                                       4.0
When Harry Met Sally... (1989)                           0.0
Star Wars: Episode V - The Empire Strikes Back (1980)    0.0
Notting Hill (1999)                                      0.0
Love Story (1970)                                        0.0
Lord of the Rings: The Return of the King, The (2003)    0.0
Inception (2010)                                         0.0
Name: (rating, 75212), dtype: float64

Predicted rating of the same user:
4.949419178469805 Titanic (1997)
4.648944602401086 When Harry Met Sally... (1989)
4.6011962815762955 Pretty Woman (1990)
4.466691303353013 Inception (2010)
4.428913739067579 Notting Hill (1999)
4.319574539395297 Dark Knight, The (2008)
4.182461804852902 Love Story (1970)
4.1411611[...]
{'fraction_non_alphanumeric': 0.1046235921754594, 'fraction_numerical': 0.032404663110057304, 'mean_word_length': 4.003954522985665, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '21215', 'license': 'MIT', 'n_tokens_mistral': 3499, 'n_tokens_neox': 3182, 'n_words': 905, 'path': 'machine-learning/collaborative-filtering.ipynb', 'repo_name': 'SupratimH/learning-data-science', 'size': '10125'}