<jupyter_start><jupyter_text><jupyter_code>! pip install surprise
! pip install matrix_factorization
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
from surprise import Reader, SVD, Dataset
from surprise.model_selection.validation import cross_validate
from matrix_factorization import KernelMF
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from google.colab import files

uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>EDA and preprocessing

Preprocessing of the movie dataframe has been done in an other notebook. We downloaded here the final dataframe. We will keep rating only for movies that are in the cleaned dataframe. 

The number of ratings per user can vary a lot. However it can be very challenging to predict mivie ratings for users with few ratings. That why we will run our model with different minimum of rating per user and see the impacts on the metrics.

First let do a little EDA on the rating dataframe.<jupyter_code>movies_df = pd.read_csv('cleaned_movies.csv')
ratings_df = pd.read_csv('ratings.csv')
movies_df.head(10)
movies_df.shape
ratings_df.head(10)
ratings_df.shape<jupyter_output><empty_output><jupyter_text>Merge ratings and movie table to get the movie title in the same table<jupyter_code>df=ratings_df.merge(movies_df[['id','original_title']],how='inner', left_on='movieId', right_on='id')
df.head(10)
df.shape<jupyter_output><empty_output><jupyter_text>Remove duplicate<jupyter_code>df = df.drop_duplicates()<jupyter_output><empty_output><jupyter_text>Number of ratings per user distribution<jupyter_code>df['userId'].value_counts().plot(kind='hist')<jupyter_output><empty_output><jupyter_text>Let's zoom on less that 100 ratings per user<jupyter_code>df['userId'].value_counts()[-550:].plot(kind='hist')<jupyter_output><empty_output><jupyter_text>Modeling

We will run two different matrix factorization models/packages with 4 different minimum of rating per user. Then we will chose our best option. Finally, we will hypertune the parameters of this best option.<jupyter_code>df['count']=df['userId'].map(df['userId'].value_counts())
df = df.rename(columns={"userId":"user_id", "movieId":"item_id"})
df_10 = df[df['count']>10]
df_5 = df[df['count']>5]
df_50 = df[df['count']>50]
df_20 = df[df['count']>20]<jupyter_output><empty_output><jupyter_text>Min 10 ratings

 Matrix factorization<jupyter_code>X = df_10[['user_id','item_id']]
y = df_10['rating']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, \
                                                    random_state=1000, \
                                    stratify = df_10['user_id'])
matrix_fact = KernelMF(n_epochs=200, n_factors=100, lr=0.001, reg=0.005)
matrix_fact.fit(X_train, y_train)
# matrix_fact.update_users(X_train_update, y_train_update, lr=0.001, n_epochs=200, verbose=1)
pred = matrix_fact.predict(X_test)
# pred = [round(x) for x in pred]
rmse = mean_squared_error(y_test, pred, squared=False)
r2 = r2_score(y_test, pred)
metrics_mf_10 = (rmse, r2)
metrics_mf_10<jupyter_output><empty_output><jupyter_text>SVD<jupyter_code>df_train =  pd.concat([X_train, y_train], axis=1)
reader = Reader()
data = Dataset.load_from_df(df_train[['user_id', 'item_id', 'rating']], reader)
svd = SVD()
cross_validate(svd, data, measures=['RMSE','MAE'])
trainset = data.build_full_trainset()
svd.fit(trainset)
y_pred = []
for line in range(len(X_test)):
  y_pred.append(svd.predict(X_test.iloc[line,0],X_test.iloc[line,1],3)[3])
rmse = mean_squared_error(y_test, y_pred, squared=False)
r2 = r2_score(y_test, y_pred)
metrics_svd_10 = (rmse, r2)
metrics_svd_10<jupyter_output><empty_output><jupyter_text>Min 5 ratings
 Matrix factorization<jupyter_code>X = df_5[['user_id','item_id']]
y = df_5['rating']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, \
                                                    random_state=1000, \
                                    stratify = df_5['user_id'])
matrix_fact = KernelMF(n_epochs=200, n_factors=100, lr=0.001, reg=0.005)
matrix_fact.fit(X_train, y_train)
pred = matrix_fact.predict(X_test)
# pred = [round(x) for x in pred]
rmse = mean_squared_error(y_test, pred, squared=False)
r2 = r2_score(y_test, pred)
metrics_mf_5 = (rmse, r2)
metrics_mf_5<jupyter_output><empty_output><jupyter_text>SVD<jupyter_code>df_train =  pd.concat([X_train, y_train], axis=1)
reader = Reader()
data = Dataset.load_from_df(df_train[['user_id', 'item_id', 'rating']], reader)
svd = SVD()
cross_validate(svd, data, measures=['RMSE','MAE'])
trainset = data.build_full_trainset()
svd.fit(trainset)
y_pred = []
for line in range(len(X_test)):
  y_pred.append(svd.predict(X_test.iloc[line,0],X_test.iloc[line,1],3)[3])
rmse = mean_squared_error(y_test, y_pred, squared=False)
r2 = r2_score(y_test, y_pred)
metrics_svd_5 = (rmse, r2)
metrics_svd_5<jupyter_output><empty_output><jupyter_text>Min 50 ratings
 Matrix factorization<jupyter_code>X = df_50[['user_id','item_id']]
y = df_50['rating']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, \
                                                    random_state=1000, \
                                    stratify = df_50['user_id'])
matrix_fact = KernelMF(n_epochs=200, n_factors=100, lr=0.001, reg=0.005)
matrix_fact.fit(X_train, y_train)
pred = matrix_fact.predict(X_test)
# pred = [round(x) for x in pred]
rmse = mean_squared_error(y_test, pred, squared=False)
r2 = r2_score(y_test, pred)
metrics_mf_50 = (rmse, r2)
metrics_mf_50<jupyter_output><empty_output><jupyter_text>SVD<jupyter_code>df_train =  pd.concat([X_train, y_train], axis=1)
reader = Reader()
data = Dataset.load_from_df(df_train[['user_id', 'item_id', 'rating']], reader)
svd = SVD()
cross_validate(svd, data, measures=['RMSE','MAE'])
trainset = data.build_full_trainset()
svd.fit(trainset)
y_pred = []
for line in range(len(X_test)):
  y_pred.append(svd.predict(X_test.iloc[line,0],X_test.iloc[line,1],3)[3])
rmse = mean_squared_error(y_test, y_pred, squared=False)
r2 = r2_score(y_test, y_pred)
metrics_svd_50 = (rmse, r2)
metrics_svd_50<jupyter_output><empty_output><jupyter_text>Min 20 ratings
 Matrix factorization<jupyter_code>df_20 = df[df['count']>20]
X = df_20[['user_id','item_id']]
y = df_20['rating']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, \
                                                    random_state=1000, \
                                    stratify = df_20['user_id'])
matrix_fact = KernelMF(n_epochs=200, n_factors=100, lr=0.001, reg=0.005)
matrix_fact.fit(X_train, y_train)
pred = matrix_fact.predict(X_test)
# pred = [round(x) for x in pred]
rmse = mean_squared_error(y_test, pred, squared=False)
r2 = r2_score(y_test, pred)
metrics_mf_20 = (rmse, r2)
metrics_mf_20<jupyter_output><empty_output><jupyter_text>SVD<jupyter_code>df_train =  pd.concat([X_train, y_train], axis=1)
reader = Reader()
data = Dataset.load_from_df(df_train[['user_id', 'item_id', 'rating']], reader)
svd = SVD()
cross_validate(svd, data, measures=['RMSE','MAE'])
trainset = data.build_full_trainset()
svd.fit(trainset)
y_pred = []
for line in range(len(X_test)):
  y_pred.append(svd.predict(X_test.iloc[line,0],X_test.iloc[line,1],3)[3])
rmse = mean_squared_error(y_test, y_pred, squared=False)
r2 = r2_score(y_test, y_pred)
metrics_svd_20 = (rmse, r2)
metrics_svd_20<jupyter_output><empty_output><jupyter_text>Model Selection<jupyter_code>mins = [5,5,10,10,50,50,20,20]
methods = [0,1,0,1,0,1,0,1]
metrics = [metrics_mf_5,metrics_svd_5,metrics_mf_10, metrics_svd_10, metrics_mf_50, metrics_svd_50,metrics_mf_20, metrics_svd_20 ]
rmses = []
r2s = []
for met in metrics:
  rmses.append(met[0])
  r2s.append(met[1])
rmses
plt.scatter(x=rmses, y=r2s, s=mins, c=methods)
plt.xlabel('RMSE')
plt.ylabel('R2')
plt.title('R2 in function of RMSE for different packages and min rating')
plt.colorbar()
plt.show()<jupyter_output><empty_output><jupyter_text>We want to maximize R2 and minimize RMSE. So we want the most left upper corner point!

First, the SVD matrix factorization - yellow points- returns better metrics, we will use this package.

Concerning the minimum of rating per users, limited users with more than 50 ratings is too much, we will lose to many data. Let use the SVD model with minimum number of 20 and try to tune its hyperparameters. 

Let know hypertune the parameters of this model.  Hyperparameters of SVD with 20 min user ratings

We will hypertune the number of factors, epochs and learning rate.<jupyter_code>df_20 = df[df['count']>20]

df_train, df_test = train_test_split(df_20, test_size=0.20, \
                                                    random_state=1000, \
                                    stratify = df_20['user_id'])
factors = [100,200,500]
epochs = [20,50,100]
lr_all = [0.001,0.005,0.01]
reader = Reader()
data = Dataset.load_from_df(df_train[['user_id', 'item_id', 'rating']], reader)
rmse_20 = []
r2_20 = []
params = []
for fac in factors:
  for epoch in epochs:
    for lr in lr_all:
      svd = SVD(n_factors=fac, n_epochs=epoch, lr_all=lr)
      cross_validate(svd, data, measures=['RMSE','MAE'])
      trainset = data.build_full_trainset()
      svd.fit(trainset)
      y_pred = []
      for line in range(len(df_test)):
        y_pred.append(svd.predict(df_test.iloc[line,0],df_test.iloc[line,1],3)[3])
      rmse_20.append(mean_squared_error(df_test['rating'], y_pred, squared=False))
      r2_20.append(r2_score(df_test['rating'], y_pred))
      params.append((fac,epoch,lr))


np.argmin(rmse_20), np.argmax(r2_20), params[np.argmin(rmse_20)]
factors_plt = [100,100,100,100,100,100,100,100,100,200,200,200,200,200,200,200,200,200,500,500,500,500,500,500,500,500,500]
epochs_plt = [20,20,20,50,50,50,100,100,100,20,20,20,50,50,50,100,100,100,20,20,20,50,50,50,100,100,100]
marker_plt = ['o','o','o','o','o','o','o','o','o','+','+','+','+','+','+','+','+','+','s','s','s','s','s','s','s','s','s']
for _m, c, _s, _x, _y in zip(marker_plt, epochs_plt, factors_plt, rmse_20, r2_20):
    plt.scatter(_x, _y, marker=_m, c=c, s=_s)
# plt.scatter(x=rmse_20, y=r2_20, s=factors_plt, c=epochs_plt, marker=marker_plt)
plt.xlabel('RMSE')
plt.ylabel('R2')
plt.title('R2 in function of RMSE for different hyperparameters')
plt.colorbar()
plt.show()<jupyter_output><empty_output><jupyter_text>Again, we want to most left upper point. The best option is the default one: n_factors=500, epochs=100 and learning rate=0.01  Get 10 best reco<jupyter_code>df_20 = df[df['count']>20]

df_train, df_test = train_test_split(df_20, test_size=0.20, \
                                                    random_state=1000, \
                                    stratify = df_20['user_id'])

reader = Reader()
data = Dataset.load_from_df(df_train[['user_id', 'item_id', 'rating']], reader)
svd = SVD(n_factors=500, n_epochs=100, lr_all=0.01)
cross_validate(svd, data, measures=['RMSE','MAE'])
trainset = data.build_full_trainset()
svd.fit(trainset)
def get_10(user_id, top_n=10):
  movies_id = df_20['item_id'].unique().tolist()
  movie_rates = [(id, svd.predict(user_id, id)[3]) for id in movies_id]#
  movie_rates = sorted(movie_rates, key=lambda x:x[1], reverse=True)
  return dict(movie_rates[:top_n])


get_10(1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13902586728955793, 'fraction_numerical': 0.046716844017648584, 'mean_word_length': 4.862068965517241, 'pattern_counts': {'":': 2, '<': 73, '<?xml version=': 0, '>': 80, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '426010', 'license': 'MIT', 'n_tokens_mistral': 4770, 'n_tokens_neox': 4474, 'n_words': 851, 'path': 'Collaborative_system.ipynb', 'repo_name': 'OhadHayoun/ITC_project_2', 'size': '11559'}