<jupyter_start><jupyter_text>Recommendation System Notebook- User based recommendation- User based prediction & evaluation- Item based recommendation- Item based prediction & evaluation Different Approaches to develop Recommendation System -1. Demographich based Recommendation System2. Content Based Recommendation System3. Collaborative filtering Recommendation System<jupyter_code># import libraties
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Reading ratings file from GitHub. # MovieLens
ratings = pd.read_csv('sample30.csv' , encoding='latin-1')
ratings.head()
ratings = ratings[["reviews_username","name","reviews_rating"]]<jupyter_output><empty_output><jupyter_text>Dividing the dataset into train and test<jupyter_code># Test and Train split of the dataset.
from sklearn.model_selection import train_test_split
train, test = train_test_split(ratings, test_size=0.30, random_state=31)
print(train.shape)
print(test.shape)
train
train = train.groupby(by=["reviews_username",'name']).mean()
train
train['reviews_rating']
train.reset_index(level=0, inplace=True)

train.reset_index(level=0, inplace=True)
train
# Pivot the train ratings' dataset into matrix format in which columns are product name and the rows are username.
df_pivot = train.pivot(
    index='reviews_username',
    columns='name',
    values='reviews_rating'
).fillna(0)

df_pivot.head(3)
df_pivot['Chex Muddy Buddies Brownie Supreme Snack Mix']<jupyter_output><empty_output><jupyter_text>Creating dummy train & dummy test datasetThese dataset will be used for prediction - Dummy train will be used later for prediction of the product which has not been rated by the user. To ignore the product rated by the user, we will mark it as 0 during prediction. The product not rated by user is marked as 1 for prediction in dummy train dataset. - Dummy test will be used for evaluation. To evaluate, we will only make prediction on the product rated by the user. So, this is marked as 1. This is just opposite of dummy_train.<jupyter_code># Copy the train dataset into dummy_train
dummy_train = train.copy()
# The products not rated by user is marked as 1 for prediction. 
dummy_train['reviews_rating'] = dummy_train['reviews_rating'].apply(lambda x: 0 if x>=1 else 1)
# Convert the dummy train dataset into matrix format.
dummy_train = dummy_train.pivot(
    index='reviews_username',
    columns='name',
    values='reviews_rating'
).fillna(1)
dummy_train.head()<jupyter_output><empty_output><jupyter_text>**Cosine Similarity**Cosine Similarity is a measurement that quantifies the similarity between two vectors [Which is Rating Vector in this case] **Adjusted Cosine**Adjusted cosine similarity is a modified version of vector-based similarity where we incorporate the fact that different users have different ratings schemes. In other words, some users might rate items highly in general, and others might give items lower ratings as a preference. To handle this nature from rating given by user , we subtract average ratings for each user from each user's rating for different movies.  User Similarity Matrix  Using Cosine Similarity<jupyter_code>from sklearn.metrics.pairwise import pairwise_distances

# Creating the User Similarity Matrix using pairwise_distance function.
user_correlation = 1 - pairwise_distances(df_pivot, metric='cosine')
user_correlation[np.isnan(user_correlation)] = 0
print(user_correlation)
user_correlation.shape<jupyter_output><empty_output><jupyter_text>Using adjusted Cosine   Here, we are not removing the NaN values and calculating the mean only for the products rated by the user<jupyter_code># Create a user-movie matrix.
df_pivot = train.pivot(
    index='reviews_username',
    columns='name',
    values='reviews_rating'
)
df_pivot.head()<jupyter_output><empty_output><jupyter_text>Normalising the rating of the product for each user around 0 mean<jupyter_code>mean = np.nanmean(df_pivot, axis=1)
df_subtracted = (df_pivot.T-mean).T
df_subtracted.head()<jupyter_output><empty_output><jupyter_text>Finding cosine similarity<jupyter_code>from sklearn.metrics.pairwise import pairwise_distances
# Creating the User Similarity Matrix using pairwise_distance function.
user_correlation = 1 - pairwise_distances(df_subtracted.fillna(0), metric='cosine')
user_correlation[np.isnan(user_correlation)] = 0
print(user_correlation)<jupyter_output>[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]<jupyter_text>Prediction - User User Doing the prediction for the users which are positively related with other users, and not the users which are negatively related as we are interested in the users which are more similar to the current users. So, ignoring the correlation for values less than 0.<jupyter_code>user_correlation[user_correlation<0]=0
user_correlation<jupyter_output><empty_output><jupyter_text>Rating predicted by the user (for product rated as well as not rated) is the weighted sum of correlation with the product rating (as present in the rating dataset).<jupyter_code>user_predicted_ratings = np.dot(user_correlation, df_pivot.fillna(0))
user_predicted_ratings
user_predicted_ratings.shape<jupyter_output><empty_output><jupyter_text>Since we are interested only in the products not rated by the user, we will ignore the products rated by the user by making it zero.<jupyter_code>user_final_rating = np.multiply(user_predicted_ratings,dummy_train)
user_final_rating.head()
for i in user_final_rating["100:Complete First Season (blu-Ray)"]:
  if i > 0:
    print(i)<jupyter_output>10.177320760990225
11.850259125235349
3.1499166132335263
4.702847075210474
0.13572531765245
1.449821866091257
13.254930542653748
3.061862178478973
2.206474848722799
1.5649528464044595
4.211094289736992
0.9241242301415896
3.75
4.979153908129584
11.92060168682202
2.0864972611714134
5.354139110507989
2.2064748487227988
12.113174883553366
10.177320760990225
1.4548516392823951
0.42920113993159914
5.520620726159658
1.4548516392823951
3.860864048875153
0.150717690849421
1.9358541225631414
10.177320760990225
1.8772769422910538
0.6554709342475236
4.514153164245808
5.191158065552841
5.1754945283363
3.1847181262706004
12.683592458568668
4.702847075210474
3.1499166132335263
1.9358541225631414
10.177320760990225
5.520620726159658
0.42920113993159914
8.721060460284406
2.066404598251252
3.87728290708791
10.830127018922191
10.177320760990225
3.6410688066865524
12.464621406529922
2.7669929526473327
7.963495319798332
1.9358541225631414
0.17522063164221668
1.0106073612822533
6.211778791712499
10.17732076[...]<jupyter_text>Finding the top 5 recommendation for the *user*<jupyter_code># Take the user ID as input.
user_input = (input("Enter your user name"))
print(user_input)
user_final_rating.head(2)
d = user_final_rating.loc[user_input].sort_values(ascending=False)#[0:5]
d
#Mapping with Movie Title / Genres 
#movie_mapping = pd.read_csv('https://raw.githubusercontent.com/antrikshsaxena/NLPCapstone/main/movies.csv')
#movie_mapping.head()
#d = pd.merge(d,movie_mapping,left_on='movieId',right_on='movieId', how = 'left')
#d.head()<jupyter_output><empty_output><jupyter_text>Evaluation - User User  Evaluation will we same as you have seen above for the prediction. The only difference being, you will evaluate for the product already rated by the user insead of predicting it for the product not rated by the user.<jupyter_code># Find out the common users of test and train dataset.
common = test[test.reviews_username.isin(train.reviews_username)]
common.shape
common.head()
# convert into the user-product matrix.
common_user_based_matrix = common.pivot_table(index='reviews_username', columns='name', values='reviews_rating')
# Convert the user_correlation matrix into dataframe.
user_correlation_df = pd.DataFrame(user_correlation)
df_subtracted.head(1)

user_correlation_df['reviews_username'] = df_subtracted.index
user_correlation_df.set_index('reviews_username',inplace=True)
user_correlation_df.head()
common.head(1)
list_name = common.reviews_username.tolist()

user_correlation_df.columns = df_subtracted.index.tolist()


user_correlation_df_1 =  user_correlation_df[user_correlation_df.index.isin(list_name)]
user_correlation_df_1.shape
user_correlation_df_2 = user_correlation_df_1.T[user_correlation_df_1.T.index.isin(list_name)]
user_correlation_df_3 = user_correlation_df_2.T
user_correlation_df_3.head()
user_correlation_df_3.shape
user_correlation_df_3[user_correlation_df_3<0]=0

common_user_predicted_ratings = np.dot(user_correlation_df_3, common_user_based_matrix.fillna(0))
common_user_predicted_ratings
dummy_test = common.copy()

dummy_test['reviews_rating'] = dummy_test['reviews_rating'].apply(lambda x: 1 if x>=1 else 0)

dummy_test = dummy_test.pivot_table(index='reviews_username', columns='name', values='reviews_rating').fillna(0)
dummy_test.shape
common_user_predicted_ratings = np.multiply(common_user_predicted_ratings,dummy_test)
common_user_predicted_ratings.head(2)<jupyter_output><empty_output><jupyter_text>Calculating the RMSE for only the products rated by user. For RMSE, normalising the rating to (1,5) range.<jupyter_code>from sklearn.preprocessing import MinMaxScaler
from numpy import *

X  = common_user_predicted_ratings.copy() 
X = X[X>0]

scaler = MinMaxScaler(feature_range=(1, 5))
print(scaler.fit(X))
y = (scaler.transform(X))

print(y)
common_ = common.pivot_table(index='reviews_username', columns='name', values='reviews_rating')
# Finding total non-NaN value
total_non_nan = np.count_nonzero(~np.isnan(y))
rmse = (sum(sum((common_ - y )**2))/total_non_nan)**0.5
print(rmse)<jupyter_output>2.498437231297542<jupyter_text>Using Item similarity  Item Based Similarity Taking the transpose of the rating matrix to normalize the rating around the mean for different product name. In the user based similarity, we had taken mean for each user instead of each *product*.<jupyter_code>df_pivot = train.pivot(
    index='reviews_username',
    columns='name',
    values='reviews_rating'
).T

df_pivot.head()<jupyter_output><empty_output><jupyter_text>Normalising the product rating for each product for using the Adujsted Cosine<jupyter_code>mean = np.nanmean(df_pivot, axis=1)
df_subtracted = (df_pivot.T-mean).T
df_subtracted.head()<jupyter_output><empty_output><jupyter_text>Finding the cosine similarity using pairwise distances approach<jupyter_code>from sklearn.metrics.pairwise import pairwise_distances
# Item Similarity Matrix
item_correlation = 1 - pairwise_distances(df_subtracted.fillna(0), metric='cosine')
item_correlation[np.isnan(item_correlation)] = 0
print(item_correlation)<jupyter_output>[[ 1.          0.          0.         ...  0.          0.
   0.        ]
 [ 0.          1.          0.         ... -0.00657044  0.
   0.        ]
 [ 0.          0.          0.         ...  0.          0.
   0.        ]
 ...
 [ 0.         -0.00657044  0.         ...  1.          0.
   0.        ]
 [ 0.          0.          0.         ...  0.          1.
   0.        ]
 [ 0.          0.          0.         ...  0.          0.
   1.        ]]<jupyter_text>Filtering the correlation only for which the value is greater than 0. (Positively correlated)<jupyter_code>item_correlation[item_correlation<0]=0
item_correlation<jupyter_output><empty_output><jupyter_text>Prediction - Item Item<jupyter_code>item_predicted_ratings = np.dot((df_pivot.fillna(0).T),item_correlation)
item_predicted_ratings
item_predicted_ratings.shape
dummy_train.shape<jupyter_output><empty_output><jupyter_text>Filtering the rating only for the product not rated by the user for recommendation<jupyter_code>item_final_rating = np.multiply(item_predicted_ratings,dummy_train)
item_final_rating.head()<jupyter_output><empty_output><jupyter_text>Finding the top 5 recommendation for the *user*<jupyter_code># Take the user name as input
user_input = (input("Enter your user name"))
print(user_input)
# Recommending the Top 5 products to the user.
d = item_final_rating.loc[user_input].sort_values(ascending=False)#[0:5]
d
#Mapping with Movie Title / Genres 
#movie_mapping = pd.read_csv('https://raw.githubusercontent.com/antrikshsaxena/NLPCapstone/main/movies.csv', encoding='latin-1')
#d = pd.merge(d,movie_mapping,left_on='movieId',right_on='movieId',how = 'left')
#d.head()
#train_new = pd.merge(train,movie_mapping,left_on='movieId',right_on='movieId',how='left')
#train_new[train_new.userId == 1] .head()<jupyter_output><empty_output><jupyter_text>Evaluation - Item Item Evaluation will we same as you have seen above for the prediction. The only difference being, you will evaluate for the product already rated by the user insead of predicting it for the product not rated by the user.<jupyter_code>test.columns
common =  test[test.name.isin(train.name)]
common.shape
common.head(4)
common_item_based_matrix = common.pivot_table(index='reviews_username', columns='name', values='reviews_rating').T
common_item_based_matrix.shape
item_correlation_df = pd.DataFrame(item_correlation)
item_correlation_df.head(1)
item_correlation_df['name'] = df_subtracted.index
item_correlation_df.set_index('name',inplace=True)
item_correlation_df.head()
list_name = common.name.tolist()
item_correlation_df.columns = df_subtracted.index.tolist()

item_correlation_df_1 =  item_correlation_df[item_correlation_df.index.isin(list_name)]
item_correlation_df_2 = item_correlation_df_1.T[item_correlation_df_1.T.index.isin(list_name)]

item_correlation_df_3 = item_correlation_df_2.T
item_correlation_df_3.head()
item_correlation_df_3[item_correlation_df_3<0]=0

common_item_predicted_ratings = np.dot(item_correlation_df_3, common_item_based_matrix.fillna(0))
common_item_predicted_ratings

common_item_predicted_ratings.shape<jupyter_output><empty_output><jupyter_text>Dummy test will be used for evaluation. To evaluate, we will only make prediction on the products rated by the user. So, this is marked as 1. This is just opposite of dummy_train<jupyter_code>dummy_test = common.copy()

dummy_test['reviews_rating'] = dummy_test['reviews_rating'].apply(lambda x: 1 if x>=1 else 0)

dummy_test = dummy_test.pivot_table(index='reviews_username', columns='name', values='reviews_rating').T.fillna(0)

common_item_predicted_ratings = np.multiply(common_item_predicted_ratings,dummy_test)<jupyter_output><empty_output><jupyter_text>The products not rated is marked as 0 for evaluation. And make the item- item matrix representaion.<jupyter_code>common_ = common.pivot_table(index='reviews_username', columns='name', values='reviews_rating').T
from sklearn.preprocessing import MinMaxScaler
from numpy import *

X  = common_item_predicted_ratings.copy() 
X = X[X>0]

scaler = MinMaxScaler(feature_range=(1, 5))
print(scaler.fit(X))
y = (scaler.transform(X))

print(y)
# Finding total non-NaN value
total_non_nan = np.count_nonzero(~np.isnan(y))
rmse = (sum(sum((common_ - y )**2))/total_non_nan)**0.5
print(rmse)<jupyter_output>3.5599247722439755<jupyter_text>Item-Item coloborative approach is better as we are getting the correct result<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0950618100085939, 'fraction_numerical': 0.07437033119587493, 'mean_word_length': 5.990757855822551, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 98, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '271094', 'license': 'MIT', 'n_tokens_mistral': 5773, 'n_tokens_neox': 5050, 'n_words': 1232, 'path': 'Final_Capstone_Recommendation_System_by_Rakesh_Sept_9th.ipynb', 'repo_name': 'rr1989/capstone-project-recommendation-system-by-rr', 'size': '15127'}