<jupyter_start><jupyter_text>ObjectiveThe goal is to develop a model that predicts the movies that a user might like based on collaborative filtering.  load data<jupyter_code># load data
ratings = pd.read_csv('input/ratings.csv')
movies = pd.read_csv('input/movies.csv')<jupyter_output><empty_output><jupyter_text>EDA  Ratings dataset<jupyter_code>ratings.shape
ratings.head(5)<jupyter_output><empty_output><jupyter_text>- How many users have we got?- How many movies?- What is the best ranked movie?<jupyter_code>ratings.userId.nunique(), ratings.movieId.nunique()<jupyter_output><empty_output><jupyter_text>`Ratings` dataset contains 610 users and 9724 ranked movies<jupyter_code>plt.figure()
plt.hist(ratings.rating, rwidth=.9, bins=5, alpha=.8, color='skyblue')
plt.title('Distribution of ratings')
plt.xlabel('ratings')
plt.ylabel('count')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's have a look on `movies` dataset<jupyter_code>movies.head(5)
movies.shape
movies.movieId.nunique()<jupyter_output><empty_output><jupyter_text>18 movies are not ranked in ranking dataset, let's find out which movies are they<jupyter_code>movies[~movies.movieId.isin(ratings.movieId)]<jupyter_output><empty_output><jupyter_text>No user has interacted whatsoever with the movies above, that's way we have no records about them in ratings table. The user could have watched the movie but has not submitted any rating.  Top 10 movies<jupyter_code># best ranked movies
best_ranking = ratings.groupby('movieId')[['rating']].mean()
best_ranking['nbr_voters'] = ratings.groupby('movieId')[['userId']].count()
best_ranking = best_ranking.sort_values(by='nbr_voters', ascending=False)[:10]
best_ranking['movie_title'] = movies[movies.movieId.isin(best_ranking.index)].set_index('movieId').title
best_ranking<jupyter_output><empty_output><jupyter_text>Movies per category<jupyter_code>categories_dict = {}
for idx, value in movies.genres.items():
    categories = value.split('|')
    for category in categories:
        categories_dict[category] = categories_dict.get(category, 0) + 1
categories_dict_sorted = {k: v for k, v in sorted(categories_dict.items(), key=lambda item: item[1])}
plt.figure(figsize=(20,6))
ax = plt.axes()
sns.barplot(x=list(categories_dict_sorted.keys()), y=list(categories_dict_sorted.values()), ax=ax)
plt.xticks(rotation=60)
plt.ylabel('nbr of movies')
plt.title('nbr of movies Vs genre')

for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x()+p.get_width()/2.,
            height+20,
            f'{int(height)}',
            ha="center", fontsize=10)

plt.show()<jupyter_output><empty_output><jupyter_text>Recommendation system To build the recommendation system, we will use LightFM model. LightFM is a Python implementation of a number of popular recommendation algorithms for both implicit and explicit feedback.It also makes it possible to incorporate both item and user metadata into the traditional matrix factorization algorithms. It represents each user and item as the sum of the latent representations of their features, thus allowing recommendations to generalise to new items (via item features) and to new users (via user features).You will find the documentation here: https://making.lyst.com/lightfm/docs/home.html<jupyter_code>n_rows = ratings['userId'].unique().shape[0]
n_cols = ratings['movieId'].unique().shape[0]
sparsity = float(ratings.shape[0]) / float(n_rows*n_cols)
print(f'Number of users: {n_rows}')
print(f'Number of movies: {n_cols}')
print(f'Sparsity: {sparsity:.2%}')<jupyter_output>Number of users: 610
Number of movies: 9724
Sparsity: 1.70%<jupyter_text>This means that our interaction matrix is highly sparse since only 1.7% of values are none zero.Let's try to make our interaction matrix less sparse. We have some options for that:- keep users that have interacted with a minimum numbers of movies.- keep movies that have been rated by a minimum number of users.Note: from the EDA we saw that a user can vote only once for a given movie which makes sense :)<jupyter_code>min_nbr_ratings_per_movie = 2
min_nbr_of_rated_movies_by_usr = 4
ratings_filtered = ratings.copy()

n_rows = ratings_filtered['userId'].unique().shape[0]
n_cols = ratings_filtered['movieId'].unique().shape[0]
sparsity = float(ratings_filtered.shape[0]) / float(n_rows*n_cols)

print('Evaluate interaction matrix sparsity before optimization')
print(f'Number of users: {n_rows}')
print(f'Number of movies: {n_cols}')
print(f'Sparsity: {sparsity:.2%}')

# filter movies that do not match our criterion
count_movies_vs_usr = ratings_filtered.groupby(['movieId'])['userId'].count()
movieID_to_drop = count_movies_vs_usr[count_movies_vs_usr<min_nbr_ratings_per_movie].index.to_list()
print(f'{len(movieID_to_drop)} movies have been dropped')
ratings_filtered = ratings_filtered[~ratings_filtered.movieId.isin(movieID_to_drop)]

# filter users that do not match our criterion
count_user_vs_movies = ratings_filtered.groupby(['userId'])['movieId'].count()
userId_to_drop = count_user_vs_movies[count_user_vs_movies<min_nbr_of_rated_movies_by_usr].index.to_list()
print(f'{len(userId_to_drop)} users have been dropped')
ratings_filtered = ratings_filtered[~ratings_filtered.userId.isin(userId_to_drop)]

print('\nEvaluate interaction matrix sparsity after optimization')
n_rows = ratings_filtered['userId'].unique().shape[0]
n_cols = ratings_filtered['movieId'].unique().shape[0]
sparsity = float(ratings_filtered.shape[0]) / float(n_rows*n_cols)
print(f'Number of users: {n_rows}')
print(f'Number of movies: {n_cols}')
print(f'Sparsity: {sparsity:.2%}')<jupyter_output>Evaluate interaction matrix sparsity before optimization
Number of users: 610
Number of movies: 9724
Sparsity: 1.70%
3446 movies have been dropped
0 users have been dropped

Evaluate interaction matrix sparsity after optimization
Number of users: 610
Number of movies: 6278
Sparsity: 2.54%<jupyter_text>Our algorithm has filtered movie that do not have more then 2 interactions with users. As a results the number of movies dropped from 9724 movies to 6278.  Now that we have optimized our matrix sparsity let's convert it from dense to sparse format as LightFM model expects the dataset to be a sparse matrix.<jupyter_code># let's construct our (n_users x n_movies) matrix

# Create mappings
rid_to_idx = {}
idx_to_rid = {}
for (idx, rid) in enumerate(ratings_filtered['userId'].unique().tolist()):
    rid_to_idx[rid] = idx
    idx_to_rid[idx] = rid

cid_to_idx = {}
idx_to_cid = {}
for (idx, cid) in enumerate(ratings_filtered['movieId'].unique().tolist()):
    cid_to_idx[cid] = idx
    idx_to_cid[idx] = cid
import scipy.sparse as sparse

def map_ids(row, mapper):
    return mapper[row]

userId = ratings_filtered['userId'].apply(map_ids, args=[rid_to_idx]).values
movieId = ratings_filtered['movieId'].apply(map_ids, args=[cid_to_idx]).values
#V = np.ones(userId.shape[0])
V = ratings_filtered.rating
interaction_matrix = sparse.coo_matrix((V, (userId, movieId)), dtype=np.float64)
interaction_matrix = interaction_matrix.tocsr()<jupyter_output><empty_output><jupyter_text>This basically tells sparse matrix the coordinates of the non zero values. In our case when ever there is an interaction between a user and a moive we put 1  Interaction matrix<jupyter_code># our interaction matrix looks like this
plt.figure(figsize=(20, 15))
sns.heatmap(interaction_matrix.toarray(), cbar=True)
plt.show()<jupyter_output><empty_output><jupyter_text>As you see most of our interaction matrix is composed of zeros. Only 2.54% of its elements are none zeros values. That's why it is import to convert to sparse format using scipy as we did to make it more memory friendly and efficient in terms of calculations.<jupyter_code># check top 10 movies if they have lots of interactions<jupyter_output><empty_output><jupyter_text>Modeling We will use collaborative filtering using matrix factorization approach based on SVD.<jupyter_code>from lightfm import cross_validation

train, test = cross_validation.random_train_test_split(interaction_matrix, test_percentage=.2,
                                                       random_state=np.random.RandomState(seed=12))
test, train<jupyter_output><empty_output><jupyter_text>We will use 77912 interaction to train our model, and 19478 interactions for the testing set<jupyter_code># instanciate
import time

import numpy as np

%matplotlib inline

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from lightfm import LightFM
from lightfm.datasets import fetch_movielens
from lightfm.evaluation import auc_score


num_components = 30
k=15

model = LightFM(no_components=num_components,
                loss='warp',
                k=k)


model.fit(train, epochs=30, verbose=True)<jupyter_output>Epoch 0
Epoch 1
Epoch 2
Epoch 3
Epoch 4
Epoch 5
Epoch 6
Epoch 7
Epoch 8
Epoch 9
Epoch 10
Epoch 11
Epoch 12
Epoch 13
Epoch 14
Epoch 15
Epoch 16
Epoch 17
Epoch 18
Epoch 19
Epoch 20
Epoch 21
Epoch 22
Epoch 23
Epoch 24
Epoch 25
Epoch 26
Epoch 27
Epoch 28
Epoch 29<jupyter_text>Evaluation<jupyter_code>from lightfm.evaluation import precision_at_k
f"Precision @k=10 train : {precision_at_k(model, test, train_interactions=train).mean():.2%}"<jupyter_output><empty_output><jupyter_text>Precision at K is very sensitive to order, if the predicted item ranking is not correct this will penalize the score. Let's also check AUC score.<jupyter_code>test_auc = auc_score(model, test, train_interactions=train).mean()
print('Collaborative filtering test AUC: %s' % test_auc)
test_auc = auc_score(model, train).mean()
print('Collaborative filtering train AUC: %s' % test_auc)<jupyter_output>Collaborative filtering train AUC: 0.9786382<jupyter_text>Prediction We can predict the recommended items for a given user based on historical training data of interactions<jupyter_code>train.shape[1]
user_rid = 2
user_idx = rid_to_idx[user_rid]
score_pred = model.predict(user_idx, np.arange(train.shape[1]))

# let's take top 5 movies
top_items_idx = np.argsort(-score_pred)[:10]
top_movie_ids = [idx_to_cid[idx] for idx in top_items_idx]

print('Recommended movies by the model')
movies.set_index('movieId').loc[top_movie_ids,:][['title', 'genres']]<jupyter_output>Recommended movies by the model<jupyter_text>Known positives for this user<jupyter_code>pd.merge(ratings_filtered[ratings_filtered.userId==user_rid].sort_values(by='rating',ascending=False)[:10], movies, on='movieId')[['title', 'rating', 'genres']]<jupyter_output><empty_output><jupyter_text>the model has calculated embeddings or so called latent vectors for each movie and each user in our dataset. Embeddings are a vectorized version of an object. We will use them later to calculate distances and make inferences.The model has used matrix factorization technique to decompose our interaction matrix into a product of item_embeddings x user_embeddings = Interaction_matrix<jupyter_code>from sklearn.metrics.pairwise import cosine_similarity

# score
similarity_scores_movies = cosine_similarity(model.item_embeddings)
similarity_scores_movies = pd.DataFrame(similarity_scores_movies)
similarity_scores_movies.shape<jupyter_output><empty_output><jupyter_text>Now that we have computed the simularity score between all the movies in our dataset. Let's see what are the 10 most similar movie to `Toy Story (1995` which has the movieId `1`<jupyter_code>movieId = 1
idx = cid_to_idx[movieId]
top_10_index = similarity_scores_movies.iloc[:,idx].sort_values(ascending=False)[:10].index.to_list()
top_10_index = [idx_to_cid[idx] for idx in top_10_index]

movies.set_index('movieId').loc[top_10_index,:]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09932978174944149, 'fraction_numerical': 0.018044337515036948, 'mean_word_length': 6.552887735236859, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 80, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '306680', 'license': 'MIT', 'n_tokens_mistral': 4042, 'n_tokens_neox': 3742, 'n_words': 1073, 'path': 'recommendation-engine/Recommendation-engine-movies.ipynb', 'repo_name': 'bizankad/DS-portfolio', 'size': '11638'}