<jupyter_start><jupyter_text>Recommendation Systems - KNN Item-Based Collaborating Filtering (Part 3)> The three part series on building a beginner's recommendation system with Python. This blog provides a simple implementation of collaborating filtering in Python.- toc: true- comments: true- categories: [python, recommendation system, relevancy, collaborative filtering, content-based filtering, demographic filtering]  Collaborating filtering Collaborative filtering approach builds a model from a user’s past behaviors (items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items (or ratings for items) that the user may have an interest in.![](img/collaborative-filtering.png)  Item-Based vs User-Based Collaborative Filtering User based collaborating filtering uses the patterns of users similar to me to recommend a product (users like me also looked at these other items). Item based collaborative filtering uses the patterns of users who browsed the same item as me to recommend me a product (users who looked at my item also looked at these other items). Item-based approach is usually prefered than user-based approach. User-based approach is often harder to scale because of the dynamic nature of users, whereas items usually don't change much, so item-based approach often can be computed offline.  KNN Model Collaborative Filtering models are developed using machine learning algorithms to predict a user’s rating of unrated items. There are several techniques for modeling such as K-Nearest Neighbors (KNN), Matrix Factorization, Deep Learning Models, etc. In this blog, we will be using KNN model.The k-nearest neighbors (KNN) algorithm doesn’t make any assumptions on the underlying data distribution, but it relies on item feature similarity. When a KNN makes a prediction about an item (say, movie), it will calculate the 'distance' between the target movie and every other movie in its database. It then ranks its distances and returns the top 'k' nearest neighbor movies (smallest distance measure) as the most similar movie recommendations. KNN is widely used algorithm in classification models where an item is classified to one of the classes based on majority class in the k-nearest neighbour items. We have to parametrize the value of 'k' in the model. With increasing 'k', the accuracy first increases (due to over-fitting at lower 'k' values) and then decreases at large values of 'k'. The optimum values of 'k' usually lies between 3-10. (Reference - https://machinelearningmastery.com/k-nearest-neighbors-for-machine-learning/)In this blog, we will be developing an item-based KNN collaborating filtering model.  Data Pre-Processing We have taken the data of Movies from Kaggle - https://www.kaggle.com/grouplens/movielens-latest-small which contains over 20 Mn+ ratings of movies by users. Let's explore the dataset.<jupyter_code>import pandas as pd 
import numpy as np
import warnings
warnings.filterwarnings("ignore")

df1=pd.read_csv('movielens-20m-dataset/movie.csv')
df2=pd.read_csv('movielens-20m-dataset/rating.csv')<jupyter_output><empty_output><jupyter_text>The 'movie' file contains movie id, title of the movie and genres. Let's see first few rows of the dataset.<jupyter_code>df1.head()<jupyter_output><empty_output><jupyter_text>The 'rating' file contains ratings given by users to the movies. Let's see first few rows of the dataset.<jupyter_code>df2.head()<jupyter_output><empty_output><jupyter_text>Let's combine these two files on movieId column!<jupyter_code>data = df2.merge(df1,on='movieId')
data.head()<jupyter_output><empty_output><jupyter_text>Let's group the movies and find the total rating count for each movie. This will help in understanding the distribution of rating across movies.<jupyter_code>df3 = (df2.groupby(by=['movieId'])['rating'].count().reset_index().rename(columns={'rating':'totalRatingCount_movies'})
       [['movieId','totalRatingCount_movies']])
df3.tail()<jupyter_output><empty_output><jupyter_text>We can see that there are lot of movies with only single rating also. In order for statistical significance, we should take the movies only with some minimum threshold of ratings. Let's check the statistical measures of the totalRatingCount field.<jupyter_code>df3['totalRatingCount_movies'].describe()<jupyter_output><empty_output><jupyter_text>About 25% of the movies received 205 or more ratings. Because we have so many movies in our data, we will limit it to the top 25% for statistical significance, and this will give us ~6700 movies from total 26744 unique movies. Also, intuitively, the recommendations should be of the popular movies with some threshold of reviews.<jupyter_code>df4 = (df2.groupby(by=['userId'])['rating'].count().reset_index().rename(columns={'rating':'totalRatingCount_users'})
       [['userId','totalRatingCount_users']])
df4.tail()
df4['totalRatingCount_users'].describe()<jupyter_output><empty_output><jupyter_text>About 25% of the users gave less than 35 movie ratings. Because we have so many users in our data, we will limit it to the top users with total movie ratings greater than or equal to 50 for statistical significance, and this will give us nearly 80,000 users from total 1,38,493 users. Also, intuitively, it would be better to consider only movie reviews from avid watchers. Let's add the totalRatingCount_movies and totalRatingCount_users to our combined data!<jupyter_code>#Let's combine data with totalRatingCount
data1 = data.merge(df3,on='movieId')
data2 = data1.merge(df4,on='userId')
data2.head()<jupyter_output><empty_output><jupyter_text>Let's keep only the movies with totalRatingCount_movies and total_ratingCount_users greather than the chosen threshold and store in popular movies dataframe!<jupyter_code>#Let's keep only the movies with totalRatingCount >=205
popular_movies1 = data2.query('totalRatingCount_movies >= 205')
popular_movies = popular_movies1.query ('totalRatingCount_users >= 50')
popular_movies.head()<jupyter_output><empty_output><jupyter_text>KNN Modeling  Reshaping the Data For K-Nearest Neighbors, we want the data to be in an array, where each row is a movie and each column is a different user. To reshape the dataframe, we'll pivot the dataframe to the wide format with movies as rows and users as columns. Then we'll fill the missing observations with 0s since we're going to be performing linear algebra operations (calculating distances between vectors). Finally, we transform the values of the dataframe into a scipy sparse matrix for more efficient calculations. Let's create a pivot of the data with movie ids as rows and user ids as columns!<jupyter_code>#pivot ratings into movie features
movie_user_rating_pivot = popular_movies.pivot(
    index='movieId',
    columns='userId',
    values='rating'
).fillna(0)

# create mapper from movie title to index
movie_to_idx = {
    movie: i for i, movie in 
    enumerate(list(df1.set_index('movieId').loc[movie_user_rating_pivot.index].title))
}
movie_user_rating_pivot<jupyter_output><empty_output><jupyter_text>Thus our data contains movie ratings of 6690 movies by 85307 users. We can see that there are lot of fields with rating as '0'. This means that users have not provided ratings for that movie. It makes sense since there will be large number of ratings for most popular movies and few ratings for less popular or recently released movies.However, working with sparse matrices causes space and time complexity. The solution to representing and working with sparse matrices is to use an alternate data structure to represent the sparse data. The zero values can be ignored and only the data or non-zero values in the sparse matrix need to be stored or acted upon. More on this in the link - https://machinelearningmastery.com/sparse-matrices-for-machine-learning/<jupyter_code>from scipy.sparse import csr_matrix

# convert dataframe of movie features to scipy sparse matrix
movie_user_rating_matrix = csr_matrix(movie_user_rating_pivot.values)<jupyter_output><empty_output><jupyter_text>Fitting the Model Time to implement the model. We'll initialize the NearestNeighbors class as model_knn and fit our sparse matrix (movie_user_rating_matrix) to the instance. By specifying the metric = cosine, the model will measure similarity between movie vectors by using cosine similarity (We have explained cosine similarity in details in part 2 of this blog on content-based filtering).More on nearest neighbours algorithm can be found on the link - https://scikit-learn.org/stable/modules/neighbors.html<jupyter_code># import libraries
from sklearn.neighbors import NearestNeighbors
# define model
model_knn = NearestNeighbors(metric='cosine', algorithm='brute', n_neighbors=20, n_jobs=-1)
# fit
model_knn.fit(movie_user_rating_matrix)
from fuzzywuzzy import fuzz

def fuzzy_matching(mapper, fav_movie, verbose=True):
    """
    return the closest match via fuzzy ratio. If no match found, return None
    
    Parameters
    ----------    
    mapper: dict, map movie title name to index of the movie in data
    fav_movie: str, name of user input movie
    verbose: bool, print log if True

    Return
    ------
    index of the closest match
    """
    match_tuple = []
    # get match
    for title, idx in mapper.items():
        ratio = fuzz.ratio(title.lower(), fav_movie.lower())
        if ratio >= 60:
            match_tuple.append((title, idx, ratio))
    # sort
    match_tuple = sorted(match_tuple, key=lambda x: x[2])[::-1]
    if not match_tuple:
        print('Oops! No match is found')
        return
    if verbose:
        print('Found possible matches in our database: {0}\n'.format([x[0] for x in match_tuple]))
    return match_tuple[0][1]
def make_recommendation(model_knn, data, mapper, fav_movie, n_recommendations):
    """
    return top n similar movie recommendations based on user's input movie

    Parameters
    ----------
    model_knn: sklearn model, knn model
    data: movie-user matrix
    mapper: dict, map movie title name to index of the movie in data
    fav_movie: str, name of user input movie
    n_recommendations: int, top n recommendations

    Return
    ------
    list of top n similar movie recommendations
    """
    # fit
    model_knn.fit(data)
    # get input movie index
    print('You have input movie:', fav_movie)
    idx = fuzzy_matching(mapper, fav_movie, verbose=True)
    # inference
    print('Recommendation system start to make inference')
    print('......\n')
    distances, indices = model_knn.kneighbors(data[idx], n_neighbors=n_recommendations+1)
    # get list of raw idx of recommendations
    raw_recommends = \
        sorted(list(zip(indices.squeeze().tolist(), distances.squeeze().tolist())), key=lambda x: x[1])[:0:-1]
    # get reverse mapper
    reverse_mapper = {v: k for k, v in mapper.items()}
    # print recommendations
    print('Recommendations for {}:'.format(fav_movie))
    for i, (idx, dist) in enumerate(raw_recommends):
        print('{0}: {1}, with distance of {2}'.format(i+1, reverse_mapper[idx], dist))
my_favorite = 'The Dark Knight Rises'

make_recommendation(
    model_knn=model_knn,
    data=movie_user_rating_matrix,
    fav_movie=my_favorite,
    mapper=movie_to_idx,
    n_recommendations=10)<jupyter_output>You have input movie: The Dark Knight Rises
Found possible matches in our database: ['Dark Knight Rises, The (2012)']

Recommendation system start to make inference
......

Recommendations for The Dark Knight Rises:
1: <NAME>: A Game of Shadows (2011), with distance of 0.5127881899481235
2: Amazing Spider-Man, The (2012), with distance of 0.49290540194859955
3: Hunger Games, The (2012), with distance of 0.4854909300858494
4: Hobbit: An Unexpected Journey, The (2012), with distance of 0.47720058748608674
5: Looper (2012), with distance of 0.47505264255188184
6: X-Men: First Class (2011), with distance of 0.4742094475174321
7: Inception (2010), with distance of 0.46427478653960397
8: Skyfall (2012), with distance of 0.44506859529221166
9: Django Unchained (2012), with distance of 0.43321771381173113
10: Avengers, The (2012), with distance of 0.3402240240654465
{'fraction_non_alphanumeric': 0.07778504825781123, 'fraction_numerical': 0.027809586127924096, 'mean_word_length': 4.941205053449951, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 54, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '535688', 'license': 'Apache-2.0', 'n_tokens_mistral': 3726, 'n_tokens_neox': 3369, 'n_words': 1428, 'path': '_notebooks/2020-05-04_Recommendation System_Part 3.ipynb', 'repo_name': 'rahuls0959/ds-blog', 'size': '12235'}