<jupyter_start><jupyter_text>Recommendation Systems - Content Based Filtering (Part 2)> The three part series on building a beginner's recommendation system with Python. This blog provides a simple implementation of content-based filtering in Python.- toc: true- comments: true- categories: [python, recommendation system, relevancy, collaborative filtering, content-based filtering, demographic filtering]  Content Based Filtering This recommendation systems works by finding similarities between the items. If a user has liked or wishlisted some items in the past, this would try to find similar items and recommend to the user. Content-based filtering is also used in Google PageRank algorithm to recommend the relevant webpages basis search keyworks. This is used along with citation model (reference of webpage in other webpages) and behavioral model (the activity on the webpage) to arrive at the final results.We see this type of recommendation in work while searching for items in various apps. In Netflix, we can see some sort of weightage to content based filtering in the section 'Because you watched xxx' (Read the article - https://qz.com/1059434/netflix-finally-explains-how-its-because-you-watched-recommendation-tool-works/)![](img/Netflix3.png)  Loading the Data Let's load our CSV file which we have saved in the Part-1 of the blog!<jupyter_code>import pandas as pd 
import numpy as np
import warnings
warnings.filterwarnings("ignore")

data=pd.read_csv('movies_database.csv')<jupyter_output><empty_output><jupyter_text>Let's look at first few rows of our data!<jupyter_code>data = data[['movie_title','overview','cast','genres','keywords','director']]
data.head()<jupyter_output><empty_output><jupyter_text>We can find the similarity scores between movies based on various metadata. We will first build a model looking at movie plot summaries given in the 'overview' column and then refine our recommendations by including actor, director, genre, etc.  Plot Description based Recommendation  Creating a TF-IDF Vectorizer We first need to convert each overview into its word vector. Next, we will have the find the Term Frequency - Inverse Document Frequency (TF-IDF) vector for each overview. The TF-IDF algorithm is used to weight a word in each document and assign the importance of the word based on the following two factors:- Term Frequency (TF): The number of times the word appears in the document (in our case, the movie plot description)- Inverse Document Frequency (IDF): The number of times the word appears in the corpus, representing how significant the term is in the whole corpus  (in our case, corpus of movie plot descriptions)The below formula is used for TFIDF calculation:![](img/tfidf-formula.png) In Python, 'scikit-learn' library has a pre-built TF-IDF vectorizer that calculates the TF-IDF score for each document’s description, word-by-word.Let's now implement a TFIDF matrix for our data! (Link - https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)<jupyter_code>#Import TfIdfVectorizer from scikit-learn
from sklearn.feature_extraction.text import TfidfVectorizer

#Define a TF-IDF Vectorizer Object. Remove all english stop words such as 'the', 'a', 'an'
tfidf = TfidfVectorizer(stop_words='english')

#Replace NaN with an empty string
data['overview'] = data['overview'].fillna('')

#Construct the required TF-IDF matrix by fitting and transforming the data
tfidf_matrix = tfidf.fit_transform(data['overview'])

#Output the shape of tfidf_matrix
tfidf_matrix.shape<jupyter_output><empty_output><jupyter_text>We can see that over 20000 unique words are used to describe 4803 movies in our dataset. Let's see how the TFIDF matrix looks like!<jupyter_code>#Convert TFIDF matrix to Pandas Dataframe if you want to see the word frequencies.
doc_term_matrix = tfidf_matrix.todense()
df = pd.DataFrame(doc_term_matrix, 
                  columns=tfidf.get_feature_names(), index=data.overview)
df.to_csv('movies_database_tfidf.csv', index=True)
df.head()<jupyter_output><empty_output><jupyter_text>Computing Similarity Score using Cosine Similarity With this matrix in hand, we can now compute a similarity score. We will be using the cosine similarity to calculate a numeric quantity that denotes the similarity between two movies.Mathematically, it measures the cosine of the angle between two vectors projected in a multi-dimensional space. The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance (due to the size of the document), chances are they may still be oriented closer together. The smaller the angle, higher the cosine similarity.![](img/cosine-similarity.png)Reference - https://www.machinelearningplus.com/nlp/cosine-similarity/<jupyter_code># Compute Cosine Similarity
from sklearn.metrics.pairwise import cosine_similarity
cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)
print(cosine_sim)
#Let's create a dataframe of the similarity matrix with rows and columns as movie titles

sim = pd.DataFrame(cosine_sim, 
                  columns=data.movie_title, index=data.movie_title)
sim.head()<jupyter_output><empty_output><jupyter_text>We have now computed the similarity score of each movie with all the other movies based on plot description. Please note, similarity of the movie with itself is 1 and this can be seen in the diagonal in the above matrix.  Implementing the Recommendation System Let's implement a recommendation system where we can input a movie title and the model returns the top 10 movies similar to the movie.<jupyter_code># Create a column of movie titles
indices = pd.Series(data.index, index=data['movie_title']).drop_duplicates()
print(indices)
# Function that takes in movie title as input and outputs most similar movies

def get_recommendations(title, cosine_sim=cosine_sim):
    # Get the index of the movie that matches the title
    idx = indices[title]

    # Get the pairwise similarity scores of all movies with that movie
    sim_scores = list(enumerate(cosine_sim[idx]))

    # Sort the movies in descending order of similarity scores
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)

    # Get the scores of the 10 most similar movies ignoring the first one as it is itself movie
    sim_scores = sim_scores[1:11]

    # Get the movie indices
    movie_indices = [i[0] for i in sim_scores]

    # Return the top 10 most similar movies
    return data['movie_title'].iloc[movie_indices]
get_recommendations('The Godfather')<jupyter_output><empty_output><jupyter_text>We can see the model did a good job in finding the Godfather trilogy movies and other crime movies such as 'Blood Ties' . However, it can be further improved by the following:- Including other features such as Director, Genre, etc. : People interested in 'The Godfather' may be more interested in movies directed by <NAME>. Let's try to include this information too in our model. - People might be interested in different genres based on the movie watched by other users. We can solve this using collaborative filtering, which will be discussed in Part 3 of this blog.  Cast, Genres and Keywords Based Recommendation Now we will build our model based on top 3 actors in the movie, director, top 3 genres of the movie and top 3 keywords of the movie. First, let's load our dataset.<jupyter_code>#Let's load our data
data.head()<jupyter_output><empty_output><jupyter_text>Data Cleaning Firstly, let's clean our data by converting all the text into lowercase and removing spaces in a single name. Example: <NAME> would be converted to christianbale<jupyter_code># Function to convert all strings to lower case and strip names of spaces
def clean_data(x):
    if isinstance(x, list):
        return [str.lower(i.replace(" ", "")) for i in x]
    else:
        #Check if director exists. If not, return empty string
        if isinstance(x, str):
            return str.lower(x.replace(" ", ""))
        else:
            return ''
# Apply clean_data function to your features.
features = ['cast', 'keywords', 'director', 'genres']

for feature in features:
    data[feature] = data[feature].apply(clean_data)<jupyter_output><empty_output><jupyter_text>Let's now combine all the feature data into a single string which combines all the metadata (such as actors, director, keywords and genres) to be feed into the count vectorizer.<jupyter_code>def create_combined_features(x):
    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])
data['combined_features'] = data.apply(create_combined_features, axis=1)<jupyter_output><empty_output><jupyter_text>Creating a Count Vectorizer Now, we have a combined features column in our data. We now apply the count vectorizer which creates a word vector of the entire corpus and provides the frequency of the each word in the document.<jupyter_code># Import CountVectorizer and create the count matrix
from sklearn.feature_extraction.text import CountVectorizer

count = CountVectorizer(stop_words='english')
count_matrix = count.fit_transform(data['combined_features'])
count_matrix.shape<jupyter_output><empty_output><jupyter_text>We can see that 2469 unique words are used to describe 4803 movies in our dataset. Let's see how the Count matrix looks like!<jupyter_code>#Convert count matrix to Pandas Dataframe if you want to see the word frequencies.
doc_term_matrix = count_matrix.todense()
df = pd.DataFrame(doc_term_matrix, 
                  columns=count.get_feature_names(), index=data.combined_features)
df.to_csv('movies_database_countmatrix.csv', index=True)
df.head()<jupyter_output><empty_output><jupyter_text>Computing Similarity Score using Cosine Similarity<jupyter_code># Compute the Cosine Similarity matrix based on the count_matrix
from sklearn.metrics.pairwise import cosine_similarity
cosine_sim2 = cosine_similarity(count_matrix, count_matrix)
#Let's create a dataframe of the similarity matrix with rows and columns as movie titles
sim = pd.DataFrame(cosine_sim2, 
                  columns=data.movie_title, index=data.movie_title)
sim.head()<jupyter_output><empty_output><jupyter_text>We have now computed the similarity score of each movie with all the other movies based on actors, director, keywords and genres. Please note, similarity of the movie with itself is 1 and this can be seen in the diagonal of the above matrix.  Implementing the Recommendation System<jupyter_code># Reset index of our main DataFrame and construct reverse mapping as before
data = data.reset_index()
indices = pd.Series(data.index, index=data['movie_title'])
get_recommendations('The Godfather', cosine_sim2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07072399032198028, 'fraction_numerical': 0.00446677833612507, 'mean_word_length': 5.295840656121851, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 56, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '532122', 'license': 'Apache-2.0', 'n_tokens_mistral': 3046, 'n_tokens_neox': 2864, 'n_words': 1257, 'path': '_notebooks/2020-05-03_Recommendation System_Part 2.ipynb', 'repo_name': 'rahuls0959/ds-blog', 'size': '10768'}