<jupyter_start><jupyter_text>Importing the relevant libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import os
import seaborn as sns
# We can override the default matplotlib styles with those of Seaborn
sns.set()## Importing the relevant libraries
# Load the data from a .csv
raw_credits_data = pd.read_csv(
    os.path.join(os.path.pardir,'data','processed','credits.csv'), low_memory=False)
raw_movies_metadata = pd.read_csv(
    os.path.join(os.path.pardir,'data','processed','movies_metadata.csv'), low_memory=False)
raw_keywords_data = pd.read_csv(
    os.path.join(os.path.pardir,'data','processed','keywords.csv'), low_memory=False)
credits_df = raw_credits_data.copy()
metadata_df = raw_movies_metadata.copy()
keywords_df = raw_keywords_data.copy()<jupyter_output><empty_output><jupyter_text>> 1. Simple Recommenders - simple recommenders are basic systems that recommends the top items based on a certain metric or score.>In this section, we will build a simplified clone of IMDB Top 250 Movies using metadata collected from IMDB.> The following are the steps involved:>- Decide on the metric or score to rate movies on.>- Calculate the score for every movie.> - Sort the movies based on the score and output the top results.<jupyter_code>metadata_df.head()<jupyter_output><empty_output><jupyter_text>>One of the most basic metrics we can think of is the rating. However, using this metric has a few caveats. For one, it does not take into consideration the popularity of a movie. Therefore, a movie with a rating of 9 from 10 voters will be considered 'better' than a movie with a rating of 8.9 from 10,000 voters.***>On a related note, this metric will also tend to favor movies with smaller number of voters with skewed and/or extremely high ratings. As the number of voters increase, the rating of a movie regularizes and approaches towards a value that is reflective of the movie's quality. It is more difficult to discern the quality of a movie with extremely few voters.***>Taking these shortcomings into consideration, it is necessary that we come up with a weighted rating that takes into account the average rating and the number of votes it has garnered. Such a system will make sure that a movie with a 9 rating from 100,000 voters gets a (far) higher score than a YouTube Web Series with the same rating but a few hundred voters.***>Since we are trying to build a clone of IMDB's Top 250, we will use its weighted rating formula as our metric/score. Mathematically, it is represented as follows:![image.png](attachment:image.png)where,>>v is the number of votes for the movie;  >>m is the minimum votes required to be listed in the chart;  >>R is the average rating of the movie; And  >>C is the mean vote across the whole report  ***>We already have the values to `v (vote_count)` and `R (vote_average)` for each movie in the dataset. It is also possible to directly calculate C from this data.***>What we need to determine is an appropriate value for m, the minimum votes required to be listed in the chart. There is no right value for m. We can view it as a preliminary negative filter that ignores movies which have less than a certain number of votes. The selectivity of our filter is up to our discretion.***>In this case, we will use the 90th percentile as our cutoff. In other words, for a movie to feature in the charts, it must have more votes than at least 90% of the movies in the list. (On the other hand, if we had chosen the 75th percentile, we would have considered the top 25% of the movies in terms of the number of votes garnered. As the percentile decreases, the number of movies considered increases. Feel free to play with this value and observe the changes in our final chart).***>As a first step, let's calculate the value of C, the mean rating across all movies:<jupyter_code># Calculate C
C = metadata_df['vote_average'].mean()
C
# The average rating of a movie on IMDB is around 5.6, on a scale of 10.<jupyter_output><empty_output><jupyter_text>>Next, let's calculate the number of votes, m, received by a movie in the 90th percentile. The pandas library makes this task extremely trivial using the .quantile() method of a pandas Series:<jupyter_code># Calculate the minimum number of votes required to be in the chart, m
m = metadata_df['vote_count'].quantile(0.90)
m<jupyter_output><empty_output><jupyter_text>> Next, we can filter the movies that qualify for the chart, based on their vote counts:<jupyter_code># Filter out all qualified movies into a new DataFrame
qualified_movies = metadata_df.copy().loc[metadata_df['vote_count'] >= m]
qualified_movies.shape
qualified_movies.sort_values('vote_average', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>>We see that there are 4555 movies which qualify to be in this list. Now, we need to calculate our metric for each qualified movie. To do this, we will define a function, weighted_rating() and define a new feature score, of which we'll calculate the value by applying this function to our DataFrame of qualified movies:<jupyter_code># Function that computes the weighted rating of each movie
def weighted_rating(x, m=m, C=C):
    v = x['vote_count']
    R = x['vote_average']
    # Calculation based on the IMDB formula
    return (v/(v+m) * R) + (m/(m+v) * C)
# Define a new feature 'score' and calculate its value with `weighted_rating()`
qualified_movies['score'] = qualified_movies.apply(weighted_rating, axis=1)
qualified_movies<jupyter_output><empty_output><jupyter_text>>Finally, let's sort the DataFrame based on the score feature and output the title, vote count, vote average and weighted rating or score of the top 15 movies.<jupyter_code>#Sort movies based on score calculated above
qualified_movies = qualified_movies.sort_values('score', ascending=False)

#Print the top 15 movies
qualified_movies[['title', 'vote_count', 'vote_average', 'score']].head(15)
# To get a graph of the top 10 IMDB movies:

plt.figure(figsize=(16,6))
ax = sns.barplot(x=qualified_movies['score'].head(10), y=qualified_movies['title'].head(10),
                 data=qualified_movies, palette='deep')

plt.xlim(6.75, 8.5)
plt.title('"Top" Movies by IMBD based on score', weight='bold')
plt.xlabel('Weighted Average Score', weight='bold')
plt.ylabel('Movie Title', weight='bold')

# plt.savefig('best_movies.png')<jupyter_output><empty_output><jupyter_text>>We see that the chart has a lot of movies in common with the IMDB Top 250 chart: for example, our top two movies, "<NAME>" and "The Godfather", are the same as IMDB.  2.0 Content-Based Recommender > Plot Description Based Recommender>In this section, we will try to build a system that recommends movies that are similar to a particular movie. More specifically, we will compute pairwise similarity scores for all movies based on their plot descriptions and recommend movies based on that similarity score.***>The plot description is available to we as the `overview` feature in our `metadata_df` dataset<jupyter_code>metadata_df
#Print plot overviews of the first 5 movies.
metadata_df['overview'].head()<jupyter_output><empty_output><jupyter_text>>In its current form, it is not possible to compute the similarity between any two overviews. To do this, we need to compute the word vectors of each overview or document, as it will be called from now on.***>We will compute Term Frequency-Inverse Document Frequency (TF-IDF) vectors for each document. This will give us a matrix where each column represents a word in the overview vocabulary (all the words that appear in at least one document) and each column represents a movie, as before.***>In its essence, the TF-IDF score is the frequency of a word occurring in a document, down-weighted by the number of documents in which it occurs. This is done to reduce the importance of words that occur frequently in plot overviews and therefore, their significance in computing the final similarity score.***>Fortunately, scikit-learn gives we a built-in TfIdfVectorizer class that produces the TF-IDF matrix<jupyter_code>#Import TfIdfVectorizer from scikit-learn
from sklearn.feature_extraction.text import TfidfVectorizer

#Define a TF-IDF Vectorizer Object. Remove all english stop words such as 'the', 'a'
tfidf = TfidfVectorizer(stop_words='english')

#Replace NaN with an empty string
metadata_df['overview'] = metadata_df['overview'].fillna('')

#Construct the required TF-IDF matrix by fitting and transforming the data
tfidf_matrix = tfidf.fit_transform(metadata_df['overview'])

#Output the shape of tfidf_matrix
tfidf_matrix.shape<jupyter_output><empty_output><jupyter_text>>We see that over 75,000 different words were used to describe the 45,000 movies in our dataset.***>With this matrix in hand, we can now compute a similarity score. There are several candidates for this; such as the euclidean, the Pearson and the cosine similarity scores. Again, there is no right answer to which score is the best. Different scores work well in different scenarios and it is often a good idea to experiment with different metrics.***>We will be using the cosine similarity to calculate a numeric quantity that denotes the similarity between two movies. We use the cosine similarity score since it is independent of magnitude and is relatively easy and fast to calculate (especially when used in conjunction with TF-IDF scores). Mathematically, it is defined as follows: > Weighted Rating(WR) = $(\frac{v}{v + m} . R) + (\frac{m}{v + m} . C)$***>Since we have used the TF-IDF vectorizer, calculating the dot product will directly give we the cosine similarity score. Therefore, we will use sklearn's **linear_kernel()** instead of **cosine_similarities()** since it is faster.<jupyter_code># Import linear_kernel
from sklearn.metrics.pairwise import linear_kernel

# Compute the cosine similarity matrix
cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)<jupyter_output><empty_output><jupyter_text>>We're going to define a function that takes in a movie title as an input and outputs a list of the 10 most similar movies. Firstly, for this, we need a reverse mapping of movie titles and DataFrame indices. In other words, we need a mechanism to identify the index of a movie in our metadata DataFrame, given its title<jupyter_code>#Construct a reverse map of indices and movie titles
indices = pd.Series(metadata_df.index, index=metadata_df['title']).drop_duplicates()
indices<jupyter_output><empty_output><jupyter_text>>We are now in a good position to define our recommendation function. These are the following steps we'll follow:***>Get the index of the movie given its title.>1. Get the list of cosine similarity scores for that particular movie with all movies. Convert it into a list of tuples where the first element is its position and the second is the similarity score.>1. Sort the aforementioned list of tuples based on the similarity scores; that is, the second element.>1. Get the top 10 elements of this list. Ignore the first element as it refers to self (the movie most similar to a particular movie is the movie itself).>1. Return the titles corresponding to the indices of the top elements.<jupyter_code># Function that takes in movie title as input and outputs most similar movies
def get_recommendations(title, cosine_sim=cosine_sim):
    # Get the index of the movie that matches the title
    idx = indices[title]

    # Get the pairwsie similarity scores of all movies with that movie
    sim_scores = list(enumerate(cosine_sim[idx]))

    # Sort the movies based on the similarity scores
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)

    # Get the scores of the 10 most similar movies
    sim_scores = sim_scores[1:11]

    # Get the movie indices
    movie_indices = [i[0] for i in sim_scores]
    
    # Return the top 10 most similar movies
    return metadata_df['title'].iloc[movie_indices]
get_recommendations('The Dark Knight Rises')
get_recommendations('The Godfather')<jupyter_output><empty_output><jupyter_text>> We see that, while our system has done a decent job of finding movies with similar plot descriptions, the quality of recommendations is not that great. "The Dark Knight Rises" returns all Batman movies while it more likely that the people who liked that movie are more inclined to enjoy other Christopher Nolan movies. This is something that cannot be captured by our present system.  Credits, Genres and Keywords Based Recommender > We are going to build a recommender based on the following metadata: the 3 top actors, the director, related genres and the movie plot keywords.***> The keywords, cast and crew data is not available in the current dataset so the first step would be merge them into our main DataFrame.<jupyter_code># Convert IDs to int. Required for merging
keywords_df['id'] = keywords_df['id'].astype('int')
credits_df['id'] = credits_df['id'].astype('int')
# # Remove rows with bad IDs.
metadata_df = metadata_df.drop([19730, 29503, 35587])
metadata_df['id'] = metadata_df['id'].astype('int')

# Merge keywords and credits into our main metadata dataframe
metadata_df = metadata_df.merge(credits_df, on='id')
metadata_df = metadata_df.merge(keywords_df, on='id')
metadata_df.head()<jupyter_output><empty_output><jupyter_text>> From our new features, cast, crew and keywords, we need to extract the three most important actors, the director and the keywords associated with that movie. Right now, our data is present in the form of "stringified" lists. We need to convert them into a form that is usable for us.<jupyter_code># Parse the stringified features into their corresponding python objects
from ast import literal_eval

features = ['cast', 'crew', 'keywords', 'genres']
for feature in features:
    metadata_df[feature] = metadata_df[feature].fillna('').apply(literal_eval)
metadata_df<jupyter_output><empty_output><jupyter_text>>Next, we write functions that will help us extract the required information from each feature.<jupyter_code># Get the director's name from the crew feature. If director is not listed, return NaN
def get_director(x):
    for i in x:
        if i['job'] == 'Director':
            return i['name']
    return np.nan
# Returns the list top 3 elements or entire list; whichever is more.
def get_list(x):
    if isinstance(x, list):
        names = [i['name'] for i in x]
        #Check if more than 3 elements exist. If yes, return only first three. If no, return entire list.
        if len(names) > 3:
            names = names[:3]
        return names

    #Return empty list in case of missing/malformed data
    return []
# Define new director, cast, genres and keywords features that are in a suitable form.
metadata_df['director'] = metadata_df['crew'].apply(get_director)

features = ['cast', 'keywords', 'genres']
for feature in features:
    metadata_df[feature] = metadata_df[feature].apply(get_list)
metadata_df
# Print the new features of the first 3 films
metadata_df[['title', 'cast', 'director', 'keywords', 'genres']].head(3)<jupyter_output><empty_output><jupyter_text>> The next step would be to convert the names and keyword instances into lowercase and strip all the spaces between them. This is done so that our vectorizer doesn't count the Johnny of "Johnny Depp" and "Johnny Galecki" as the same. After this processing step, the aforementioned actors will be represented as "johnnydepp" and "johnnygalecki" and will be distinct to our vectorizer.<jupyter_code># Function to convert all strings to lower case and strip names of spaces
def clean_data(x):
    if isinstance(x, list):
        return [str.lower(i.replace(" ", "")) for i in x]
    else:
        #Check if director exists. If not, return empty string
        if isinstance(x, str):
            return str.lower(x.replace(" ", ""))
        else:
            return ''
# Apply clean_data function to our features.
features = ['cast', 'keywords', 'director', 'genres']

for feature in features:
    metadata_df[feature] = metadata_df[feature].apply(clean_data)
metadata_df<jupyter_output><empty_output><jupyter_text>> We are now in a position to create our "metadata soup", which is a string that contains all the metadata that we want to feed to our vectorizer (namely actors, director and keywords).<jupyter_code>def create_soup(x):
    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])
# Create a new soup feature
metadata_df['soup'] = metadata_df.apply(create_soup, axis=1)
metadata_df<jupyter_output><empty_output><jupyter_text>>The next steps are the same as what you did with your plot description based recommender. One important difference is that you use the CountVectorizer() instead of TF-IDF. This is because you do not want to down-weight the presence of an actor/director if he or she has acted or directed in relatively more movies. It doesn't make much intuitive sense.<jupyter_code># Import CountVectorizer and create the count matrix
from sklearn.feature_extraction.text import CountVectorizer

count = CountVectorizer(stop_words='english')
count_matrix = count.fit_transform(metadata_df['soup'])

# Compute the Cosine Similarity matrix based on the count_matrix
from sklearn.metrics.pairwise import cosine_similarity

cosine_sim2 = cosine_similarity(count_matrix, count_matrix)
# Reset index of your main DataFrame and construct reverse mapping as before
metadata_df = metadata_df.reset_index()
indices = pd.Series(metadata_df.index, index=metadata_df['title'])<jupyter_output><empty_output><jupyter_text>> We can now reuse our get_recommendations() function by passing in the new cosine_sim2 matrix as our second argument.<jupyter_code>get_recommendations('The Dark Knight Rises', cosine_sim2)
get_recommendations('The Godfather', cosine_sim2)<jupyter_output><empty_output><jupyter_text>> We see that our recommender has been successful in capturing more information due to more metadata and has given us (arguably) better recommendations. There are, of course, numerous ways of playing with this system in order to improve recommendations.***> **Some suggestions:**> 1. Introduce a popularity filter: this recommender would take the list of the 30 most similar movies, calculate the weighted ratings (using the IMDB formula from above), sort movies based on this rating and return the top 10 movies.>1. Other crew members: other crew member names, such as screenwriters and producers, could also be included.>1. Increasing weight of the director: to give more weight to the director, he or she could be mentioned multiple times in the soup to increase the similarity scores of movies with the same director.  3.0 Collaborative Filtering with Python > Collaborative filters can further be classified into two types:> **User-based Filtering:** these systems recommend products to a user that similar users have liked. For example, let's say Alice and Bob have a similar interest in books (that is, they largely like and dislike the same books). Now, let's say a new book has been launched into the market and Alice has read and loved it. It is therefore, highly likely that Bob will like it too and therefore, the system recommends this book to Bob.***> **Item-based Filtering**: these systems are extremely similar to the content recommendation engine that you built. These systems identify similar items based on how people have rated it in the past. For example, if Alice, Bob and Eve have given 5 stars to The Lord of the Rings and The Hobbit, the system identifies the items as similar. Therefore, if someone buys The Lord of the Rings, the system also recommends The Hobbit to him or her.<jupyter_code>metadata_df
df_matrix = pd.pivot_table(metadata_df, values='purchase_count', index='customerId', columns='productId')
df_matrix.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07506932190572221, 'fraction_numerical': 0.0069573985379379885, 'mean_word_length': 5.1468856523086455, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 147, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '330992', 'license': 'MIT', 'n_tokens_mistral': 5538, 'n_tokens_neox': 5241, 'n_words': 2589, 'path': 'notebooks/Simple recommenders | Content-based recommenders | Collaborative filtering engines.ipynb', 'repo_name': 'Hesbon5600/movie-recommendation-system', 'size': '19849'}