<jupyter_start><jupyter_text>Movie Recommendation System  I have used https://www.kaggle.com/tmdb/tmdb-movie-metadata for building recommendation systems.There are three types of recommender systems :- 1. Demographic Filtering - It uses the demographic data of a user to determine which items may be appropriate for recommendation.2. Content Based Filtering - It uses item features to recommend other items similar to what the user likes, based on their previous actions or explicit feedback.3. Collaborative Filtering - This system matches persons with similar interests and provides recommendations based on this matching. Collaborative filters do not require item metadata like its content-based counterparts.   Importing libraries and loading data<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
from ast import literal_eval
d1=pd.read_csv('tmdb_5000_credits.csv')
d2=pd.read_csv('tmdb_5000_movies.csv')<jupyter_output><empty_output><jupyter_text>Merging d1 & d2<jupyter_code>d1.columns = ['id','title','cast','crew']
d2 = d2.merge(d1,on='id')
d2.head()<jupyter_output><empty_output><jupyter_text>Demographic FilteringBefore getting started with this -1. we need a metric to score or rate movie2. Calculate the score for every movie3. Sort the scores and recommend the best rated movie to the users.We can use the average ratings of the movie as the score but using this won't be fair enough since a movie with 8.9 average rating and only 3 votes cannot be considered better than the movie with 7.8 as as average rating but 40 votes. So, I'll be using IMDB's weighted rating (wr) which is given as :- ![alt text](wr.png "") where,1. v is the number of votes for the movie;2. m is the minimum votes required to be listed in the chart;3. R is the average rating of the movie; And4. C is the mean vote across the whole reportWe already have v(vote_count) and R (vote_average) and C can be calculated as<jupyter_code>C= d2['vote_average'].mean()
C<jupyter_output><empty_output><jupyter_text>Calculating m, we use 90% percentile as cutoff. It must have more votes than at least 90% of the movies in the list.<jupyter_code>m= d2['vote_count'].quantile(0.9)
m<jupyter_output><empty_output><jupyter_text>Qualified movies<jupyter_code>q_movies = d2.copy().loc[d2['vote_count'] >= m]
q_movies.head(2)
q_movies.shape<jupyter_output><empty_output><jupyter_text>Calculate our metric for each qualified movie using function w_rating<jupyter_code>def w_rating(x, m=m, C=C):
    v = x['vote_count']
    R = x['vote_average']
    # Calculation based on the IMDB formula
    return (v/(v+m) * R) + (m/(m+v) * C)
# Define a new feature 'score' and calculate its value with `w_rating()`
q_movies['score'] = q_movies.apply(w_rating, axis=1)
#Sort movies based on score calculated above
q_movies = q_movies.sort_values('score', ascending=False)

#Print the top 15 movies
q_movies[['title_x', 'vote_count', 'vote_average', 'score']].head(10)<jupyter_output><empty_output><jupyter_text>This recommendation system shows the ' Trending Now '  tab of a streaming app<jupyter_code>pop= d2.sort_values('popularity', ascending=False)

plt.figure(figsize=(12,4))
plt.barh(pop['title_x'].head(6),pop['popularity'].head(6), align='center',color='blue')
plt.gca().invert_yaxis()
plt.xlabel("Popularity")
plt.title("Popular Movies")<jupyter_output><empty_output><jupyter_text>Content Based Filtering 1. Plot description based RecommenderWe are going use Plot description based Recommender for all movies  based on their similarity score.<jupyter_code>d2['overview'].head()<jupyter_output><empty_output><jupyter_text>TfIdfVectorizerWe'll use Term Frequency-Inverse Document Frequency (TF-IDF) vectors for each overview.This is very common algorithm to transform text into a meaningful representation of numbers which is used to fit machine algorithm for prediction. ![alt text](tfidf.png "")<jupyter_code>#Define a TF-IDF Vectorizer Object. Remove all english stop words such as 'the', 'a'
tfidf = TfidfVectorizer(stop_words='english')

#Replace NaN with an empty string
d2['overview'] = d2['overview'].fillna('')

#Construct the required TF-IDF matrix by fitting and transforming the data
tfidf_matrix = tfidf.fit_transform(d2['overview'])

#Output the shape of tfidf_matrix
tfidf_matrix.shape<jupyter_output><empty_output><jupyter_text>Using linear_kernal()We will be using the cosine similarity to calculate a numeric quantity that denotes the similarity between two movies. We use the cosine similarity score since it is independent of magnitude and is relatively easy and fast to calculate. Mathematically, it is defined as follows: ![alt text](simi.png "")<jupyter_code># Compute the cosine similarity matrix
cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)<jupyter_output><empty_output><jupyter_text>Function that takes in a movie title as an input and outputs a list of the 10 most similar movies.<jupyter_code>#Construct a reverse map of indices and movie titles
indices = pd.Series(d2.index, index=d2['title_x']).drop_duplicates()
# Function that takes in movie title as input and outputs most similar movies
def get_recommendations(title, cosine_sim=cosine_sim):
    # Get the index of the movie that matches the title
    idx = indices[title]

    # Get the pairwsie similarity scores of all movies with that movie
    sim_scores = list(enumerate(cosine_sim[idx]))

    # Sort the movies based on the similarity scores
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)

    # Get the scores of the 10 most similar movies
    sim_scores = sim_scores[1:11]

    # Get the movie indices
    movie_indices = [i[0] for i in sim_scores]

    # Return the top 10 most similar movies
    return d2['title_x'].iloc[movie_indices]
get_recommendations('The Godfather')
get_recommendations('Inception')<jupyter_output><empty_output><jupyter_text>2. Credits, Genres and Keywords Based RecommenderWe are going to build a recommender based on the following metadata: the 3 top actors, the director, related genres and the movie plot keywords. This system quality will incraese because of better usage of data.<jupyter_code>features = ['cast', 'crew', 'keywords', 'genres']
for feature in features:
    d2[feature] = d2[feature].apply(literal_eval)
# Get the director's name from the crew feature. If director is not listed, return NaN
def get_director(x):
    for i in x:
        if i['job'] == 'Director':
            return i['name']
    return np.nan
# Returns the list top 3 elements or entire list; whichever is more.
def get_list(x):
    if isinstance(x, list):
        names = [i['name'] for i in x]
        #Check if more than 3 elements exist. If yes, return only first three. If no, return entire list.
        if len(names) > 3:
            names = names[:3]
        return names

    #Return empty list in case of missing/malformed data
    return []
# Define new director, cast, genres and keywords features that are in a suitable form.
d2['director'] = d2['crew'].apply(get_director)

features = ['cast', 'keywords', 'genres']
for feature in features:
    d2[feature] = d2[feature].apply(get_list)
d2[['title_x', 'cast', 'director', 'keywords', 'genres']].head(2)
# Function to convert all strings to lower case and strip names of spaces
def clean_data(x):
    if isinstance(x, list):
        return [str.lower(i.replace(" ", "")) for i in x]
    else:
        #Check if director exists. If not, return empty string
        if isinstance(x, str):
            return str.lower(x.replace(" ", ""))
        else:
            return ''
# Apply clean_data function to your features.
features = ['cast', 'keywords', 'director', 'genres']

for feature in features:
    d2[feature] = d2[feature].apply(clean_data)<jupyter_output><empty_output><jupyter_text>We are now in a position to create our "metadata soup", which is a string that contains all the metadata that we want to feed to our vectorizer<jupyter_code>def create_soup(x):
    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])
d2['soup'] = d2.apply(create_soup, axis=1)<jupyter_output><empty_output><jupyter_text>Using CountVectorizerIt is used to transform a given text into a vector on the basis of the frequency (count) of each word that occurs in the entire text.<jupyter_code># Import CountVectorizer and create the count matrix
from sklearn.feature_extraction.text import CountVectorizer

count = CountVectorizer(stop_words='english')
count_matrix = count.fit_transform(d2['soup'])
# Compute the Cosine Similarity matrix based on the count_matrix
from sklearn.metrics.pairwise import cosine_similarity

cosine_sim2 = cosine_similarity(count_matrix, count_matrix)
# Reset index of our main DataFrame and construct reverse mapping as before
d2 = d2.reset_index()
indices = pd.Series(d2.index, index=d2['title_x'])
get_recommendations('Batman Begins', cosine_sim2)
get_recommendations('The Avengers', cosine_sim2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0961369622475856, 'fraction_numerical': 0.011194029850746268, 'mean_word_length': 5.267537826685007, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 59, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '642897', 'license': 'Apache-2.0', 'n_tokens_mistral': 2865, 'n_tokens_neox': 2727, 'n_words': 996, 'path': 'Recommendation Systems.ipynb', 'repo_name': 'wandererabir/Movie-Recommendation-System', 'size': '9112'}