<jupyter_start><jupyter_text>This is a content based movie recommendation system (Based on tags)  Importing required libraries<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Importing the data file<jupyter_code>movies = pd.read_csv('tmdb_5000_movies.csv')
credits = pd.read_csv('tmdb_5000_credits.csv')<jupyter_output><empty_output><jupyter_text>Analyzing the data<jupyter_code>movies.head(2)
credits.head(2)<jupyter_output><empty_output><jupyter_text>we will merge the data based on 'title' or 'id' to make analysis easier and create a new column 'df'<jupyter_code>df = movies.merge(credits,on="title") #merging the data file based on title
df.merge(credits,on="title").shape
movies.shape
credits.shape<jupyter_output><empty_output><jupyter_text>Know we will filter only the required and important data for the model<jupyter_code>df.head(2)
#RequiredData =
#genres
#id
#keywords
#title
#overview
#cast
#crew
df = df[['genres', 'id', 'keywords', 'title', 'overview', 'cast', 'crew']]
df.head(2)<jupyter_output><empty_output><jupyter_text>Check for duplicate and missing data<jupyter_code>df.isnull().sum() # check for missing data
df.dropna(inplace=True)   # only 3 missing data so we will remove and replace them
df.isnull().sum()      #Checking them again for verification
df.duplicated().sum()         # check for Duplicate data
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 4806 entries, 0 to 4808
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype 
---  ------    --------------  ----- 
 0   genres    4806 non-null   object
 1   id        4806 non-null   int64 
 2   keywords  4806 non-null   object
 3   title     4806 non-null   object
 4   overview  4806 non-null   object
 5   cast      4806 non-null   object
 6   crew      4806 non-null   object
dtypes: int64(1), object(6)
memory usage: 300.4+ KB<jupyter_text>know we will clean each columns<jupyter_code>df.iloc[0].genres
#'[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 878, "name": "Science Fiction"}]'
#this is a list of dictionaries

#we have to bring it to this form ["Action", "Adventure", "Fantasy", "Science Fiction"]
##genre is a an object we have to convert them into list
# Therefor we will use a new function "ast.literal_eval()"
import ast
ast.literal_eval('[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 878, "name": "Science Fiction"}]'
)

#This is just an example, we will use ast function in next step
def convert(obj):
    L = []
    for i in ast.literal_eval(obj):
        L.append(i['name'])
    return L
df['genres'] = df.genres.apply(convert)
df.head(2)
# Genres column is comlete, know we will do the same for key words
df['keywords'] = df.keywords.apply(convert)    #NOW WE WILL GET LIST OF ALL KEY WORDS
df.head(2)
df['cast'][0]  
# this is the list of all actors in a single movie, 
#But we need only top 3 cast (for example hero heroine, director)
#This is to extract onlt top 3 names in cast

def convert3(obj):
    L = []
    counter = 0
    for i in ast.literal_eval(obj):
        if counter != 3:
            L.append(i['name'])
            counter+=1
        else: 
            break
    return L
df['cast'] = df.cast.apply(convert3) 
df.head(2)
# Now repeat same i.e here we need only director
df['crew'][0]
 def get_director(obj):
    L =[]
    for i in ast.literal_eval(obj):
        if i['job']== 'Director':
            L.append(i['name'])
            break
    return L 
df['crew'].apply(get_director)
df['crew'] = df['crew'].apply(get_director)
df.head(2)
#Now only overview is in form of string, we will convert that also to list
df['overview'] = df['overview'].apply(lambda x:x.split())
df.head()   # checking wether the job is done<jupyter_output><empty_output><jupyter_text>tags Now all columns are converted into lists, we will join all list, convert that large list into a string. That string ("Large Paragraph") will act as the "TAG"  ASLO WE HAVE TO JOIN 'FIRST NAME' AND 'LAST NAME' in all columns  for example: Science Fiction = "ScienceFiction",  "<NAME>" = "SamWorthington"<jupyter_code>df['genres'].apply(lambda x:[i.replace(" ","") for i in x])
df.info()
df.isnull().sum()
df.dropna(inplace=True) 
df.isnull().sum()
df['genres'] = df['genres'].apply(lambda x:[i.replace(" ","") for i in x])
df['keywords'] = df['keywords'].apply(lambda x:[i.replace(" ","") for i in x])
df['crew'] = df['crew'].apply(lambda x:[i.replace(" ","") for i in x])
df['cast'] = df['cast'].apply(lambda x:[i.replace(" ","") for i in x])
df.head(2)<jupyter_output><empty_output><jupyter_text>Now we will concat the data of 'genre', 'keyword', 'crew', 'cast' , 'overview' into Tags<jupyter_code>df['tags'] = df['genres'] + df['keywords'] + df['crew'] + df['cast'] + df['overview'] 
df.head(2)
new_df = df[['id', 'title', 'tags']]
new_df<jupyter_output><empty_output><jupyter_text>the data in 'tags' is a list, now we will convert them into string<jupyter_code>new_df['tags'].apply(lambda x:" ".join(x))
new_df['tags'] = new_df['tags'].apply(lambda x:" ".join(x))
new_df.head(1)
new_df['tags'][0]<jupyter_output><empty_output><jupyter_text>we will convert all data to 'lower case' [Highly Recommedded]<jupyter_code>new_df['tags'].apply(lambda x: x.lower())
new_df['tags'] = new_df['tags'].apply(lambda x: x.lower())
new_df
new_df['tags'][0]<jupyter_output><empty_output><jupyter_text>Vectorization<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(max_features=5000, stop_words='english')
cv.fit_transform(new_df['tags']).toarray().shape
vectors = cv.fit_transform(new_df['tags']).toarray()
vectors[0] # this is the array of the 1st movie
#let see the 5000 features names now
cv.get_feature_names()<jupyter_output><empty_output><jupyter_text>Stemming<jupyter_code>import nltk
from nltk.stem.porter import PorterStemmer
ps = PorterStemmer()

def stem(text):
    y = []
    for i in text.split():
        y.append(ps.stem(i))
        
    return " ".join(y)
stem('action adventure fantasy sciencefiction cultureclash future spacewar spacecolony society spacetravel futuristic romance space alien tribe alienplanet cgi marine soldier battle loveaffair antiwar powerrelations mindandsoul 3d jamescameron samworthington zoesaldana sigourneyweaver in the 22nd century, a paraplegic marine is dispatched to the moon pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization.')<jupyter_output><empty_output><jupyter_text>Our stemming is working, apply that to our tags<jupyter_code>new_df['tags'].apply(stem)

new_df['tags'] = new_df['tags'].apply(stem)<jupyter_output><ipython-input-118-be18a4346d89>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  new_df['tags'] = new_df['tags'].apply(stem)<jupyter_text>apply vectorization again<jupyter_code>cv.fit_transform(new_df['tags']).toarray().shape
vectors = cv.fit_transform(new_df['tags']).toarray()
cv.get_feature_names()<jupyter_output><empty_output><jupyter_text>Measuring Cosine distances<jupyter_code>from sklearn.metrics.pairwise import cosine_similarity

cosine_similarity(vectors)
cosine_similarity(vectors).shape
similatity_matrix = cosine_similarity(vectors)
similatity_matrix[0]
similatity_matrix[1]
similatity_matrix  # this is our similarity matrix the diagonal will be 1 becouse, avatar will be similar to avatar movie

# we havew to fix the index of similarity matrix because there shouldn't be ant confusion for our model about the index of similarity matrix
# we will use "enumerate" function to do indexing
enumerate(similatity_matrix[0])
list(enumerate(similatity_matrix[0])) # here we are clearly getting the simalrity matrix with indexes
sorted(similatity_matrix[0], reverse=True)  # sorting of similarity matrix os movies similar to 'Avatar'=[0]
sorted(list(enumerate(similatity_matrix[0])), reverse=True)
####here sorting hapend according to the 1st number(4799) i.e the index position,
###but we have to do sorting based on similarity number


sorted(list(enumerate(similatity_matrix[0])), reverse=True, key=lambda x:x[1])

sorted(list(enumerate(similatity_matrix[0])), reverse=True, key=lambda x:x[1])[1:6]

# here we are getting movies similar to 'AVATAR'[0] (but only 'id' and wee need the movie title of that id<jupyter_output><empty_output><jupyter_text>Now we willl define the "recommend" function<jupyter_code># 1st we will write function to fetch the index
new_df[new_df['title'] == 'Batman'].index[0]
def recommend(movie):
    movie_index = new_df[new_df['title'] == movie].index[0]
    distances = similatity_matrix[movie_index]
    movies_list = sorted(list(enumerate(distances)), reverse=True, key=lambda x:x[1])[1:6]

    for i in movies_list:
        print(i[0])
recommend('Avatar')   # this is giving recommendation of ids, but we need title
df.iloc[3].title 
def recommend(movie):
    movie_index = new_df[new_df['title'] == movie].index[0]
    distances = similatity_matrix[movie_index]
    movies_list = sorted(list(enumerate(distances)), reverse=True, key=lambda x:x[1])[1:6]

    for i in movies_list:
        print(df.iloc[(i[0])].title )

recommend('Avatar')  # testing our madel
recommend('Batman')  # testing our madel
recommend('<NAME>')  # testing our madel<jupyter_output>The Dark Knight
The Shipping News
The Dark Knight Rises
Batman v Superman: Dawn of Justice
Synecdoche, New York
{'fraction_non_alphanumeric': 0.13454208303286289, 'fraction_numerical': 0.016998042649634285, 'mean_word_length': 5.303896103896104, 'pattern_counts': {'":': 16, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '303145', 'license': 'MIT', 'n_tokens_mistral': 3474, 'n_tokens_neox': 3287, 'n_words': 943, 'path': 'Projects/TMDB_Movie_Recommendation/Movie_Recommendation_System.ipynb', 'repo_name': 'ShivarajShivaprakasha/Machine-Learning', 'size': '9723'}