<jupyter_start><jupyter_text>now using this dataframe we have to make a new dataframe which has only three columns  that include : movie_id, title , tags tags column = merge of (overview + genres + keywords + cast + crew)''' before doing that we have to preprocess these columns and for cast and crew ,we will add only the first/top threecast/crew member names and we would need to remove missing data and duplicate data'''<jupyter_code># lets see if there is any missing data

movies.isnull().sum()
''' we can see that there are three movies for which overview is not present ..so we will remove those movies as its
 only three in this case '''
#movies.dropna(inplace=True)   # dropping those movies
movies['overview']=movies['overview'].fillna("")
movies.isnull().sum()
movies.isnull().sum()  #we can see now we don't have any missing data
# now checking if we have any duplicate data or not
movies.duplicated().sum()
#so there is no duplicate data
movies.iloc[0]
movies.iloc[0].genres
''' we want to convert this list of dictionary in this format == > ['Action', 'Adventure', 'Fantansy', 'SciFi']'''
# we will make a function that will do this task

''' def convert(obj):
    L = []
    for i in obj:
        L.append(i['name'])
    return L  '''
# now the problem is that the output of movies.iloc[0].genres is actually a string ,so we need to convert
# this string into list so that we can pass it into convert function to get the correct ans

#import ast
#ast.literal_eval
import ast

def convert(obj):  # this obj is a string
    L = []
    for i in ast.literal_eval(obj):  # and in this step it will become a list
        L.append(i['name'])
    return L 
movies['genres'].apply(convert)
movies['genres'] = movies['genres'].apply(convert)
movies.head()
# we will do the same things on 'keywords' column

movies['keywords'] = movies['keywords'].apply(convert)
movies.head()
#movies['cast'][0]
def convert3(obj):  # this obj is a string
    L = []
    counter = 0
    for i in ast.literal_eval(obj):  # and in this step it will become a list
        if(counter != 3):
            L.append(i['name'])
            counter+=1
        else:
            break
    return L 
movies['cast'] = movies['cast'].apply(convert3)
movies.head()
# movies['crew'][0]
# this has many dictinaries but we want only that dictionary which has the job: director and in that dict
# we need the 'name' value

def fetch_director(obj):  # this obj is a string
    L = []
    for i in ast.literal_eval(obj):  # and in this step it will become a list
        if(i['job'] == 'Director'):
            L.append(i['name'])
            break   # as we want only one director
    return L 
movies['crew'] = movies['crew'].apply(fetch_director)
movies.head()
movies['overview'][0]

# now we can see that the overview column is a string , we will convert this into a list so that we can
# concatanate this with other lists
#func = lambda x:x.split()
#movies['overview'].apply(func)
movies.sample(5)
# now we need to remove the spaces in between --> eg : <NAME> will have two tags 'sam' and 'worthington'
# these are two diff tags pointing to the same person and lets say there is another name <NAME>  ... this 
# will also have the 'sam' tag ..so  recommender will get confused which person to find..thats why we will be removing
# the space in between the two words 
movies['genres'].apply(lambda x: [i.replace(" ","") for i in x])
movies['genres'] = movies['genres'].apply(lambda x: [i.replace(" ","") for i in x])
movies['keywords'] = movies['keywords'].apply(lambda x: [i.replace(" ","") for i in x])
movies['cast'] = movies['cast'].apply(lambda x: [i.replace(" ","") for i in x])
movies['crew'] = movies['crew'].apply(lambda x: [i.replace(" ","") for i in x])
movies.head()
movies['overview'] = movies['overview'].apply(lambda x:x.split())
movies.head()
movies['tags'] = movies['overview'] + movies['genres'] + movies['keywords'] + movies['cast'] + movies['crew']
movies.head()
# now we only need three columns : movie_id,title,tags
# so we will create new dataframe which will have these three columns from movies dataframe

new_df = movies[['movie_id','title','tags']]
new_df
#new_df['tags'][0] # we want to connect all lists with space
new_df['tags'].apply(lambda x : " ".join(x))
new_df['tags'] = new_df['tags'].apply(lambda x : " ".join(x))
new_df.head()
new_df['tags'][0]   #in its output we have overview then genres then keywords and cast and director 
#making all words in lowercase
new_df['tags'] = new_df['tags'].apply(lambda x :x.lower())
new_df.head()
# now our task is to check the similarity of movies....we will use vectorisation for that
# converting text to vector ...known as text vectorisation ....we will be using 'bag of words' technique<jupyter_output><empty_output><jupyter_text>in "bag of words" technique, we combine all the tags...and in that merged large text we will try to find the wordswhich were repeated the most ......in short we will calculate the frequency of all the words  ---> take top 5000 words(means the words that have greater frequency --> top 5000)now in every movie tags for eg : we will check how many times the word "action" came ,"fiction","romantic" ,....5,3,0,1,2....for next movie tag --> 4,6,5,1......(for eg) ...etc so in the end i will have a table with size(5000,5000) ==> (no of movies,most common 5000words) [ you can take any number of words ..5000,10,000 etc) ]now this....5,3,0,1,2......(value at 5000 word)....is a 5000 dimensional vector during this vectorisation we will not consider the stop words ...(example : "are","from","is","a","to")<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(max_features = 5000,stop_words = 'english') # the 2nd attribute will
                                                                 # will remove the stop words of english language
# CountVectorizer returns a scipi sparse matrix by default ,we will convert it to a numpy array
vectors = cv.fit_transform(new_df['tags']).toarray()
vectors[0]
cv.get_feature_names()   # these are the top 5000 words
# we will remove similar words by one word...like actor,actors...activity ,acitivities...etc
# we will use a process called steming ...example how it works---['loved','loving','love'] ==> ['love','love','love']
import nltk
from nltk.stem.porter import PorterStemmer
ps = PorterStemmer()
def stem(text):
    y = []
    for i in text.split():  # string converted to list 
        y.append(ps.stem(i))   #converting every word to stem and storing it in y
    
    string = " ".join(y)  # again converting this list into a string   
    return string    
#example:
print(ps.stem('loved'))
print(ps.stem('love'))
print(ps.stem('dancing'))
print(ps.stem('dance'))
new_df['tags'] = new_df['tags'].apply(stem)
cv.get_feature_names()   # now we can see the stemmed words :)
movies.shape
# we have 4806 movies, that means we have 4806 vectors and every vector has 5000 numbers
# we have to calculate the distance of every movie with every other movie
# we wont be calculating the eucladian distance(tip to tip distance).....we will calculate the cosine distance (angle)
# if(angle == 0)  almost same,if it is 5,10...similar....90 - > dissimilar ...180 --> completely opposite
# hence for more similarity ,the distance has to be less
'''therefore  : similarity is inverse of distance'''
from sklearn.metrics.pairwise import cosine_similarity
similarity = cosine_similarity(vectors)  # we are storing all these diatnces in a variable called similarity
'''1st movie will be compared to 4806 movies ,2nd movie will be comp to 4806 movies and 4806th movie will be compared ot 
4806 movies ...hence size of similarity will be (4806,4806)'''
similarity.shape
similarity  #array of arrays
similarity[0]
# this is the similarity of 1st movie with 4806 movies.....so the similarity of the first movie with itself is 1 .
# and with others is as shown in the above output
similarity[1]   # 2nd movie --> 2nd movie ke saath similarity score is 1...so basically the diagonal of the similarity 
# matrix wil be one
# we will make a function that will return the first 5 similar movies when given a movie name
'''first movie given --> find its index,using that index we will enter into the similarity matrix and 
will find that particular index movie ---- > we will get the similarity array (the distances ) and will sort those distances:
that is more similar movies will come in the beginning ....and then we will fetch the top five movies and display them'''
# now fetching the index of the given movie -- lets say "batman begins"
new_df['title'] == 'Avatar'
new_df[new_df['title'] == 'Avatar']  #masking 
new_df[new_df['title'] == 'Avatar'].index
new_df[new_df['title'] == 'Avatar'].index[0]
new_df[new_df['title'] == 'Batman Begins'].index[0]
# so the index of batman begins is 119 
# now the problem that comes with sorting is that first of all it gets sorted in ascending order,we want it to be 
# in descending order and 2ndly when we sort the data the index gets lost ....meaning like initially the first index of 
# similarity is similarity of the given movie with the first movie,the value at 2nd posn shows the sim of the given movie
# with the 2nd movie and when we sort this array ..these indexes will get lost ,we dont want that to happen
# we want to hold the value of those indexes
# so we will use enumeration
similarity[0]
enumerate(similarity[0])
list(enumerate(similarity[0]))  # now that list has become list of tuples
'''now this is showing that similarity of 0th movie with 1st movie is 1, similarity of 0th movie with 2nd movie is 
0.896..... similarity of 0th movie with 3rd movie is 0.59.....  etc'''
sorted(list(enumerate(similarity[0])),reverse = True)  # reverse = True condition for sorting in descending
#order (highest similarity movie at the top)
# we see that the sorting has happened but not on similarity values but index values ..to fix this problem
sorted(list(enumerate(similarity[0])),reverse = True,key= lambda x : x[1])   # this tells the sorted function
# to do the sorting based on second number(value at 1st index ) rather than 1st number (value at 0th index)
# the top 5 movies
sorted(list(enumerate(similarity[0])),reverse = True,key= lambda x : x[1])[1:6]
'''def recommend(movie):
    movie_index = new_df[new_df['title'] == movie].index[0]
    distances = similarity[movie_index]
    movies_list = sorted(list(enumerate(distances)),reverse=True,key = lambda x: x[1])[1:6]  # to take the first five values
    
    for i in movies_list:
        
        print(i[0])'''
#recommend('Avatar')
#new_df.iloc[1216]
#new_df.iloc[1216].title
def recommend(movie):
    movie_index = new_df[new_df['title'] == movie].index[0]
    distances = similarity[movie_index]
    movies_list = sorted(list(enumerate(distances)),reverse=True,key = lambda x: x[1])[1:6]  # to take the first five values
    
    for i in movies_list:
        print(new_df.iloc[i[0]].title)     # now this will print the movies titles ..instead of the movie indexes of
                                           # top five similar movies
recommend('Avatar')
recommend('Batman Begins')<jupyter_output>The Dark Knight
The Dark Knight Rises
Batman
Batman & Robin
Batman
{'fraction_non_alphanumeric': 0.11567596470902772, 'fraction_numerical': 0.016754299973264414, 'mean_word_length': 3.9414354909731397, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '115331', 'license': 'MIT', 'n_tokens_mistral': 3475, 'n_tokens_neox': 3261, 'n_words': 1417, 'path': 'ML Projects/Movie recommendation system/Movie recommendation system.ipynb', 'repo_name': 'jatin-code21/Recommender', 'size': '11234'}