<jupyter_start><jupyter_text>Content-based filtering<jupyter_code>#Dataframe manipulation library
import pandas as pd
#Math functions, we'll only need the sqrt function so let's import only that
from math import sqrt
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
#Storing the movie information into a pandas dataframe
movies_df = pd.read_csv('/home/ritesh/Desktop/dataset/movies_data/ml-latest/movies.csv')
movies_df.head()
#Storing the user information into a pandas dataframe
ratings_df = pd.read_csv('/home/ritesh/Desktop/dataset/movies_data/ml-latest/ratings.csv')
ratings_df.head()
#Using regular expressions to find a year stored between parentheses
#We specify the parantheses so we don't conflict with movies that have years in their titles
movies_df["year"] = movies_df.title.str.extract('(\(\d\d\d\d\))',expand=False)
movies_df['year']
#Removing the parenthesesis
movies_df["year"] = movies_df.year.str.extract('(\d\d\d\d)',expand=False)
movies_df['year']
#Removing the years from the 'title' column
movies_df['title'] = movies_df.title.str.replace('(\(\d\d\d\d\))',"")
movies_df['title']
#Applying the strip function to get rid of any ending whitespace characters that may have appeared
movies_df["title"] = movies_df["title"].apply(lambda x:x.strip())
movies_df
#Every genre is separated by a | so we simply have to call the split function on |
movies_df['genres'] = movies_df.genres.str.split("|")
movies_df.head()
#Copying the movie dataframe into a new one since we won't need to use the genre information in our first case
moviesWithGenres_df = movies_df.copy()
#For every row in the dataframe, iterate through the list of genres and place a 1 into the corresponding column
for index, row in movies_df.iterrows():
    for genre in row['genres']:
        moviesWithGenres_df.at[index,genre] = 1
        
#Filling in the NaN values with 0 to show that a movie doesn't have that column's genre
moviesWithGenres_df = moviesWithGenres_df.fillna(0)
moviesWithGenres_df<jupyter_output><empty_output><jupyter_text>Now rating dataframe<jupyter_code>ratings_df.head()<jupyter_output><empty_output><jupyter_text>Every row in the ratings dataframe has a user id associated with at least one movie, a rating and a timestamp showing when they reviewed it. We won't be needing the timestamp column,so let's drop it to save on memory.<jupyter_code>ratings_df = ratings_df.drop('timestamp', 1)
ratings_df.head()<jupyter_output><empty_output><jupyter_text>Content-Based recommendation system<jupyter_code>userInput = [
    {"title":"Breakfast Club,The","rating":5},
    {"title":"Toy Story","rating":5},
    {'title':'Jumanji', 'rating':2},
    {'title':"Pulp Fiction", 'rating':5},
            {'title':'Akira', 'rating':4.5}
         ] 
inputMovies= pd.DataFrame(userInput)
inputMovies<jupyter_output><empty_output><jupyter_text>Add movieId to input user<jupyter_code>#Filtering out the movies by title
inputID = movies_df[movies_df["title"].isin(inputMovies["title"].tolist())]
inputID
#Then merging it so we can get the movieId. It's implicitly merging it by title.
inputMovies = pd.merge(inputID, inputMovies)
inputMovies
#Dropping information we won't use from the input dataframe
inputMovies = inputMovies.drop('genres', 1).drop('year', 1)
inputMovies<jupyter_output><empty_output><jupyter_text>We're going to start by learning the input's preferences, so let's get the subset of movies that the input has watched from the Dataframe containing genres defined with binary values.<jupyter_code>#Filtering out the movies from the input
userMovies = moviesWithGenres_df[moviesWithGenres_df['movieId'].isin(inputMovies['movieId'].tolist())]
userMovies<jupyter_output><empty_output><jupyter_text>We'll only need the actual genre table, so let's clean this up a bit by resetting the index and dropping the movieId, title, genres and year columns.<jupyter_code>#Resetting the index to avoid future issues
userMovies = userMovies.reset_index(drop=True)
userMovies
#Dropping unnecessary issues due to save memory and to avoid issues
userGenreTable = userMovies.drop('movieId', 1).drop('title', 1).drop('genres', 1).drop('year', 1)
userGenreTable<jupyter_output><empty_output><jupyter_text>Now we're ready to start learning the input's preferences!To do this, we're going to turn each genre into weights. We can do this by using the input's reviews and multiplying them into the input's genre table and then summing up the resulting table by column. This operation is actually a dot product between a matrix and a vector, so we can simply accomplish by calling Pandas's "dot" function.<jupyter_code>inputMovies['rating']
#Dot produt to get weights
userProfile = userGenreTable.transpose().dot(inputMovies["rating"])
userProfile<jupyter_output><empty_output><jupyter_text>Now, we have the weights for every of the user's preferences. This is known as the User Profile. Using this, we can recommend movies that satisfy the user's preferences.<jupyter_code>#Now let's get the genres of every movie in our original dataframe
genreTable = moviesWithGenres_df.set_index(moviesWithGenres_df['movieId'])
#And drop the unnecessary information
genreTable = genreTable.drop('movieId', 1).drop('title', 1).drop('genres', 1).drop('year', 1)
genreTable.head()
genreTable.shape<jupyter_output><empty_output><jupyter_text>With the input's profile and the complete list of movies and their genres in hand, we're going to take the weighted average of every movie based on the input profile and recommend the top twenty movies that most satisfy it.<jupyter_code>#Multiply the genres by the weights and then take the weighted average
recommendationTable_df = ((genreTable*userProfile).sum(axis=1))/(userProfile.sum())
recommendationTable_df.head()
#Sort our recommendations in descending order
recommendationTable_df = recommendationTable_df.sort_values(ascending=False)
#Just a peek at the values
recommendationTable_df.head()<jupyter_output><empty_output><jupyter_text>Now here's the recommendation table!<jupyter_code>#The final recommendation table
movies_df.loc[movies_df['movieId'].isin(recommendationTable_df.head(20).keys())]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09917221230319753, 'fraction_numerical': 0.003895471514364551, 'mean_word_length': 6.598027127003699, 'pattern_counts': {'":': 4, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '257351', 'license': 'BSD-4-Clause-UC', 'n_tokens_mistral': 1969, 'n_tokens_neox': 1860, 'n_words': 616, 'path': 'content_base.ipynb', 'repo_name': 'kbd95/Content-Based-recommendation-system', 'size': '6161'}