<jupyter_start><jupyter_text>Recommender System importing the libraries<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt 
%matplotlib inline<jupyter_output><empty_output><jupyter_text>importing the dataset and converting it into DataFrame<jupyter_code>df = pd.read_csv('D:/Data Science Stuff/Python for Data Science/3. Jupyter Overview/Refactored_Py_DS_ML_Bootcamp-master/19-Recommender-Systems/u.data', sep='\t', names= ['user_id', 'item_id', 'rating', 'timestamp'])
#checking first few rows of the data
df.head()
#seems like this is a tab separated data, we can use sep="\t" while reading the file<jupyter_output><empty_output><jupyter_text>The data we have imported is called as the movie lens data<jupyter_code># But we don't have movie titles in it, we have it in a separate file, we will import that too..
movie_titles = pd.read_csv('D:/Data Science Stuff/Python for Data Science/3. Jupyter Overview/Refactored_Py_DS_ML_Bootcamp-master/19-Recommender-Systems/Movie_Id_Titles')
movie_titles[:5]<jupyter_output><empty_output><jupyter_text>Now we will merge this movie_titles to the original dataframe based on the item_id<jupyter_code>df = pd.merge(df, movie_titles, on='item_id')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Number of unique movies present in the data<jupyter_code>df['title'].nunique()<jupyter_output><empty_output><jupyter_text>now let's find the mean ratings for every movie<jupyter_code>df.groupby('title')['rating'].mean().sort_values(ascending= False).head()<jupyter_output><empty_output><jupyter_text>* What we did here is that, we had ratings given from individual users and recorded in a dataset. * So now we grouped the movies and calculated their average ratings given from all the users to a movie. * the data shown above might not be correct because, these might be those movies where only 1 user might have watched it rated them a 5 star Movies with the most ratings<jupyter_code>df.groupby('title')['rating'].count().sort_values(ascending= False).head()<jupyter_output><empty_output><jupyter_text>* as we can see here that these are the popular movies and they have high number of reviews/ ratings from the users* From this data we can see that these are the popular movies from the entire collection of dataset We create a separate dataframe off average ratings from this data<jupyter_code>ratings = pd.DataFrame(df.groupby('title')['rating'].mean())
ratings.head()<jupyter_output><empty_output><jupyter_text>Now let's add a column to this ratings dataframe which tells us about the number of ratings based off the title<jupyter_code>ratings['num_of_ratings'] = pd.DataFrame(df.groupby('title')['rating'].count())
ratings.head()<jupyter_output><empty_output><jupyter_text>*TIME FOR SOME EDA* A histogram which tells us about the distribution of number of ratings<jupyter_code>sns.set_style('whitegrid')
sns.displot(ratings.num_of_ratings, bins=70, color= 'green')<jupyter_output><empty_output><jupyter_text>* Here, we can see that there are lot of movies where only one of the user has rated the movie A histogram which tells us about the distribution of ratings<jupyter_code>sns.set_style('darkgrid')
sns.displot(ratings.rating, color= 'red')<jupyter_output><empty_output><jupyter_text>* From this visualisation, we see that the distribution is normally distributed* Except, for the outliers at 1, which clearly states that many movies have been rated poor * We see another intreseting fact that the frequency of the ratings is more on whole numbered ratings which shows how the users rate the movies in the scale of 1-5 Let's have a look at bi-variate analysis between ratings and number of ratings<jupyter_code>sns.set_style('whitegrid')
sns.jointplot(x='rating', y='num_of_ratings', data=ratings, alpha= 0.5, color='red')<jupyter_output><empty_output><jupyter_text>* From this visualisation we can make out that, there are many number of ratings for some of the popular movies maybe* As expected, there are few movies which are rated 5 and 1 and not many users have rated them, so we have to filter this items in the further process. **Now building a recommendation engine based on item** Create a matrix that has user user_Id one one axis and the movie titles on the another axis<jupyter_code>movie_matrix = df.pivot_table(index='user_id', columns='title', values= 'rating')
movie_matrix.head()<jupyter_output><empty_output><jupyter_text>we see that there are many null values present which is obvious that a single user might have not watched all the movies and have rated them. Listing out the popular movies<jupyter_code>ratings.sort_values(by='num_of_ratings', ascending= False).head()<jupyter_output><empty_output><jupyter_text>We filter out 2 from the popular movies and compare them to the correaltion between them with other movies<jupyter_code>starwars_ratings = movie_matrix['Star Wars (1977)']
liarliar_ratings = movie_matrix['<NAME> (1997)']<jupyter_output><empty_output><jupyter_text>Now, we group the movies which are similar to starwars<jupyter_code>similar_to_starwars = movie_matrix.corrwith(starwars_ratings)
similar_to_starwars[:10]
corr_starwars = pd.DataFrame(similar_to_starwars, columns= ['Correlation'])
corr_starwars.dropna(inplace=True)
corr_starwars[:5]<jupyter_output><empty_output><jupyter_text>* This tells us that how correlated this movies user ratings were to the use ratings of star wars movies.* But there are some movies which doesn't give the result we are expecting, because there might be some users who might have watched only one movie and would have rated similar to Star wars.<jupyter_code>corr_starwars.sort_values(by='Correlation', ascending= False).head(10)<jupyter_output><empty_output><jupyter_text>So we can set a threshold/ minimum number of ratings necessary in order to consider it for our recommendation modelSo we filter out the movies which has <100 num_of_rating<jupyter_code>corr_starwars = corr_starwars.join(ratings.num_of_ratings)
corr_starwars.sort_values(by='Correlation', ascending= False)[:10]
corr_starwars[corr_starwars.num_of_ratings > 100].sort_values(by='Correlation', ascending= False)[1:10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08537371134020619, 'fraction_numerical': 0.006926546391752577, 'mean_word_length': 7.169736842105263, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '506638', 'license': 'MIT', 'n_tokens_mistral': 1952, 'n_tokens_neox': 1879, 'n_words': 662, 'path': 'Machine Learning/Recommender Systems/Recommender.ipynb', 'repo_name': 'Mithun162001/Python-Notebooks', 'size': '6211'}