<jupyter_start><jupyter_text>Recommender Systems with PythonIn this notebook we'll develop basic recommendation systems using python and pandas. Our focus will be on providing a basic recommendation system by suggesting items that are most similar to a particular item, in this case, movies. Keep in mind, this is not a true robust recommendation system, it just tells what movie/items are most similar to the movie choice.  Import Libraries<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Get the Data<jupyter_code>column_names = ['user_id', 'item_id', 'rating', 'timestamp']
df = pd.read_csv('../../datasets/movies/u.data', sep='\t', names=column_names)
df.head()<jupyter_output><empty_output><jupyter_text>Now let's get the movies title<jupyter_code>movies_title = pd.read_csv('../../datasets/movies/Movie_Id_Titles')
movies_title.head()<jupyter_output><empty_output><jupyter_text>We can merge them together<jupyter_code>df = pd.merge(df, movies_title, on='item_id')
df.head()<jupyter_output><empty_output><jupyter_text>EADLet's explore the data a bit and get a look at some of the best rated movies<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set_style('white')<jupyter_output><empty_output><jupyter_text>Let's create a ratings dataframes with average rating and number of ratings<jupyter_code>df.groupby('title')['rating'].mean().sort_values(ascending=False).head()
df.groupby('title')['rating'].count().sort_values(ascending=False).head()
ratings = pd.DataFrame(df.groupby('title')['rating'].mean())
ratings.head()
ratings['number of ratings'] = pd.DataFrame(df.groupby('title')['rating'].count())
ratings.head()<jupyter_output><empty_output><jupyter_text>Now let's check some histograms<jupyter_code>plt.figure(figsize=(10,6))
ratings['number of ratings'].hist(bins=70)
plt.figure(figsize=(10,6))

ratings['rating'].hist(bins=70)<jupyter_output><empty_output><jupyter_text>We can see some outlines in the classifications 5,4,3,2,1. This is probably caused by films that have had few views and, consequently, few ratings (positive or negative). For example, one or two people may have watched a particular film and given it a certain grade, with no significance in the sample space.------<jupyter_code>sns.jointplot(x='rating', y='number of ratings', data=ratings, alpha=0.5)<jupyter_output><empty_output><jupyter_text>Here we can see that the greater the evaluation, the greater the number of evaluations performed.  Recommending Similar MoviesNow let's create a matrix that has the user ids on one access and the movie title on another axis. Each cell will then consist of the rating the user gave to that movie. Note there will be a lot of NaN values, because most people have not seen most of the movies.<jupyter_code>moviemat = df.pivot_table(index='user_id', columns='title', values='rating')
moviemat.head()<jupyter_output><empty_output><jupyter_text>Most rated movie<jupyter_code>ratings.sort_values('number of ratings', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Let's choose two movies: starwars, a sci-fi movie. And Liar Liar, a comedy.<jupyter_code>starwars_user_ratings = moviemat['Star Wars (1977)']
liarliar_user_ratings = moviemat['Liar Liar (1997)']
starwars_user_ratings.head()<jupyter_output><empty_output><jupyter_text>We can then user corrwith() method to get correlations between two pandas series<jupyter_code>similar_to_starwars = moviemat.corrwith(starwars_user_ratings)
similar_to_liarliar = moviemat.corrwith(liarliar_user_ratings)<jupyter_output>C:\Users\Junior\Anaconda3\lib\site-packages\numpy\lib\function_base.py:3175: RuntimeWarning: Degrees of freedom <= 0 for slice
  c = cov(x, y, rowvar)
C:\Users\Junior\Anaconda3\lib\site-packages\numpy\lib\function_base.py:3109: RuntimeWarning: divide by zero encountered in double_scalars
  c *= 1. / np.float64(fact)<jupyter_text>Let's clean this by removing NaN values and using a DataFrame instead of a series<jupyter_code>corr_starwars = pd.DataFrame(similar_to_starwars, columns=['Correlation'])
corr_starwars.dropna(inplace=True)
corr_starwars.head()<jupyter_output><empty_output><jupyter_text>Now if we sort the dataframe by correlation, we should get the most similar movies, however, note that we get some results that don't really make sense. This is because there are a lot of movies only watched once by users who also watched star wars (it was the most popular movie).<jupyter_code>corr_starwars.sort_values('Correlation', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Let's fix this by filtering out movies that have less than 100 reviews (this value was chosen based off the histogram from earlier).<jupyter_code>corr_starwars = corr_starwars.join(ratings['number of ratings'])
corr_starwars.head()<jupyter_output><empty_output><jupyter_text>Now sort the values and notice how the title make a lot more sense<jupyter_code>corr_starwars[corr_starwars['number of ratings'] > 100].sort_values('Correlation', ascending=False).head()<jupyter_output><empty_output><jupyter_text>Now the same for the comedy Liar Liar<jupyter_code>corr_liarliar = pd.DataFrame(similar_to_liarliar, columns=['Correlation'])
corr_liarliar.dropna(inplace=True)
corr_liarliar = corr_liarliar.join(ratings['number of ratings'])
corr_liarliar[corr_liarliar['number of ratings'] > 100].sort_values('Correlation', ascending=False).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10711021751050996, 'fraction_numerical': 0.009504660939499178, 'mean_word_length': 7.825806451612904, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '605672', 'license': 'MIT', 'n_tokens_mistral': 1851, 'n_tokens_neox': 1760, 'n_words': 477, 'path': 'notebooks/theory/Recommender-Systems-Theory.ipynb', 'repo_name': 'jralbbuquerque/ds-machine-learning', 'size': '5471'}