<jupyter_start><jupyter_text>Anime Recommendation System using Pearson Correlation<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from warnings import simplefilter
simplefilter(action='ignore', category=FutureWarning)<jupyter_output><empty_output><jupyter_text>Load the datasets* 2 different datasets will be loaded in to dataframes * Dataset can be downloaded in https://www.kaggle.com/CooperUnion/anime-recommendations-database<jupyter_code>anime = pd.read_csv('datasets/anime.csv')
rating = pd.read_csv('datasets/rating.csv')<jupyter_output><empty_output><jupyter_text>anime.csv* anime_id - myanimelist.net's unique id identifying an anime.* name - full name of anime.* genre - comma separated list of genres for this anime.* type - movie, TV, OVA, etc.* episodes - how many episodes in this show. (1 if movie).* rating - average rating out of 10 for this anime.* members - number of community members that are in this anime's "group". rating.csv* user_id - non identifiable randomly generated user id.* anime_id - the anime that this user has rated.* rating - rating out of 10 this user has assigned (-1 if the user watched it but didn't assign a rating).<jupyter_code>anime.info()
rating.info()
print('anime.csv (shape):',anime.shape)
print('rating.csv (shape):',rating.shape)
anime.head()
rating.head()
# checking for null values

anime.isnull().sum()
# filling all anime without rating with 0

anime.fillna({'rating':0},inplace=True)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis* Distribution of anime type* Anime with highest rating counts* Movie and TV ratings* Number of episodes and rating* Number of ratings given by the user* Anime rating count  Distribution of anime type<jupyter_code>sns.countplot(x='type',data=anime)
plt.title('Distribution of anime type')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see from the chart the different types of anime present in the dataset.  Anime with highest rating counts<jupyter_code>top_anime_rating_count = rating.groupby(by='anime_id').count()['rating'].sort_values(ascending=False).head(10)
top_anime_rating_count = pd.DataFrame({'rating_count':top_anime_rating_count}).reset_index()
top_anime_rating = pd.merge(top_anime_rating_count,anime,on='anime_id')

plt.title("Top anime with highest rating count")
sns.barplot(x='rating_count',y='name',data=top_anime_rating,orient='h')
plt.show()<jupyter_output><empty_output><jupyter_text>The anime 'Death Note' has the highest rating count with almost 40000 ratings.  Movie and TV ratings<jupyter_code>fig,ax = plt.subplots(ncols=2,figsize=(14,6))

sns.kdeplot(x='rating',color='b',shade=True,data=anime[(anime['type']=='Movie')],label='Movies',ax=ax[0])
sns.kdeplot(x='rating',color='r',shade=True,data=anime[(anime['type']=='TV')],label='TV',ax=ax[0])
sns.boxplot(x='type',y='rating',data=anime[(anime['type']=='Movie')|(anime['type']=='TV')],ax=ax[1])
ax[0].legend()
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the ratings in movies are more spread out than ratings in TV  Number of episodes(TV) and rating<jupyter_code># we will only get the episodes for anime series in TV

episodes_rating = anime[(anime['episodes']!='Unknown')&(anime['type']=='TV')].astype({'episodes':int})

# we can see that there are outliers present
# for the sake of visualization, we will only consider anime with below 100 episodes

episodes_rating['episodes'].describe()
episodes_rating_filtered = episodes_rating[episodes_rating['episodes']<100]
sns.kdeplot(x='episodes', y='rating',data=episodes_rating_filtered,shade=True)
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that most of the anime with episodes ranging 10-20,20-30 and 45-55 receives a rating of around 7.  Number of ratings given by the user<jupyter_code>user_rating_count = rating.groupby(by='user_id').count()['anime_id']
user_rating_count.describe()
sns.kdeplot(x=user_rating_count,shade=True)
plt.title('Number of ratings given by the user')
plt.xlabel("Rating count")
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that most users gave ratings around 250 anime  Anime rating count<jupyter_code>anime_rating_count = rating.groupby(by='anime_id').count()['user_id']
anime_rating_count.describe()
sns.kdeplot(x=anime_rating_count,shade=True)
plt.title('Anime rating count')
plt.xlabel("Rating count")
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that most anime have rating count below 500  Collaborative Filtering using Pearson Correlation```* In this recommendation system, we will be utilizing the collaborative filtering technique.* By using this technique, the system will recommend anime based on the correlation between the ratings of   the user's anime and the ratings of other anime.* For example, I watched 10 anime and gave each of them a rating. Now, my friend watched an anime from my   anime list and now asks me to recommend three anime. With that, I will recommend three anime with closest   rating to the rating I gave for the anime that my friend watched.``` Process```* Remove anime with low count of ratings and users who gave low count of ratings* Construct Rating Matrix* Correlate user's anime with other anime based on ratings* Output ten recommended anime```![collaborative-filtering](images/collaborative-filtering.png)  Remove anime with low count of ratings and users who gave low count of ratings* We will only consider popular anime (rating count over 250) and users who gave lots of rating on different anime (>100)<jupyter_code>anime_rating_count = rating.groupby(by='anime_id').count()['rating'].reset_index().rename(columns={'rating':'rating_count'})
anime_rating_count['rating_count'].describe()
filtered_anime = anime_rating_count[anime_rating_count['rating_count']>250]
# anime with over 250 rating count

filtered_anime.head()
user_rating_count = rating.groupby(by='user_id').count()['rating'].reset_index().rename(columns={'rating':'rating_count'})
user_rating_count['rating_count'].describe()
# users who gave over 100 ratings to different anime

filtered_user = user_rating_count[user_rating_count['rating_count']>100]
filtered_user.head()
filtered_rating_anime = rating[rating['anime_id'].isin(filtered_anime['anime_id'])]
filtered_rating = filtered_rating_anime[filtered_rating_anime['user_id'].isin(filtered_user['user_id'])]
# this dataset now contains popular anime and users wth high rating counts

filtered_rating.head()<jupyter_output><empty_output><jupyter_text>Construct Rating Matrix* We will construct a matrix by using pivot table wherein users will be indexes and anime in columns<jupyter_code># we can see that most of the values are zero since most of the users does not have ratings for every anime

rating_matrix = filtered_rating.pivot_table(index='user_id',columns='anime_id',values='rating').fillna(0)
print(rating_matrix.shape)
rating_matrix.head()<jupyter_output>(24676, 3318)<jupyter_text>Correlate user's anime with other anime based on ratings<jupyter_code>user_anime = anime[anime['name']=='Bleach']
user_anime
user_anime_ratings = rating_matrix[int(user_anime['anime_id'])]
correlated_anime = rating_matrix.corrwith(user_anime_ratings).reset_index().rename(columns={0:'Correlation'})
correlated_anime.head()<jupyter_output><empty_output><jupyter_text>Output ten recommended anime<jupyter_code>recommended_anime = correlated_anime.sort_values(by='Correlation',ascending=False).head(11).iloc[1:]
pd.merge(recommended_anime,anime,on='anime_id',how='left').drop(columns=['anime_id','Correlation'])<jupyter_output><empty_output><jupyter_text>Saving dataframes* Save anime list* Save rating matrix by converting it to csr matrix<jupyter_code># anime list

anime_to_save = pd.merge(filtered_anime,anime,on='anime_id',how='left').sort_values(by='rating',ascending=False)
anime_to_save.to_pickle('output/anime_list.pickle')
anime_list_open = pd.read_pickle('output/anime_list.pickle')
anime_list_open.head()
# converting sparse matrix to csr matrix to save memory

from scipy.sparse import csr_matrix, save_npz, load_npz
import json
csr_rating_matrix =  csr_matrix(rating_matrix.values)
save_npz('output/csr_rating_matrix.npz', csr_rating_matrix)
# saving rating matrix columns and indices

anime_id = {
    'anime_id': rating_matrix.columns.to_list()
}

user_id = {
    'user_id': rating_matrix.index.to_list()
}


with open("output/rating_matrix_anime_id.json","w") as f:
    f.write(json.dumps(anime_id))
with open("output/rating_matrix_user_id.json","w") as f:
    f.write(json.dumps(user_id))
csr_rating_matrix_open = load_npz('output/csr_rating_matrix.npz')

with open('output/rating_matrix_anime_id.json') as f:
    anime_id = json.load(f)
with open('output/rating_matrix_user_id.json') as f:
    user_id = json.load(f)
pd.DataFrame(csr_rating_matrix_open.toarray(),index=user_id['user_id'],columns=anime_id['anime_id']).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11711510550407503, 'fraction_numerical': 0.009266495478396785, 'mean_word_length': 7.834319526627219, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 59, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '304799', 'license': 'MIT', 'n_tokens_mistral': 3012, 'n_tokens_neox': 2870, 'n_words': 737, 'path': 'model/anime_recommendation_pearson_r.ipynb', 'repo_name': 'markbirds/Anime-Recommendation-System', 'size': '8957'}