<jupyter_start><jupyter_text>Hybrid book recommender system The objetive of this project is to create a hybrid book recommendation system that combines three types of recommenders:- Simple recommender- Collaborative filtering engine- Content-based recommendersThe dataset user is available at https://www.kaggle.com/arashnic/book-recommendation-dataset/version/1More information on the three types of recommenders available at https://www.datacamp.com/community/tutorials/recommender-systems-python (partial tutorial, using a different dataset).  Imports<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Preprocess Books.csv  Load data<jupyter_code>books_raw = pd.read_csv ('dataset1/Books.csv',
                         dtype={
                             'ISBN' : str, # some values are not integers
                             'Book-Title' : str,
                             'Book-Author' : str,
                             'Year-Of-Publication' : str, # some values are not integers
                             'Publisher' : str,
                             'Image-URL-S' : str},
     usecols=['ISBN','Book-Title','Book-Author','Year-Of-Publication','Publisher','Image-URL-S'])<jupyter_output><empty_output><jupyter_text>Explore 'books_raw' Books are identified by their respective ISBN. Invalid ISBNs have already been removed from the dataset. Moreover, some content-based information is given (Book-Title, Book-Author, Year-Of-Publication, Publisher), obtained from Amazon Web Services. Note that in case of several authors, only the first is provided. URLs linking to cover images are also given, appearing in three different flavours (Image-URL-S, Image-URL-M, Image-URL-L), i.e., small, medium, large. These URLs point to the Amazon web site.<jupyter_code>books_raw.shape
books_raw.count()<jupyter_output><empty_output><jupyter_text>There are missing values at least in Book-Author and Publisher<jupyter_code>books_raw.isna().sum()<jupyter_output><empty_output><jupyter_text>Analyse ISBN  Are all ISBN numbers?<jupyter_code>pd.to_numeric(books_raw['ISBN'], errors='coerce').isna().sum()<jupyter_output><empty_output><jupyter_text>Some ISBM have the letter X at the end, so we'll leave them as strings  Any ISBN duplicates?<jupyter_code>books_raw.ISBN.value_counts().max()<jupyter_output><empty_output><jupyter_text>No repeated ISBN  Analyse Book-Title  Any missing values?<jupyter_code>pd.isnull(books_raw["Book-Title"]).sum()<jupyter_output><empty_output><jupyter_text>Great! No missing Book-Title  Any Book-Title duplicates?<jupyter_code>books_raw[books_raw.duplicated(['Book-Title'])]
# let's look at the duplicates of the first value
books_raw.loc[books_raw['Book-Title'] == 'Angels &amp; Demons']<jupyter_output><empty_output><jupyter_text>These are different editions of the same title, so they are not duplicates.  Any empty title names?<jupyter_code>books_raw.loc[books_raw['Book-Title'].isnull(),'Book-Title'].count()<jupyter_output><empty_output><jupyter_text>No. We're all good!  Any empty or zero years?<jupyter_code>books_raw.loc[books_raw['Year-Of-Publication'].isnull(),'Year-Of-Publication'].count()<jupyter_output><empty_output><jupyter_text>No empty Year-Of-Publication<jupyter_code>books_raw.loc[books_raw['Year-Of-Publication']=='0','Year-Of-Publication'].count()<jupyter_output><empty_output><jupyter_text>There are at least 4618 books without a valid Year-Of-Publication.  Preprocess Ratings.csv  Load data<jupyter_code>ratings_raw = pd.read_csv ('dataset1/Ratings.csv', dtype={
    'User-ID':str, # some values are not integers
    'ISBN':str, # some values are not integers
    'Book-Rating':int
})<jupyter_output><empty_output><jupyter_text>Explore Ratings_raw dataframe Contains the book rating information. Ratings (Book-Rating) are either explicit, expressed on a scale from 1-10 (higher values denoting higher appreciation), or implicit, expressed by 0.<jupyter_code>list(ratings_raw.columns)
ratings_raw.shape
ratings_raw.count()
ratings_raw.dtypes<jupyter_output><empty_output><jupyter_text>Analyse User-ID  Any missing values?<jupyter_code>ratings_raw.loc[ratings_raw['User-ID'].isnull(),'User-ID'].count()<jupyter_output><empty_output><jupyter_text>Are all User-ID numbers?<jupyter_code>ID_error = pd.to_numeric(ratings_raw['User-ID'], errors='coerce').isna()
ratings_raw.loc[ID_error]<jupyter_output><empty_output><jupyter_text>All are numbers, so we will convert them to integers.<jupyter_code>ratings_raw["User-ID"] = pd.to_numeric(ratings_raw["User-ID"])
ratings_raw['User-ID'].dtypes<jupyter_output><empty_output><jupyter_text>How many unique User-ID?<jupyter_code>pd.unique(pd.Series(ratings_raw['User-ID'])).shape[0]<jupyter_output><empty_output><jupyter_text>Any Unique-ID with value zero?<jupyter_code># User-ID are stored as strings
(ratings_raw['User-ID'] == "0").sum()<jupyter_output><empty_output><jupyter_text>No ID zero present  Analyse Book-Rating<jupyter_code>sorted(set(ratings_raw['Book-Rating'].tolist()))<jupyter_output><empty_output><jupyter_text>The zeros are inferred values. Let's see how many are there.<jupyter_code>ratings_raw['Book-Rating'].tolist().count(0)
hist = ratings_raw.hist(bins=11,column='Book-Rating')<jupyter_output><empty_output><jupyter_text>Preprocess Users.csv  Load data<jupyter_code>users_raw = pd.read_csv ('dataset1/Users.csv', dtype={
    'User-ID':int,
    'Location':str,
    'Age':str # some values are not integers
})<jupyter_output><empty_output><jupyter_text>Explore dataframe Contains the users. Note that user IDs (User-ID) have been anonymized and map to integers. Demographic data is provided (Location, Age) if available. Otherwise, these fields contain NULL-values.<jupyter_code>list(users_raw.columns)
users_raw.shape
users_raw.count()
users_raw.dtypes<jupyter_output><empty_output><jupyter_text>Analyse User-ID  Any missing values?<jupyter_code>users_raw.loc[ratings_raw['User-ID'].isnull(),'User-ID'].count()<jupyter_output><empty_output><jupyter_text>Great! No missing IDs.  Analyse Location<jupyter_code>users_raw['Location']
users_raw.loc[users_raw['Location'].isnull(),'Location'].count()<jupyter_output><empty_output><jupyter_text>Great! No missing locations.  Age   Missing values<jupyter_code>users_raw['Age'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Thera re many users missing age information.<jupyter_code>users_raw['Age'] = pd.to_numeric(users_raw['Age'], errors='coerce')<jupyter_output><empty_output><jupyter_text>Let's replace missing age with the average reader age<jupyter_code>age_mean = users_raw['Age'].mean()
age_mean
users_raw['Age'].fillna(age_mean)<jupyter_output><empty_output><jupyter_text>Remove hyphens from column names<jupyter_code>books_raw.columns
books_raw.rename(columns={'Book-Title':'Book_Title',
                      'Book-Author':'Book_Author',
                      'Year-Of-Publication':'Year_Of_Publication',
                      'Image-URL-S':'Image_URL_S'}, inplace=True)
ratings_raw.columns
ratings_raw.rename(columns={'User-ID':'User_ID',
                        'Book-Rating':'Book_Rating'}, inplace=True)
users_raw.columns
users_raw.rename(columns={'User-ID':'User_ID'}, inplace=True)<jupyter_output><empty_output><jupyter_text>Export preprocessed dataframes<jupyter_code>books_raw.to_pickle("./pickle_files/d1_books.pkl")
ratings_raw.to_pickle("./pickle_files/d1_ratings.pkl")
users_raw.to_pickle("./pickle_files/d1_users.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12548236859614106, 'fraction_numerical': 0.0027944111776447107, 'mean_word_length': 6.213051823416507, 'pattern_counts': {'":': 0, '<': 117, '<?xml version=': 0, '>': 117, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '532538', 'license': 'CC0-1.0', 'n_tokens_mistral': 2570, 'n_tokens_neox': 2483, 'n_words': 465, 'path': 'Book-Recommender-1_Simple Recommender_Data_Prep.ipynb', 'repo_name': 'ptavaressilva/book_recommender', 'size': '7515'}