<jupyter_start><jupyter_text>Book RecommendationsThe goal of the project is to build a recommender system that suggests relevant books based on person's interests. Data AnalysisThis Jupyter notebook includes:1. Information about the dataset2. Downloading the ratings data (ratings.csv)3. Splitting the ratings data into training and test sets4. Data Exploration  1. Information about the datasetgoodbooks-10k https://github.com/zygmuntz/goodbooks-10kDataset contains six million ratings for ten thousand most popular (with most ratings) books.**ratings.csv**: ratings sorted by time. Ratings go from one to five. Both book IDs and user IDs are contiguous. For books, they are 1-10000; for users, 1-53424.**books.csv**: metadata for each book (goodreads IDs, authors, title, average rating, etc.).Each book may have many editions. goodreads_book_id and best_book_id generally point to the most popular edition of a given book, while goodreads work_id refers to the book in the abstract sense.Note: book_id in ratings.csv maps to work_id, not to goodreads_book_id, meaning that ratings for different editions are aggregated.  2. Download the ratings data (from ratings.csv)<jupyter_code>import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
# Read the file with ratings into a pandas DataFrame

ratings = pd.read_csv('F:\\Downloads\\data_books\\goodbooks-10k\\ratings.csv')
ratings.head()<jupyter_output><empty_output><jupyter_text>Information about the DataFrame<jupyter_code>ratings.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5976479 entries, 0 to 5976478
Data columns (total 3 columns):
 #   Column   Dtype
---  ------   -----
 0   user_id  int64
 1   book_id  int64
 2   rating   int64
dtypes: int64(3)
memory usage: 136.8 MB<jupyter_text>Check if there are missing values:<jupyter_code>ratings.isnull().sum()

# There are no missing values<jupyter_output><empty_output><jupyter_text>Number of unique users and books in the dataset<jupyter_code># Number of unique users

num_users = len(ratings['user_id'].unique())
num_users
# Number of unique books

num_books = len(ratings['book_id'].unique())
num_books<jupyter_output><empty_output><jupyter_text>Rating values and their proportions in the dataset<jupyter_code># Rating values: 1, 2, 3, 4, 5

ratings['rating'].unique()
# rating proportions
# ratings 1 and 2 are very rare

ratings['rating'].value_counts() / len(ratings)<jupyter_output><empty_output><jupyter_text>3. Split the ratings data into training and test setsKeep the proportions of rating values.<jupyter_code>from sklearn.model_selection import train_test_split

train_set, test_set = train_test_split(ratings, test_size=0.15, random_state=42, stratify=ratings['rating'])

print(train_set.shape)
print(test_set.shape)<jupyter_output>(5080007, 3)
(896472, 3)<jupyter_text>Save the training and test sets as csv files:<jupyter_code>train_set.to_csv('../data/ratings_train.csv', index=False)
test_set.to_csv('../data/ratings_test.csv', index=False)<jupyter_output><empty_output><jupyter_text>4. Data Exploration  Training Set Info<jupyter_code>train_set.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 5080007 entries, 660939 to 349688
Data columns (total 3 columns):
 #   Column   Dtype
---  ------   -----
 0   user_id  int64
 1   book_id  int64
 2   rating   int64
dtypes: int64(3)
memory usage: 155.0 MB<jupyter_text>The training set includes all users and books from the original dataset:<jupyter_code>num_users_train = len(train_set['user_id'].unique())
num_users_train
num_books_train = len(train_set['book_id'].unique())
num_books_train<jupyter_output><empty_output><jupyter_text>RatingsSummary statistics:<jupyter_code>train_set['rating'].describe()

# average rating is about 3.92<jupyter_output><empty_output><jupyter_text>Plot the distribution of ratings:<jupyter_code># Higher ratings are more common

sns.countplot(data=train_set, x='rating');
# Percentage of available ratings (out of all possible ratings) = 0.95%

ratings_available_pt = len(train_set) / (num_users_train * num_books_train) * 100
ratings_available_pt<jupyter_output><empty_output><jupyter_text>Ratings given by each user<jupyter_code># Number of ratings and average rating per user

ratings_per_user = train_set.groupby('user_id')['rating'].agg(['count', 'mean'])
ratings_per_user
ratings_per_user.describe()

# min number of ratings per user = 15
# max number = 176<jupyter_output><empty_output><jupyter_text>Plot histograms:<jupyter_code># Number of ratings per user
# Looks close to a normal distribution

ratings_per_user['count'].hist(bins=30);
# Average rating per user
# The distribution is shifted and centered around 3.9

ratings_per_user['mean'].hist(bins=30);<jupyter_output><empty_output><jupyter_text>Ratings per book<jupyter_code># Number of ratings and average rating for each book

ratings_per_book = train_set.groupby('book_id')['rating'].agg(['count', 'mean'])
ratings_per_book.describe()<jupyter_output><empty_output><jupyter_text>- Min number of ratings = 8- 75% of books have <= 431 ratings- Some books have very high amount of ratings: max number of ratings = 19,342 Plot histograms:<jupyter_code># Number of ratings per book

ratings_per_book['count'].hist(bins=50);
# A closer look at 75% of the books (number of ratings <= 431)

books_75_percent = ratings_per_book[ratings_per_book['count'] <= 431]
books_75_percent['count'].hist(bins=50);
# Average rating for a book

ratings_per_book['mean'].hist(bins=50);<jupyter_output><empty_output><jupyter_text>Scatter plot: Average Rating vs. Number of Ratings for that book<jupyter_code>sns.scatterplot(x=ratings_per_book['mean'], y=ratings_per_book['count']);<jupyter_output><empty_output><jupyter_text>Most books with number of ratings greater than 10,000 got average ratings in the range from 3.5 to 4.5. Probably these are famous books and many people liked them.Check how many books have few ratings and what average rating they got:<jupyter_code>less_than_30_ratings = ratings_per_book[ratings_per_book['count'] < 30]
less_than_30_ratings

# just a few books<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10063301412108423, 'fraction_numerical': 0.030352215549423796, 'mean_word_length': 6.469090909090909, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 68, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '392829', 'license': 'MIT', 'n_tokens_mistral': 2154, 'n_tokens_neox': 1979, 'n_words': 575, 'path': 'notebooks/data_analysis_ratings.ipynb', 'repo_name': 'izlata/book_recommender_system', 'size': '6161'}