<jupyter_start><jupyter_text>Book-Crossing Recommendation System> Book recommender system on book crossing dataset using surprise SVD and NMF models- toc: true- badges: true- comments: true- categories: [Surprise, SVD, NMF, Book]- author: "<NAME>"- image:  Setup<jupyter_code>!pip install -q git+https://github.com/sparsh-ai/recochef.git
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import pickle

from sklearn import model_selection

import warnings
warnings.filterwarnings('ignore')

plt.style.use('seaborn-white')
plt.rcParams.update({'font.size': 15})
%matplotlib inline
from recochef.datasets.bookcrossing import BookCrossing<jupyter_output><empty_output><jupyter_text>Load the dataset<jupyter_code>bookcrossing = BookCrossing()
users = bookcrossing.load_users()
books = bookcrossing.load_items()
book_ratings = bookcrossing.load_interactions()
users.head()
books.head()
book_ratings.head()
print(f'Users: {len(users)}\nBooks: {len(books)}\nRatings: {len(book_ratings)}')<jupyter_output>Users: 278858
Books: 271360
Ratings: 1149780<jupyter_text>EDA and Data cleaning  Users<jupyter_code>users.describe(include='all').T<jupyter_output><empty_output><jupyter_text>The age range goes from 0 to 244 years old! Obviously this cannot be correct; I'll set all ages less than 5 and older than 100 to NaN to try keep them realistic.<jupyter_code>users.loc[(users.AGE<5) | (users.AGE>100), 'AGE'] = np.nan
u = users.AGE.value_counts().sort_index()
plt.figure(figsize=(20, 10))
plt.bar(u.index, u.values)
plt.xlabel('Age')
plt.ylabel('counts')
plt.show()<jupyter_output><empty_output><jupyter_text>Next, can we expand the 'Location' field to break it up into 'City', 'State', and 'Country'. > Note: Used Pandas Series.str.split method as it has an 'expand' parameter which can handle None cases<jupyter_code>user_location_expanded = users.LOCATION.str.split(',', 2, expand=True)
user_location_expanded.columns = ['CITY', 'STATE', 'COUNTRY']
users = users.join(user_location_expanded)
users.COUNTRY.replace('', np.nan, inplace=True)
users.drop(columns=['LOCATION'], inplace=True)
users.head()<jupyter_output><empty_output><jupyter_text>Books<jupyter_code>books.head(2)
books.describe(include='all').T
# Convert years to float
books.YEAR = pd.to_numeric(books.YEAR, errors='coerce')
# Replace all years of zero with NaN
books.YEAR.replace(0, np.nan, inplace=True)
yr = books.YEAR.value_counts().sort_index()
yr = yr.where(yr>5) # filter out counts less than 5
plt.figure(figsize=(20, 10))
plt.bar(yr.index, yr.values)
plt.xlabel('Year of Publication')
plt.ylabel('counts')
plt.show()<jupyter_output><empty_output><jupyter_text>Note that in the plot above we filtered out counts less than 5, as there are a few books in the dataset with publication years in the 1300s, and a few in the future (?!). The plot above show the general trend that more recent books are much more frequent.Let's take a look at some of those 'outlier' books. Maybe we'll even keep them as a separate dataset so we can filter them out if we need to later in the analysis. We'll leave them in for now, and then figure out how to handle them once we have more info later on.<jupyter_code>historical_books = books[books.YEAR<1900] # create df of old books
books_from_the_future = books[books.YEAR>2018] # create df of books with publication yrs in the future!

hist_books_mini = historical_books[['TITLE', 'YEAR']]
future_books_mini = books_from_the_future[['TITLE', 'YEAR']]
print(f'Historical books:\n{hist_books_mini}')
print('\n')
print(f'Future books:\n{future_books_mini}')<jupyter_output>Historical books:
                                                    TITLE    YEAR
171817  Complete Works 10 Volumes [2,6,7,8,9] (Notable...  1806.0
227531                   Dalan-i bihisht (Dastan-i Irani)  1378.0
245607  <NAME>, Free Quaker (2 Volumes (BCL1-PS Am...  1897.0
253750                                       Tasht-i khun  1376.0


Future books:
                                                    TITLE    YEAR
30010                                          The Royals  2020.0
37487   MY TEACHER FRIED MY BRAINS (RACK SIZE) (MY TEA...  2030.0
55676   MY TEACHER FLUNKED THE PLANET (RACK SIZE) (MY ...  2030.0
78168                                    Crossing America  2030.0
80264   Alice's Adventures in Wonderland and Through t...  2050.0
92156                     Edgar Allen Poe Collected Poems  2020.0
97826       Outline of European Architecture (Pelican S.)  2050.0
116053                       Three Plays of Eugene Oneill  2038.0
118294        Das groÃ?Â?e BÃ?Â¶se- MÃ?Â¤d[...]<jupyter_text>I think we can probably omit the 'historical_books' as they may potentially skew the model and do not seem to have much relevance to the wider userbase.Some of the 'future' books actually appear to be errors (e.g. Alice in Wonderland, Edgar Allen Poe, etc.)... Perhaps they were supposed to be e.g. 1950 instead of 2050? However, instead of investigating this further, since there are <20 books here I will simply remove them from the 'books' table.<jupyter_code>print(f'Length of books dataset before removal: {len(books)}')
books = books.loc[~(books.ITEMID.isin(historical_books.ITEMID))] # remove historical books
books = books.loc[~(books.ITEMID.isin(books_from_the_future.ITEMID))] # remove historical books
print(f'Length of books dataset after removal: {len(books)}')<jupyter_output>Length of books dataset before removal: 271360
Length of books dataset after removal: 271339<jupyter_text>We clean up the ampersand formatting in the Publisher field.<jupyter_code>books.PUBLISHER = books.PUBLISHER.str.replace('&amp', '&', regex=False)
books.head()<jupyter_output><empty_output><jupyter_text>Check that there are no duplicated book entries.<jupyter_code>uniq_books = books.ITEMID.nunique()
all_books = books.ITEMID.count()
print(f'No. of unique books: {uniq_books} | All book entries: {all_books}')<jupyter_output>No. of unique books: 271339 | All book entries: 271339<jupyter_text>Let's look at the most frequent Publishing houses in the dataset.<jupyter_code>top_publishers = books.PUBLISHER.value_counts()[:10]
print(f'The 10 publishers with the most entries in the books table are:\n{top_publishers}')<jupyter_output>The 10 publishers with the most entries in the books table are:
Harlequin                   7535
Silhouette                  4220
Pocket                      3905
Ballantine Books            3783
Bantam Books                3646
Scholastic                  3159
Simon &; Schuster           2969
Penguin Books               2844
Berkley Publishing Group    2771
Warner Books                2727
Name: PUBLISHER, dtype: int64<jupyter_text>What about authors with the most entries?<jupyter_code>top_authors = books.AUTHOR.value_counts()[:10]
print(f'The 10 authors with the most entries in the books table are:\n{top_authors}')<jupyter_output>The 10 authors with the most entries in the books table are:
<NAME>        632
<NAME>    567
<NAME>           524
<NAME>          423
<NAME>          373
<NAME>        372
<NAME>           330
<NAME>           315
<NAME>       307
<NAME>        302
Name: AUTHOR, dtype: int64<jupyter_text>We should search for empty or NaN values in these fields too.<jupyter_code>empty_string_publisher = books[books.PUBLISHER == ''].PUBLISHER.count()
nan_publisher = books.PUBLISHER.isnull().sum()
print(f'There are {empty_string_publisher} entries with empty strings, and {nan_publisher} NaN entries in the Publisher field')<jupyter_output>There are 0 entries with empty strings, and 2 NaN entries in the Publisher field<jupyter_text>Great - no empty strings in the Publisher field, and only 2 NaNs.<jupyter_code>empty_string_author = books[books.AUTHOR == ''].AUTHOR.count()
nan_author = books.AUTHOR.isnull().sum()
print(f'There are {empty_string_author} entries with empty strings, and {nan_author} NaN entries in the Author field')<jupyter_output>There are 0 entries with empty strings, and 1 NaN entries in the Author field<jupyter_text>Cool, only 1 NaN in the Author field.Let's look at the titles.<jupyter_code>top_titles = books.TITLE.value_counts()[:10]
print(f'The 10 book titles with the most entries in the books table are:\n{top_titles}')<jupyter_output>The 10 book titles with the most entries in the books table are:
Selected Poems                    27
Little Women                      24
Wuthering Heights                 21
The Secret Garden                 20
Adventures of Huckleberry Finn    20
Dracula                           20
<NAME>                         19
The Night Before Christmas        18
Pride and Prejudice               18
Great Expectations                17
Name: TITLE, dtype: int64<jupyter_text>This is actually quite an important observation. Although all of the ISBN entries are *unique* in the 'books' dataframe, different *forms* of the **same** book will have different ISBNs - i.e. paperback, e-book, etc. Therefore, we can see that some books have multiple ISBN entries (e.g. Jane Eyre has 19 different ISBNs, each corresponding to a different version of the book).Let's take a look at, for example, the entries for '<NAME>'.<jupyter_code>books[books.TITLE=='<NAME>']<jupyter_output><empty_output><jupyter_text>It looks like each ISBN assigned to the book '<NAME>' has different Publisher and Year of Publication values also.It might be more useful for our model if we simplified this to give each book a *unique* identifier, independent of the book format, as our recommendations will be for a book, not a specific version of a book. Therefore, all values in the Jane Eyre example above would stay the same, except all of the Jane Eyre entries would additionally be assigned a *unique ISBN* code as a new field.**Will create this more unique identifier under the field name 'UNIQUE_ITEMIDS'. Note that entries with only a single ISBN number will be left the same. However, will need to do this after joining to the other tables in the dataset, as some ISBNs in the 'book-rating' table may be removed if done prior.**  Interactions<jupyter_code>book_ratings.head()
book_ratings.describe(include='all').T
book_ratings.dtypes<jupyter_output><empty_output><jupyter_text>The data types already look good. Remember that the ISBN numbers may contain letters, and so should be left as strings.Which users contribute the most ratings?<jupyter_code>super_users = book_ratings.groupby('USERID').ITEMID.count().sort_values(ascending=False)
print(f'The 20 users with the most ratings:\n{super_users[:20]}')<jupyter_output>The 20 users with the most ratings:
USERID
11676     13602
198711     7550
153662     6109
98391      5891
35859      5850
212898     4785
278418     4533
76352      3367
110973     3100
235105     3067
230522     2991
16795      2948
234623     2674
36836      2529
52584      2512
245963     2507
204864     2504
55492      2459
185233     2448
171118     2421
Name: ITEMID, dtype: int64<jupyter_text>Wow! User \11676 has almost twice as many ratings as the next highest user! All of the top 20 users have thousands of ratings, which seems like a lot, although maybe I'm just a slow reader...Let's see how they are distributed.<jupyter_code># user distribution - users with more than 50 ratings removed
user_hist = super_users.where(super_users<50)
user_hist.hist(bins=30)
plt.xlabel('No. of ratings')
plt.ylabel('count')
plt.show()<jupyter_output><empty_output><jupyter_text>It looks like **_by far_** the most frequent events are users with only 1 or 2 rating entries. We can see that the 'super users' with thousands of ratings are significant outliers.This becomes clear if we make the same histogram with a cutoff for users with a minimum of 1000 ratings.<jupyter_code># only users with more than 1000 ratings
super_user_hist = super_users.where(super_users>1000)
super_user_hist.hist(bins=30)
plt.xlabel('No. of ratings (min. 1000)')
plt.ylabel('count')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's see what the distribution of **ratings** looks like.<jupyter_code>rtg = book_ratings.RATING.value_counts().sort_index()

plt.figure(figsize=(10, 5))
plt.bar(rtg.index, rtg.values)
plt.xlabel('Rating')
plt.ylabel('counts')
plt.show()<jupyter_output><empty_output><jupyter_text>Seems like most of the entries have a rating of zero!After doing some research on the internet regarding this (and similar) datasets, it appears that the rating scale is actually from 1 to 10, and a 0 indicates an 'implicit' rather than an 'explicit' rating. An implicit rating represents an interaction (may be positive or negative) between the user and the item. Implicit interactions usually need to be handled differently from explicit ones.For the modeling step we'll only be looking at *explicit* ratings, and so the 0 rating entry rows will be removed.<jupyter_code>print(f'Size of book_ratings before removing zero ratings: {len(book_ratings)}')
book_ratings = book_ratings[book_ratings.RATING != 0]
print(f'Size of book_ratings after removing zero ratings: {len(book_ratings)}')<jupyter_output>Size of book_ratings before removing zero ratings: 1149780
Size of book_ratings after removing zero ratings: 433671<jupyter_text>By removing the implicit ratings we have reduced our sample size by more than half.Let's look at how the ratings are distributed again.<jupyter_code>rtg = book_ratings.RATING.value_counts().sort_index()

plt.figure(figsize=(10, 5))
plt.bar(rtg.index, rtg.values)
plt.xlabel('Rating')
plt.ylabel('counts')
plt.show()<jupyter_output><empty_output><jupyter_text>This is much more clear! Now we can see that 8 is the most frequent rating, while users tend to give ratings > 5, with very few low ratings given.  Merge First, we'll join the 'books' table to the 'book_ratings' table on the ISBN field.<jupyter_code>print(f'Books table size: {len(books)}')
print(f'Ratings table size: {len(book_ratings)}')
books_with_ratings = book_ratings.join(books.set_index('ITEMID'), on='ITEMID')
print(f'New table size: {len(books_with_ratings)}')<jupyter_output>Books table size: 271339
Ratings table size: 433671
New table size: 433671<jupyter_text>Let's take a look at the new table.<jupyter_code>books_with_ratings.head()
print(f'There are {books_with_ratings.TITLE.isnull().sum()} books with no title/author information.')
print(f'This represents {len(books_with_ratings)/books_with_ratings.TITLE.isnull().sum():.2f}% of the ratings dataset.')<jupyter_output>There are 49861 books with no title/author information.
This represents 8.70% of the ratings dataset.<jupyter_text>There seems to be quite a few ISBNs in the ratings table that did not match an ISBN in the books table, almost 9% of all entries!There isn't really anything we can do about that, but we should really remove them from the dataset as we won't be able to access the title of the book to make a recommendation even if the model can use them.<jupyter_code>books_with_ratings.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 433671 entries, 1 to 1149779
Data columns (total 10 columns):
 #   Column     Non-Null Count   Dtype  
---  ------     --------------   -----  
 0   USERID     433671 non-null  int64  
 1   ITEMID     433671 non-null  object 
 2   RATING     433671 non-null  int64  
 3   TITLE      383810 non-null  object 
 4   AUTHOR     383809 non-null  object 
 5   YEAR       378026 non-null  float64
 6   PUBLISHER  383808 non-null  object 
 7   URLSMALL   383810 non-null  object 
 8   URLMEDIUM  383810 non-null  object 
 9   URLLARGE   383809 non-null  object 
dtypes: float64(1), int64(2), object(7)
memory usage: 36.4+ MB<jupyter_text>It looks like the ```year_of_publication``` field contains the most NaN entries, while ```USERID```, ```isbn```, and ```book_rating``` are full. The ```book_title```, ```book_author```, and ```publisher``` fields contain approximately the same number of missing entries.We'll choose to remove rows for which the ```book_title``` is empty, as this is the most crucial piece of data needed to identify the book.<jupyter_code>books_with_ratings.dropna(subset=['TITLE'], inplace=True) # remove rows with missing title/author data<jupyter_output><empty_output><jupyter_text>Let's see which books have the highest **cumulative** book rating values.<jupyter_code>cm_rtg = books_with_ratings.groupby('TITLE').RATING.sum()
cm_rtg = cm_rtg.sort_values(ascending=False)[:10]
idx = cm_rtg.index.tolist() # Get sorted book titles
vals = cm_rtg.values.tolist() # Get corresponding cm_rtg values

plt.figure(figsize=(10, 5))
plt.bar(range(len(idx)), vals)
plt.xticks(range(len(idx)), idx, rotation='vertical')
plt.ylabel('cumulative rating score')
plt.show()<jupyter_output><empty_output><jupyter_text>This seems about right as it combines the total number of ratings with the score given, so these are all really popular book titles.What about the highest **average ratings** (with a minimum of at least 50 ratings recieved)?<jupyter_code>cutoff = books_with_ratings.TITLE.value_counts()
mean_rtg = books_with_ratings[books_with_ratings.TITLE.isin(cutoff[cutoff>50].index)].groupby('TITLE')['RATING'].mean()
mean_rtg.sort_values(ascending=False)[:10] # show only top 10<jupyter_output><empty_output><jupyter_text>This looks perfectly reasonable. The Harry Potter and Lord of the Rings books rate extremely highly, as expected.How about the **lowest-rated** books?<jupyter_code>mean_rtg.sort_values(ascending=False)[-10:] # bottom 10 only<jupyter_output><empty_output><jupyter_text>Seems like the *lowest average* rating in the dataset is only a 4.39 - and all the rest of the books have average ratings higher than 5.I haven't heard of any of these books, so I can't really comment on if they seem correct here.**Now I'd like to tackle the challenge of the same book potentially having multiple ISBN numbers (for the different formats it is available in). We should clean that up here before we add the 'user' table.**  Single ISBN per bookRestrict books to a "single ISBN per book" (regardless of format) Let's look again at the book titles which have the most associated ISBN numbers.<jupyter_code>books_with_ratings.groupby('TITLE').ITEMID.nunique().sort_values(ascending=False)[:10]
multiple_isbns = books_with_ratings.groupby('TITLE').ITEMID.nunique()
multiple_isbns.value_counts()<jupyter_output><empty_output><jupyter_text>We can see that the vast majority of books have less only 1 associated ISBN number, however quite a few multiple ISBNs. We want to create a ```UNIQUE_ITEMIDS``` such that a single book will only have 1 identifier when fed to the recommendation model.<jupyter_code>has_mult_isbns = multiple_isbns.where(multiple_isbns>1)
has_mult_isbns.dropna(inplace=True) # remove NaNs, which in this case is books with a single ISBN number
print(f'There are {len(has_mult_isbns)} book titles with multiple ISBN numbers which we will try to re-assign to a unique identifier')
# Check to see that our friend <NAME> still has multiple ISBN values
has_mult_isbns['<NAME>']<jupyter_output><empty_output><jupyter_text>**Note:** Created the dictionary below and pickled it, just need to load it again (or run it if the first time on a new system).<jupyter_code># Create dictionary for books with multiple isbns
def make_isbn_dict(df):
    title_isbn_dict = {}
    for title in has_mult_isbns.index:
        isbn_series = df.loc[df.TITLE==title].ITEMID.unique() # returns only the unique ISBNs
        title_isbn_dict[title] = isbn_series.tolist()
    return title_isbn_dict

%time dict_UNIQUE_ITEMIDS = make_isbn_dict(books_with_ratings)

# As the loop takes a while to run (8 min on the full dataset), pickle this dict for future use
with open('multiple_isbn_dict.pickle', 'wb') as handle:
    pickle.dump(dict_UNIQUE_ITEMIDS, handle, protocol=pickle.HIGHEST_PROTOCOL)

# LOAD isbn_dict back into namespace
with open('multiple_isbn_dict.pickle', 'rb') as handle:
    multiple_isbn_dict = pickle.load(handle)
print(f'There are now {len(multiple_isbn_dict)} books in the ISBN dictionary that have multiple ISBN numbers')<jupyter_output>There are now 10231 books in the ISBN dictionary that have multiple ISBN numbers<jupyter_text>Let's take a quick look in the dict we just created for the 'Jane Eyre' entry - it should contain a list of 14 ISBN numbers.<jupyter_code>print(f'Length of Jane Eyre dict entry: {len(multiple_isbn_dict["Jane Eyre"])}\n')
multiple_isbn_dict['Jane Eyre']<jupyter_output>Length of Jane Eyre dict entry: 14<jupyter_text>Looking good!As I don't really know what each of the different ISBN numbers refers to (from what I understand the code actually signifies various things including publisher, year, type of print, etc, but decoding this is outside the scope of this analysis), I'll just select the **first** ISBN number that appears in the list of values to set as our ```UNIQUE_ITEMIDS``` for that particular book._**Note**_: ISBN numbers are currently 13 digits long, but used to be 10. Any ISBN that isn't 10 or 13 digits long is probably an error that should be handled somehow. Any that are 9 digits long might actually be SBN numbers (pre-1970), and can be converted into ISBN's by just pre-fixing with a zero.<jupyter_code># Add 'UNIQUE_ITEMIDS' column to 'books_with_ratings' dataframe that includes the first ISBN if multiple ISBNS,
# or just the ISBN if only 1 ISBN present anyway.
def add_UNIQUE_ITEMIDS_col(df):
    df['UNIQUE_ITEMIDS'] = df.apply(lambda row: multiple_isbn_dict[row.TITLE][0] if row.TITLE in multiple_isbn_dict.keys() else row.ITEMID, axis=1)
    return df

%time books_with_ratings = add_UNIQUE_ITEMIDS_col(books_with_ratings)
books_with_ratings.head()<jupyter_output><empty_output><jupyter_text>The table now includes our ```UNIQUE_ITEMIDS``` field.Let's check to see if the '<NAME>' entries have been assigned the ISBN '1590071212', which was the first val in the dictionary for this title.<jupyter_code>books_with_ratings[books_with_ratings.TITLE=='<NAME>'].head()<jupyter_output><empty_output><jupyter_text>Great! Seems to have worked well.We won't replace the original ISBN column with the 'UNIQUE_ITEMIDS' column, but just note that the recommendation model should be based on the 'UNIQUE_ITEMIDS' field.  Remove Small and Large book-cover URL columns<jupyter_code>books_users_ratings.drop(['URLSMALL', 'URLLARGE'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Join the 'users' table on the 'USERID' field<jupyter_code>print(f'Books+Ratings table size: {len(books_with_ratings)}')
print(f'Users table size: {len(users)}')
books_users_ratings = books_with_ratings.join(users.set_index('USERID'), on='USERID')
print(f'New "books_users_ratings" table size: {len(books_users_ratings)}')<jupyter_output>Books+Ratings table size: 383810
Users table size: 278858
New "books_users_ratings" table size: 383810<jupyter_text>Inspect the new table.<jupyter_code>books_users_ratings.head()
books_users_ratings.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 383810 entries, 1 to 1149778
Data columns (total 13 columns):
 #   Column          Non-Null Count   Dtype  
---  ------          --------------   -----  
 0   USERID          383810 non-null  int64  
 1   ITEMID          383810 non-null  object 
 2   RATING          383810 non-null  int64  
 3   TITLE           383810 non-null  object 
 4   AUTHOR          383809 non-null  object 
 5   YEAR            378026 non-null  float64
 6   PUBLISHER       383808 non-null  object 
 7   URLMEDIUM       383810 non-null  object 
 8   UNIQUE_ITEMIDS  383810 non-null  object 
 9   AGE             267996 non-null  float64
 10  CITY            383810 non-null  object 
 11  STATE           383810 non-null  object 
 12  COUNTRY         373330 non-null  object 
dtypes: float64(2), int64(2), object(9)
memory usage: 41.0+ MB<jupyter_text>There are a few missing ```age```, ```year_of_publication```, ```publisher```, and ```country``` entries, but the primary fields of ```USERID```, ```UNIQUE_ITEMIDS```, and ```book_rating``` are all full, which is good.In terms of the data types, ```USERID``` and ```book_rating``` are integers, while the ```UNIQUE_ITEMIDS``` are strings (which is expected as the ISBN numbers may also contain letters).<jupyter_code>books_users_ratings.shape<jupyter_output><empty_output><jupyter_text>Recommender modelCollaborative filtering use similarities of the 'user' and 'item' fields, with values of 'rating' predicted based on either user-item, or item-item similarity: - Item-Item CF: "Users who liked this item also liked..." - User-Item CF: "Users who are similar to you also liked..." In both cases, we need to create a user-item matrix built from the entire dataset. We'll create a matrix for each of the training and testing sets, with the users as the rows, the books as the columns, and the rating as the matrix value. Note that this will be a very sparse matrix, as not every user will have watched every movie etc.We'll first create a new dataframe that contains only the relevant columns (```USERID```, ```UNIQUE_ITEMIDS```, and ```book_rating```).<jupyter_code>user_item_rating = books_users_ratings[['USERID', 'UNIQUE_ITEMIDS', 'RATING']]
user_item_rating.head()<jupyter_output><empty_output><jupyter_text>We know what the distribution of ratings should look like (as we plotted it earlier) - let's plot it again on this new dataframe to just quickly check that it looks right.<jupyter_code>rtg = user_item_rating.RATING.value_counts().sort_index()

plt.figure(figsize=(10, 5))
plt.bar(rtg.index, rtg.values)
plt.xlabel('Rating')
plt.ylabel('counts')
plt.show()<jupyter_output><empty_output><jupyter_text>Looks perfect! Continue.  Using ```sklearn``` to generate training and testing subsets<jupyter_code>train_data, test_data = model_selection.train_test_split(user_item_rating, test_size=0.20)
print(f'Training set size: {len(train_data)}')
print(f'Testing set size: {len(test_data)}')
print(f'Test set is {(len(test_data)/(len(train_data)+len(test_data))*100):.0f}% of the full dataset.')<jupyter_output>Training set size: 307048
Testing set size: 76762
Test set is 20% of the full dataset.<jupyter_text>Map the ```USERID``` and ```UNIQUE_ITEMIDS``` fields to sequential integers for matrix processing<jupyter_code>### TRAINING SET
# Get int mapping for USERID
u_unique_train = train_data.USERID.unique()  # create a 'set' (i.e. all unique) list of vals
train_data_user2idx = {o:i for i, o in enumerate(u_unique_train)}
# Get int mapping for UNIQUE_ITEMIDS
b_unique_train = train_data.UNIQUE_ITEMIDS.unique()  # create a 'set' (i.e. all unique) list of vals
train_data_book2idx = {o:i for i, o in enumerate(b_unique_train)}

### TESTING SET
# Get int mapping for USERID
u_unique_test = test_data.USERID.unique()  # create a 'set' (i.e. all unique) list of vals
test_data_user2idx = {o:i for i, o in enumerate(u_unique_test)}
# Get int mapping for UNIQUE_ITEMIDS
b_unique_test = test_data.UNIQUE_ITEMIDS.unique()  # create a 'set' (i.e. all unique) list of vals
test_data_book2idx = {o:i for i, o in enumerate(b_unique_test)}
### TRAINING SET
train_data['USER_UNIQUE'] = train_data['USERID'].map(train_data_user2idx)
train_data['ITEM_UNIQUE'] = train_data['UNIQUE_ITEMIDS'].map(train_data_book2idx)

### TESTING SET
test_data['USER_UNIQUE'] = test_data['USERID'].map(test_data_user2idx)
test_data['ITEM_UNIQUE'] = test_data['UNIQUE_ITEMIDS'].map(test_data_book2idx)

### Convert back to 3-column df
train_data = train_data[['USER_UNIQUE', 'ITEM_UNIQUE', 'RATING']]
test_data = test_data[['USER_UNIQUE', 'ITEM_UNIQUE', 'RATING']]
train_data.tail()
train_data.dtypes<jupyter_output><empty_output><jupyter_text>This dataset is now ready to be processed via a collaborative filtering approach!**Note:** When we need to identify the user or book from the model we'll need to refer back to the ```train_data_user2idx``` and ```train_data_book2idx``` dictionaries to locate the ```USERID``` and ```UNIQUE_ITEMIDS```, respectively.<jupyter_code>### TRAINING SET
# Create user-item matrices
n_users = train_data['USER_UNIQUE'].nunique()
n_books = train_data['ITEM_UNIQUE'].nunique()

# First, create an empty matrix of size USERS x BOOKS (this speeds up the later steps)
train_matrix = np.zeros((n_users, n_books))

# Then, add the appropriate vals to the matrix by extracting them from the df with itertuples
for entry in train_data.itertuples(): # entry[1] is the user-id, entry[2] is the book-isbn
    train_matrix[entry[1]-1, entry[2]-1] = entry[3] # -1 is to counter 0-based indexing
train_matrix.shape<jupyter_output><empty_output><jupyter_text>Now do the same for the test set.<jupyter_code>### TESTING SET
# Create user-item matrices
n_users = test_data['u_unique'].nunique()
n_books = test_data['b_unique'].nunique()

# First, create an empty matrix of size USERS x BOOKS (this speeds up the later steps)
test_matrix = np.zeros((n_users, n_books))

# Then, add the appropriate vals to the matrix by extracting them from the df with itertuples
for entry in test_data.itertuples(): # entry[1] is the user-id, entry[2] is the book-isbn
    test_matrix[entry[1]-1, entry[2]-1] = entry[3] # -1 is to counter 0-based indexing
test_matrix.shape<jupyter_output><empty_output><jupyter_text>Now the matrix is in the correct format, with the user and book entries encoded from the mapping dict created above!  Calculating cosine similarity with the 'pairwise distances' functionTo determine the similarity between users/items we'll use the 'cosine similarity' which is a common n-dimensional distance metric.**Note:** since all of the rating values are positive (1-10 scale), the cosine distances will all fall between 0 and 1.<jupyter_code># It may take a while to calculate, so I'll perform on a subset initially
train_matrix_small = train_matrix[:10000, :10000]
test_matrix_small = test_matrix[:10000, :10000]

from sklearn.metrics.pairwise import pairwise_distances
user_similarity = pairwise_distances(train_matrix_small, metric='cosine')
item_similarity = pairwise_distances(train_matrix_small.T, metric='cosine') # .T transposes the matrix (NumPy)<jupyter_output><empty_output><jupyter_text>If we are looking at similarity between users we need to account for the average behaviour of that individual user. For example, one user may give all movies quite high ratings, whereas one might give all ratings between 3 and 7. These users might otherwise have quite similar preferences.To do this, we use the users average rating as a 'weighting' factor.If we are looking at item-based similarity we don't need to add this weighting factor.We can incorporate this into a ```predict()``` function, like so:<jupyter_code>def predict(ratings, similarity, type='user'): # default type is 'user'
    if type == 'user':
        mean_user_rating = ratings.mean(axis=1)
        # Use np.newaxis so that mean_user_rating has the same format as ratings
        ratings_diff = (ratings - mean_user_rating[:, np.newaxis])
        pred = mean_user_rating[:, np.newaxis] + similarity.dot(ratings_diff) / np.array([np.abs(similarity).sum(axis=1)]).T
    elif type == 'item':
        pred = ratings.dot(similarity) / np.array([np.abs(similarity).sum(axis=1)])
    return pred<jupyter_output><empty_output><jupyter_text>Then can make our predictions!<jupyter_code>item_prediction = predict(train_matrix_small, item_similarity, type='item')
user_prediction = predict(train_matrix_small, user_similarity, type='user')<jupyter_output><empty_output><jupyter_text>EvaluationHow do we know if this is making good ```rating``` predictions?We'll start by just taking the root mean squared error (RMSE) (from ```sklearn```) of predicted values in the ```test_set``` (i.e. where we know what the answer should be).Since we want to compare only predicted ratings that are in the test set, we can filter out all other predictions that aren't in the test matrix.<jupyter_code>from sklearn.metrics import mean_squared_error
from math import sqrt

def rmse(prediction, test_matrix):
    prediction = prediction[test_matrix.nonzero()].flatten()
    test_matrix = test_matrix[test_matrix.nonzero()].flatten()
    return sqrt(mean_squared_error(prediction, test_matrix))

# Call on test set to get error from each approach ('user' or 'item')
print(f'User-based CF RMSE: {rmse(user_prediction, test_matrix_small)}')
print(f'Item-based CF RMSE: {rmse(item_prediction, test_matrix_small)}')<jupyter_output>User-based CF RMSE: 7.929926397008533
Item-based CF RMSE: 7.930326958653732<jupyter_text>For the user-item and the item-item recommendations we get RMSE = 7.85 (MSE > 60) for both. This is pretty bad, but we only trained over a small subset of the data.Although this collaborative filtering setup is relatively simple to write, it doesn't scale very well at all, as it is all stored in memory! (Hence why we only used a subset of the training/testing data).----------------Instead, we should really use a model-based (based on matrix factorization) recommendation algorithm. These are inherently more scalable and can deal with higher sparsity level than memory-based models, and are considered more powerful due to their ability to pick up on "latent factors" in the relationships between what sets of items users like. However, they still suffer from the "cold start" problem (where a new user has no history).Fortunately, there is a Python library called ```surprise``` that was built specifically for the implementation of model-based recommendation systems! This library comes with many of the leading algorithms in this space already built-in. Let's try use it for our book recommender system.  Using the ```surprise``` library for building a recommender systemSeveral common model-based algorithms including SVD, KNN, and non-negative matrix factorization are built-in!  See [here](http://surprise.readthedocs.io/en/stable/getting_started.htmlbasic-usage) for the docs.<jupyter_code>from surprise import Reader, Dataset
user_item_rating.head() # take a look at our data
# First need to create a 'Reader' object to set the scale/limit of the ratings field
reader = Reader(rating_scale=(1, 10))

# Load the data into a 'Dataset' object directly from the pandas df.
# Note: The fields must be in the order: user, item, rating
data = Dataset.load_from_df(user_item_rating, reader)
# Load the models and 'evaluation' method
from surprise import SVD, NMF, model_selection, accuracy<jupyter_output><empty_output><jupyter_text>Where: SVD = Singular Value Decomposition (orthogonal factorization), NMF = Non-negative Matrix Factorization.**Note** that when using the ```surprise``` library we don't need to manually create the mapping of USERID and UNIQUE_ITEMIDS to integers in a custom dict. See [here](http://surprise.readthedocs.io/en/stable/FAQ.htmlraw-inner-note) for details.   SVD model **_Using cross-validation (5 folds)_**<jupyter_code># Load SVD algorithm
model = SVD()

# Train on books dataset
%time model_selection.cross_validate(model, data, measures=['RMSE'], cv=5, verbose=True)<jupyter_output>Evaluating RMSE of algorithm SVD on 5 split(s).

                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     
RMSE (testset)    1.6379  1.6342  1.6399  1.6333  1.6383  1.6367  0.0025  
Fit time          27.50   29.12   28.19   28.50   27.17   28.09   0.70    
Test time         1.26    1.28    1.25    1.07    1.22    1.22    0.07    
CPU times: user 2min 32s, sys: 1.44 s, total: 2min 33s
Wall time: 2min 34s<jupyter_text>The SVD model gives an average RMSE of ca. 1.64 after 5-folds, with a fit time of ca. 28 s for each fold. **_Using test-train split_**<jupyter_code># set test set to 20%.
trainset, testset = model_selection.train_test_split(data, test_size=0.2)

# Instantiate the SVD model.
model = SVD()

# Train the algorithm on the training set, and predict ratings for the test set
model.fit(trainset)
predictions = model.test(testset)

# Then compute RMSE
accuracy.rmse(predictions)<jupyter_output>RMSE: 1.6355<jupyter_text>Using a 80% train-test split, the SVD model gave a RMSE of 1.6426.-----------We can see that using the SVD algorithm has already far out-performed the memory-based collaborative filtering approach (RMSE of 1.64 vs 7.92)!  NMF model<jupyter_code># Load NMF algorithm
model = NMF()
# Train on books dataset
%time model_selection.cross_validate(model, data, measures=['RMSE'], cv=5, verbose=True)<jupyter_output>Evaluating RMSE of algorithm NMF on 5 split(s).

                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     
RMSE (testset)    2.4719  2.4753  2.4991  2.4583  2.4690  2.4747  0.0134  
Fit time          48.55   48.36   48.67   48.21   49.12   48.58   0.31    
Test time         1.00    1.22    1.24    0.89    1.27    1.12    0.15    
CPU times: user 4min 12s, sys: 2.77 s, total: 4min 15s
Wall time: 4min 15s<jupyter_text>The NMF model gave a mean RMSE of ca. 2.47, with a fit time of ca. 48 s.It seems like the SVD algorithm is the best choice for this dataset.  Optimizing the SVD algorithm with parameter tuningSince it seems like the SVD algorithm is our best choice, let's see if we can improve the predictions even further by optimizing some of the algorithm hyperparameters.One way of doing this is to use the handy ```GridSearchCV``` method from the ```surprise``` library. When passed a range of hyperparameter values, ```GridSearchCV``` will automatically search through the parameter-space to find the best-performing set of hyperparameters.<jupyter_code># We'll remake the training set, keeping 20% for testing
trainset, testset = model_selection.train_test_split(data, test_size=0.2)
### Fine-tune Surprise SVD model useing GridSearchCV
from surprise.model_selection import GridSearchCV

param_grid = {'n_factors': [80, 100, 120], 'lr_all': [0.001, 0.005, 0.01], 'reg_all': [0.01, 0.02, 0.04]}

# Optimize SVD algorithm for both root mean squared error ('rmse') and mean average error ('mae')
gs = GridSearchCV(SVD, param_grid, measures=['rmse', 'mae'], cv=3)
# Fit the gridsearch result on the entire dataset
%time gs.fit(data)
# Return the best version of the SVD algorithm
model = gs.best_estimator['rmse']

print(gs.best_score['rmse'])
print(gs.best_params['rmse'])
model_selection.cross_validate(model, data, measures=['rmse', 'mae'], cv=5, verbose=True)<jupyter_output>Evaluating RMSE, MAE of algorithm SVD on 5 split(s).

                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     
RMSE (testset)    1.6326  1.6292  1.6314  1.6323  1.6296  1.6310  0.0014  
MAE (testset)     1.2594  1.2612  1.2611  1.2587  1.2563  1.2593  0.0018  
Fit time          25.22   24.25   24.24   24.00   24.18   24.38   0.43    
Test time         1.08    1.46    1.46    1.05    1.04    1.22    0.20<jupyter_text>The mean RSME using the optimized parameters was 1.6351 over 5 folds, with an average fit time of ca. 24s.<jupyter_code>### Use the new parameters with the training set
model = SVD(n_factors=80, lr_all=0.005, reg_all=0.04)
model.fit(trainset) # re-fit on only the training data using the best hyperparameters
test_pred = model.test(testset)
print("SVD : Test Set")
accuracy.rmse(test_pred, verbose=True)<jupyter_output>SVD : Test Set
RMSE: 1.6287<jupyter_text>Using the optimized hyperparameters we see a slight improvement in the resulting RMSE (1.629) compared with the unoptimized SVD algorithm (1.635)1  Testing some of the outputs (ratings and recommendations)Would like to do an intuitive check of some of the recommendations being made.Let's just choose a random user/book pair (represented in the ```suprise``` library as ```uid``` and ```iid```, respectively).**Note:** The ```model``` being used here is the optimized SVD algorithm that has been fit on the training set.<jupyter_code># get a prediction for specific users and items.
uid = 276744  # the USERID int
iid = '038550120X' # the UNIQUE_ITEMIDS string
# This pair has an actual rating of 7!

pred = model.predict(uid, iid, verbose=True)<jupyter_output>user: 276744     item: 038550120X r_ui = None   est = 7.18   {'was_impossible': False}<jupyter_text>Can access the attributes of the ```predict``` method to get a nicer output.<jupyter_code>print(f'The estimated rating for the book with the "UNIQUE_ITEMIDS" code {pred.iid} from user #{pred.uid} is {pred.est:.2f}.\n')
actual_rtg = user_item_rating[(user_item_rating.USERID==pred.uid) & (user_item_rating.UNIQUE_ITEMIDS==pred.iid)].RATING.values[0]
print(f'The real rating given for this was {actual_rtg:.2f}.')
# get a prediction for specific users and items.
uid = 95095  # the USERID int
iid = '0140079963' # the UNIQUE_ITEMIDS string
# This pair has an actual rating of 6.0!

pred = model.predict(uid, iid, verbose=True)
print(f'The estimated rating for the book with the "UNIQUE_ITEMIDS" code {pred.iid} from user #{pred.uid} is {pred.est:.2f}.\n')
actual_rtg = user_item_rating[(user_item_rating.USERID==pred.uid) & (user_item_rating.UNIQUE_ITEMIDS==pred.iid)].RATING.values[0]
print(f'The real rating given for this was {actual_rtg:.2f}.')<jupyter_output>The estimated rating for the book with the "unique_isbn" code 0140079963 from user #95095 is 6.87.

The real rating given for this was 6.00.<jupyter_text>The following function was adapted from the ```surprise``` docs, and can be used to get the top book recommendations for each user.<jupyter_code>from collections import defaultdict

def get_top_n(predictions, n=10):
    '''Return the top-N recommendation for each user from a set of predictions.

    Args:
        predictions(list of Prediction objects): The list of predictions, as
            returned by the test method of an algorithm.
        n(int): The number of recommendation to output for each user. Default
            is 10.

    Returns:
    A dict where keys are user (raw) ids and values are lists of tuples:
        [(raw item id, rating estimation), ...] of size n.
    '''

    # First map the predictions to each user.
    top_n = defaultdict(list)
    for uid, iid, true_r, est, _ in predictions:
        top_n[uid].append((iid, est))

    # Then sort the predictions for each user and retrieve the k highest ones.
    for uid, user_ratings in top_n.items():
        user_ratings.sort(key=lambda x: x[1], reverse=True)
        top_n[uid] = user_ratings[:n]
        
    return top_n<jupyter_output><empty_output><jupyter_text>Let's get the Top 10 recommended books for each USERID in the test set.<jupyter_code>pred = model.test(testset)
top_n = get_top_n(pred)
def get_reading_list(userid):
    """
    Retrieve full book titles from full 'books_users_ratings' dataframe
    """
    reading_list = defaultdict(list)
    top_n = get_top_n(predictions, n=10)
    for n in top_n[userid]:
        book, rating = n
        title = books_users_ratings.loc[books_users_ratings.UNIQUE_ITEMIDS==book].TITLE.unique()[0]
        reading_list[title] = rating
    return reading_list
# Just take a random look at USERID=60337
example_reading_list = get_reading_list(userid=60337)
for book, rating in example_reading_list.items():
    print(f'{book}: {rating}')<jupyter_output>A Natural History of the Senses: 9.124985146110324
Grace For The Moment: 9.004232452533614
Eva Luna: 8.957717816714606
A Slender Thread : Rediscovering Hope at the Heart of Crisis: 8.81756023683983
Tell Me the Truth About Love: Ten Poems: 8.729204207356146
Papa, My Father: A Celebration of Dads: 8.708985984501158
Let the Dead Bury Their Dead (Harvest American Writing Series): 8.646273705729296
The Doctor's Book of Home Remedies : Thousands of Tips and Techniques Anyone Can Use to Heal Everyday Health Problems: 8.635645117795315
Why We Cant Wait: 8.535174763005822
Lump It or Leave It: 8.535174763005822
{'fraction_non_alphanumeric': 0.09415681322465848, 'fraction_numerical': 0.040523476064745725, 'mean_word_length': 4.6078279902150125, 'pattern_counts': {'":': 0, '<': 239, '<?xml version=': 0, '>': 245, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '60', 'dir': 'jupyter-structured-clean-dedup', 'id': '6092', 'license': 'Apache-2.0', 'n_tokens_mistral': 15046, 'n_tokens_neox': 13598, 'n_words': 4875, 'path': '_notebooks/2021-07-01-book-crossing-surprise-svd-nmf.ipynb', 'repo_name': 'recohut/notebook', 'size': '43671'}