<jupyter_start><jupyter_text>Simple Collaborative Filtering modelFor this particular model I will be using a dataset which consists of a series of books and their corresponding ratings from a set of users. My model is aimed at essentially learning how different books relate to one another and also allow for the prediction of how users might rate books they havn't already rated.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn import datasets
from scipy.stats import norm
import latex
import random
import math
import seaborn as sns
from sklearn.model_selection import train_test_split

%matplotlib inline
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Cleaning the data and creating a matrix of user ratingsRows in the matrix will represent all of the different books while columns in the matrix will represent different users. The item at matrix[book i,user j] = rating(i,j). If there was no rating then this spot will be zero because rating are between 1 - 10 inclusive.<jupyter_code># Number of total users = 2946
# Number of total books = 17384
matrix = np.zeros((17384,2946))
matrix.shape
# File oriented as: user item rating
with open("../data/book_data/book_ratings.dat") as file:
    lines = file.readlines()
    i = 0
    for line in lines:
        if i != 0:
            line = line.strip("\n")
            aline = line.split()
            matrix[int(aline[1])-1,int(aline[0])-1] = int(aline[2].split(".")[0])
        i += 1<jupyter_output><empty_output><jupyter_text>Preprocessing the data with mean normalizationBefore using the matrix for learning, I am going to apply mean normalization. Specifically, I am getting the mean ratings for all books in the matrix and using that value to update the ratings for each book in the matrix.$$ Matrix = Matrix - \mu $$I noticed that there are books with no rankings at all so I will be removing them from the original matrix and the mu vector.<jupyter_code>num_books = matrix.shape[0]
mu = []
nan_indexes = [] # saving indexes that 

for i in range(num_books):
    ratings = matrix[i,:]
    avg = np.asscalar(ratings[np.nonzero(ratings)].mean())
    if ratings[np.nonzero(ratings)].shape == (0,): # no ratings
        nan_indexes.append(i)
    else:
        mu.append(avg)
mu = np.array(mu)
mu.shape = (14684,1)
print("shape of mu vector:",mu.shape)
matrix = np.delete(matrix,nan_indexes,axis=0)
print("shape of updated matrix:",matrix.shape)
# Mean normalization:
matrix_norm = matrix
for i in range(matrix_norm.shape[0]): # go through each book
    for j in range(matrix_norm.shape[1]): # go through each user rating for a book
        if matrix_norm[i,j] != 0:
            matrix_norm[i,j] = matrix_norm[i,j] - mu[i]
print(matrix_norm.shape)<jupyter_output>(14684, 2946)<jupyter_text>Building collaborative filtering modelIn this model I am attempting to learn both Theta, parameter vector for all users, and x, feature vector for each book. Note that I will be using regularization and I am choosing for there to be 4 features to learn.Cost function: Minimizing x(1),...,x(nb) and Theta(1),...,Theta(nu) simultaneously:$$ \frac{1}{2} \sum_{(i,j): r(i,j)=1} \big( (\theta^{(j)})^T x^{(i)} - y^{(i,j)} \big)^2    + \frac{\lambda}{2} \sum^{n_b}_{i=1} \sum^{n}_{k=1} (x^{(i)}_{k})^2    + \frac{\lambda}{2} \sum^{n_u}_{i=1} \sum^{n}_{k=1} (\theta^{(j)}_{k})^2 $$ Back prop to update x, theta:$$ x^{(i)}_k := x^{(i)}_k - \alpha \bigg( \sum_{(i,j): r(i,j)=1} \big( (\theta^{(j)})^T x^{(i)} - y^{(i,j)} \big) \theta^{(j)}_k + \lambda x^{(i)}_k \bigg) $$$$ \theta^{(j)}_k := \theta^{(j)}_k - \alpha \bigg( \sum_{(i,j): r(i,j)=1} \big( (\theta^{(j)})^T x^{(i)} - y^{(i,j)} \big) x^{(i)}_k + \lambda \theta^{(j)}_k \bigg) $$- nb: Number of total books- nu: Number of total users- r(i,j) = 1: Means there is a rating by user j for book i- y(i,j): rating by user j for book i- Theta(j): Parameter vector for user j- x(i): Parameter vector for book i<jupyter_code># Initializes theta parameters and x features with small randomized values from a uniform distribution
# theta: shape(#users,4), x: shape(#books,4)
def initialize_params(num_users, num_books, num_param):
    x = np.random.normal(0, 0.05, (num_books,num_param))
    theta = np.random.normal(0, 0.05, (num_users,num_param))
    return x,theta
"""
Returns a list of tuples representing the indexes of all nonzero values as (book index, user index)
We need this, as we are only computing the cost given indexes that actually have reviews
"""
def get_nonzero_indexes(matrix):
    indexes = []
    non_zero_i = np.nonzero(matrix)
    book_i = non_zero_i[0]
    user_i = non_zero_i[1]
    
    for i in range(len(book_i)):
        indexes.append((book_i[i],user_i[i]))
    return indexes
"""
To compute the non-regularization term, this cost function only loops through entries that have reviews, this
will ultimately cut down the number of items from 46.6 million to 74k, which is much more computationally reasonable
"""
def cost_function(matrix,x,theta,indexes,lam):
    reg_x = (lam / 2) * np.sum(np.sum(np.square(x),axis=1))
    reg_theta = (lam / 2) * np.sum(np.sum(np.square(theta),axis=1))
    
    cost = 0
    for i in range(len(indexes)):
        x_i = indexes[i][0]
        theta_i = indexes[i][1]
        ax = x[x_i,:] # shape(1,#features)
        atheta = theta[theta_i,:] # shape(1,#features)
        cost += np.square(np.dot(atheta.T,ax)-matrix[x_i,theta_i])
    
    final_cost = (1/2 * cost) + reg_x + reg_theta
    return final_cost
"""
Updates all of the x values, features for books
"""
def update_x(x,new_x,theta,matrix,indexes,alpha,lam,num_params):
    
    for i in range(x.shape[0]): # updating x values one at a time
        ratings = [j for j in indexes if j[0] == i] # ratings for specific x, list of tuples
        asum = np.zeros((num_params,))
        for tup in ratings:
            asum += (np.dot(theta[tup[1],:].T,x[tup[0],:]) - matrix[tup[0],tup[1]]) * theta[tup[1],:] + lam * x[tup[0],:]
        
        new_x[i,:] = new_x[i,:] - (alpha * asum)
    
    return new_x
"""
Updates all theta values, parameters for users
"""
def update_theta(theta,new_theta,x,matrix,indexes,alpha,lam,num_params):
    
    for i in range(theta.shape[0]): # updating theta values one at a time
        ratings = [j for j in indexes if j[1] == i] # ratings for specific theta, list of tuples
        asum = np.zeros((num_params,))
        for tup in ratings:
            asum += (np.dot(theta[tup[1],:].T,x[tup[0],:]) - matrix[tup[0],tup[1]]) * x[tup[0],:] + lam * theta[tup[1],:] 
        
        new_theta[i,:] = new_theta[i,:] - (alpha * asum)
    
    return new_theta
"""
Backprop step - returns updated matrices for x and theta
"""
def update_params(x,theta,matrix,indexes,alpha,lam,num_params):
    new_x = x
    new_theta = theta
    
    new_x = update_x(x,new_x,theta,matrix,indexes,alpha,lam,num_params)
    new_theta = update_theta(theta,new_theta,x,matrix,indexes,alpha,lam,num_params)
    
    return new_x,new_theta
"""
Returns a list of cost values along with the learned parameter matrices x,theta
"""
def model(matrix, alpha=0.1, lam=0.1, num_iter=100, print_cost=True,num_params=4):
    costs = []
    num_users = matrix.shape[1]
    num_books = matrix.shape[0]
    x,theta = initialize_params(num_users,num_books,num_params)
    indexes = get_nonzero_indexes(matrix)
    
    for i in range(num_iter):
        x,theta = update_params(x,theta,matrix,indexes,alpha,lam,num_params) # update params
        
        if i % 1 == 0 and print_cost:
            cost = cost_function(matrix,x,theta,indexes,lam)
            costs.append(cost)
            print("cost at epoch "+str(i+1)+": "+str(cost))
            
    return costs,x,theta 
a_costs,a_x,a_theta = model(matrix_norm, 0.01, 0.01, 100, True, 2)
np.save("x.npy",a_x)
np.save("theta.npy",a_theta)
# Shape of learned parameters
print(a_x.shape)
print(a_theta.shape)<jupyter_output>(14684, 2)
(2946, 2)<jupyter_text>Using learned parameters to predict user ratings and find books that are similar Note that the model hasn't reached a minimum, but that isn't strict necessary to show how user ratings would be predicted or how to find books that are similar to each other.<jupyter_code># Getting predicted user ratings
pred_ratings = np.dot(a_x,a_theta.T)
pred_ratings = pred_ratings + mu # Adding back mean
print(pred_ratings.shape)
pred_ratings<jupyter_output><empty_output><jupyter_text>Finding the book most similar to a given bookThis is done by comparing the distance of the feature vectors of both books. A small distance indicated that two books are similar.$$ || x^{(i)} - x^{(j)} || $$<jupyter_code>"""
Computes the distance between two vectors of shape (2,)
"""
def distance(vec_a, vec_b):
    return np.sum(np.square(vec_a - vec_b))
# Finding five books most similar to the first book
book_1 = a_x[0,:]
print(book_1.shape)
# Finds the most similar book to the first book
smallest_dist = 1000
closest_book = () # (book id, dist value, features)

for i in range(1,a_x.shape[0]):
    temp_features = a_x[i,:]
    dist = distance(book_1, temp_features)
    if dist < smallest_dist:
        smallest_dist = dist
        closest_book = (i,smallest_dist,temp_features)
# Found that book at index 10515 is the closest to the book at index 0
closest_book<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13109333333333334, 'fraction_numerical': 0.017493333333333333, 'mean_word_length': 3.992545260915868, 'pattern_counts': {'":': 1, '<': 24, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '324509', 'license': 'MIT', 'n_tokens_mistral': 3272, 'n_tokens_neox': 3112, 'n_words': 995, 'path': 'recommender_systems/Collaborative_filtering.ipynb', 'repo_name': 'nateandre/machine_learning', 'size': '9375'}