<jupyter_start><jupyter_text>Recommender Systems in Keras> A programming introduction to recommender systems using Keras!- toc: true - badges: true- comments: true- author: <NAME>- categories: [ML] I have written a [few](../nnmf-tensorflow.html) [posts](../nmf-autograd.html) [earlier](../nmf-cvx.html) [about](../nmf-out-matrix.html) [matrix](../mf-autograd-adagrad.html) [factorisation](../contrained-nmf-cvx.html) [using](../nmf-nnls.html) various Python libraries. The main application I had in mind for matrix factorisation was [recommender systems](https://en.wikipedia.org/wiki/Recommender_system). In this post, I'll write about using [Keras](https://keras.io) for creating recommender systems. [Various](https://github.com/maciejkula/triplet_recommendations_keras) [people](http://blog.richardweiss.org/2016/09/25/movie-embeddings.html) [have](https://github.com/bradleypallen/keras-movielens-cf) [written](https://github.com/hexiangnan/neural_collaborative_filtering) [excellent](https://github.com/sonyisme/keras-recommendation) [similar](http://course.fast.ai/lessons/lesson4.html) [posts](https://github.com/maciejkula/spotlight) and code that I draw a lot of inspiration from, and give them their credit! I'm assuming that a reader has some experience with Keras, as this post is not intended to be an introduction to Keras.Specifically, in this post, I'll talk about:1. Matrix Factorisation in Keras2. Adding non-negativitiy constraints to solve non-negative matrix factorisation (NNMF)3. Using neural networks for recommendationsI'll be using the Movielens-100k dataset for illustration. There are 943 users and 1682 movies. In total there are a 100k ratings in the dataset. It should be noted that the max. total number of rating for the  would be 943*1682, which means that we have about 7% of the total ratings! All rating are on a scale of 1-5.   **Task**Given this set of ratings, can we recommend the next set of movies to a user? This would translate to: for every user, estimating the ratings for all the movies that (s)he hasn't watched and maybe recommend the top-k movies by the esimtated ratings!   Peak into the dataset<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline
dataset = pd.read_csv("/Users/nipun/Downloads/ml-100k/u.data",sep='\t',names="user_id,item_id,rating,timestamp".split(","))
dataset.head()<jupyter_output><empty_output><jupyter_text>So, each record (row) shows the rating for a user, item (movie) pair. It should be noted that I use  item and movie interchangeably in this post.<jupyter_code>len(dataset.user_id.unique()), len(dataset.item_id.unique())<jupyter_output><empty_output><jupyter_text>We assign a unique number between (0, users) to each user and do the same for movies.<jupyter_code>dataset.user_id = dataset.user_id.astype('category').cat.codes.values
dataset.item_id = dataset.item_id.astype('category').cat.codes.values
dataset.head()<jupyter_output><empty_output><jupyter_text>Train test splitWe'll now split our dataset of 100k ratings into train (containing 80k ratings) and test (containing 20k ratings). Given the train set, we'd like to accurately estimate the ratings in the test set.<jupyter_code>from sklearn.model_selection import train_test_split
train, test = train_test_split(dataset, test_size=0.2)
train.head()
test.head()<jupyter_output><empty_output><jupyter_text>Matrix factorisationOne popular recommender systems approach is called Matrix Factorisation. It works on the principle that we can learn a low-dimensional representation (embedding) of user and movie. For example, for each movie, we can have how much action it has, how long it is, and so on. For each user, we can encode how much they like action, or how much they like long movies, etc. Thus, we can combine the user and the movie embeddings to estimate the ratings on unseen movies. This approach can also be viewed as: given a matrix (A [M X N]) containing users and movies, we want to estimate low dimensional matrices (W [M X k] and H [M X k]), such that: $A \approx W.H^T$  Matrix factorisation in KerasWe'll now write some code to solve the recommendation problem by matrix factorisation in Keras. We're trying to learn two low-dimensional embeddings of users and items.<jupyter_code>import keras
from IPython.display import SVG
from keras.optimizers import Adam
from keras.utils.vis_utils import model_to_dot
n_users, n_movies = len(dataset.user_id.unique()), len(dataset.item_id.unique())
n_latent_factors = 3<jupyter_output>Using TensorFlow backend.<jupyter_text>The key thing is to learn an embedding for movies and users, and then combine them using the dot product! For estimating the rating, for each user, movie pair of interest, we'd take the dot product of the respective user and item embedding. As an example, if we have 2 dimensions in our user and item embedding, which say correspond to [how much user likes action, how much user likes long movies], and the item embedding is [how much action is in the movie, how long is the movie]. Then, we can predict for a user `u`, and movie `m` as how much `u` likes action $\times$ how much action is there in `m` $+$ how much `u` likes long movies $\times$ how long is `m`.Our model would optimise the emebedding such that we minimise the mean squared error on the ratings from the train set.<jupyter_code>movie_input = keras.layers.Input(shape=[1],name='Item')
movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors, name='Movie-Embedding')(movie_input)
movie_vec = keras.layers.Flatten(name='FlattenMovies')(movie_embedding)

user_input = keras.layers.Input(shape=[1],name='User')
user_vec = keras.layers.Flatten(name='FlattenUsers')(keras.layers.Embedding(n_users + 1, n_latent_factors,name='User-Embedding')(user_input))

prod = keras.layers.merge([movie_vec, user_vec], mode='dot',name='DotProduct')
model = keras.Model([user_input, movie_input], prod)
model.compile('adam', 'mean_squared_error')<jupyter_output><empty_output><jupyter_text>Here's a visualisation of our model for a better understanding.<jupyter_code>SVG(model_to_dot(model,  show_shapes=True, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>We can see that in the `Merge` layer, we take the dot product of the user and the item embeddings to obtain the rating.We can also summarise our model as follows:<jupyter_code>model.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Item (InputLayer)               (None, 1)            0                                            
__________________________________________________________________________________________________
User (InputLayer)               (None, 1)            0                                            
__________________________________________________________________________________________________
Movie-Embedding (Embedding)     (None, 1, 3)         5049        Item[0][0]                       
__________________________________________________________________________________________________
User-Embedding (Embedding)      (None, 1, 3)         2832        User[0][0]                       
__________[...]<jupyter_text>So, we have 7881 parameters to learn! Let's train our model now!<jupyter_code>history = model.fit([train.user_id, train.item_id], train.rating, epochs=100, verbose=0)<jupyter_output><empty_output><jupyter_text>Train error v/s epoch number Before we test how well our model does in the test setting, we can visualise the train loss with epoch number.<jupyter_code>pd.Series(history.history['loss']).plot(logy=True)
plt.xlabel("Epoch")
plt.ylabel("Train Error")<jupyter_output><empty_output><jupyter_text>Prediction error Let's now see how our model does! I'll do a small post-processing step to round off our prediction to the nearest integer. This is usually not done, and thus just a whimsical step, since the training ratings are all integers! There are better ways to encode this intger requirement (one-hot encoding!), but we won't discuss them in this post.<jupyter_code>y_hat = np.round(model.predict([test.user_id, test.item_id]),0)
y_true = test.rating
from sklearn.metrics import mean_absolute_error
mean_absolute_error(y_true, y_hat)<jupyter_output><empty_output><jupyter_text>Not bad! We're able to get a $MAE$ of 0.69! I'm sure with a bit of parameter/hyper-parameter optimisation, we may be able to improve the results. However, I won't talk about these optimisations in this post.   Extracting the learnt embeddings We can extract the learnt movie and item embeddings as follows:<jupyter_code>movie_embedding_learnt = model.get_layer(name='Movie-Embedding').get_weights()[0]
pd.DataFrame(movie_embedding_learnt).describe()
user_embedding_learnt = model.get_layer(name='User-Embedding').get_weights()[0]
pd.DataFrame(user_embedding_learnt).describe()<jupyter_output><empty_output><jupyter_text>We can see that both the user and the item embeddings have negative elements. There are some applications which require that the learnt embeddings be non-negative. This approach is also called non-negative matrix factorisation, which we'll workout now.  Non-negative Matrix factorisation (NNMF) in Keras The code for NNMF remains exactly the same as the code for matrix factorisation. The only change is that we add `non-negativity` constraints on the learnt embeddings. This is done as follows:<jupyter_code>from keras.constraints import non_neg
movie_input = keras.layers.Input(shape=[1],name='Item')
movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors, name='NonNegMovie-Embedding', embeddings_constraint=non_neg())(movie_input)
movie_vec = keras.layers.Flatten(name='FlattenMovies')(movie_embedding)

user_input = keras.layers.Input(shape=[1],name='User')
user_vec = keras.layers.Flatten(name='FlattenUsers')(keras.layers.Embedding(n_users + 1, n_latent_factors,name='NonNegUser-Embedding',embeddings_constraint=non_neg())(user_input))

prod = keras.layers.merge([movie_vec, user_vec], mode='dot',name='DotProduct')
model = keras.Model([user_input, movie_input], prod)
model.compile('adam', 'mean_squared_error')<jupyter_output><empty_output><jupyter_text>We now verify if we are indeed able to learn non-negative embeddings. I'll not compare the performance of NNMF on the test set, in the interest of space.<jupyter_code>history_nonneg = model.fit([train.user_id, train.item_id], train.rating, epochs=10, verbose=0)
movie_embedding_learnt = model.get_layer(name='NonNegMovie-Embedding').get_weights()[0]
pd.DataFrame(movie_embedding_learnt).describe()<jupyter_output><empty_output><jupyter_text>Looks good!  Neural networks for recommendationWe'll now create a simple neural network for recommendation, or for estimating rating! This model is very similar to the earlier matrix factorisation models, but differs in the following ways:1. Instead of taking a dot product of the user and the item embedding, we concatenate them and use them as features for our neural network. Thus, we are not constrained to the dot product way of combining the embeddings, and can learn complex non-linear relationships.2. Due to 1, we can now have a different dimension of user and item embeddings. This can be useful if one dimension is larger than the other.<jupyter_code>n_latent_factors_user = 5
n_latent_factors_movie = 8

movie_input = keras.layers.Input(shape=[1],name='Item')
movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors_movie, name='Movie-Embedding')(movie_input)
movie_vec = keras.layers.Flatten(name='FlattenMovies')(movie_embedding)
movie_vec = keras.layers.Dropout(0.2)(movie_vec)


user_input = keras.layers.Input(shape=[1],name='User')
user_vec = keras.layers.Flatten(name='FlattenUsers')(keras.layers.Embedding(n_users + 1, n_latent_factors_user,name='User-Embedding')(user_input))
user_vec = keras.layers.Dropout(0.2)(user_vec)


concat = keras.layers.merge([movie_vec, user_vec], mode='concat',name='Concat')
concat_dropout = keras.layers.Dropout(0.2)(concat)
dense = keras.layers.Dense(200,name='FullyConnected')(concat)
dropout_1 = keras.layers.Dropout(0.2,name='Dropout')(dense)
dense_2 = keras.layers.Dense(100,name='FullyConnected-1')(concat)
dropout_2 = keras.layers.Dropout(0.2,name='Dropout')(dense_2)
dense_3 = keras.layers.Dense(50,name='FullyConnected-2')(dense_2)
dropout_3 = keras.layers.Dropout(0.2,name='Dropout')(dense_3)
dense_4 = keras.layers.Dense(20,name='FullyConnected-3', activation='relu')(dense_3)


result = keras.layers.Dense(1, activation='relu',name='Activation')(dense_4)
adam = Adam(lr=0.005)
model = keras.Model([user_input, movie_input], result)
model.compile(optimizer=adam,loss= 'mean_absolute_error')<jupyter_output><empty_output><jupyter_text>Let's now see how our model looks like:<jupyter_code>SVG(model_to_dot(model,  show_shapes=True, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>It should be noted that we use a different number of embeddings for user (3) and items (5)! These combine to form a vector of length (5+3 = 8), which is then fed into the neural network. We also add a dropout layer to prevent overfitting!<jupyter_code>model.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Item (InputLayer)               (None, 1)            0                                            
__________________________________________________________________________________________________
User (InputLayer)               (None, 1)            0                                            
__________________________________________________________________________________________________
Movie-Embedding (Embedding)     (None, 1, 8)         13464       Item[0][0]                       
__________________________________________________________________________________________________
User-Embedding (Embedding)      (None, 1, 5)         4720        User[0][0]                       
__________[...]<jupyter_text>We can see that the number of parameters is more than what we had in the Matrix Factorisation case. Let's see how this model works. I'll run it for more epochs given that we have more parameters.<jupyter_code>history = model.fit([train.user_id, train.item_id], train.rating, epochs=250, verbose=0)<jupyter_output><empty_output><jupyter_text>Prediction performance of Neural Network based recommender system<jupyter_code>y_hat_2 = np.round(model.predict([test.user_id, test.item_id]),0)
print(mean_absolute_error(y_true, y_hat_2))

print(mean_absolute_error(y_true, model.predict([test.user_id, test.item_id])))<jupyter_output>0.6957
0.708807692927
{'fraction_non_alphanumeric': 0.10790941910075484, 'fraction_numerical': 0.012930751558910403, 'mean_word_length': 5.409760201935212, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 75, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '480838', 'license': 'Apache-2.0', 'n_tokens_mistral': 4585, 'n_tokens_neox': 4287, 'n_words': 1370, 'path': '_notebooks/2017-12-18-recommend-keras.ipynb', 'repo_name': 'nipunbatra/blog3', 'size': '15240'}