<jupyter_start><jupyter_text>Yelp Reviews Rating Prediction <NAME> (bz2280) Github: bzhouuuJames <NAME> (jl5241) Github: yjs1210Ujjwal Peshin (up2138) Github: ujjwal95<NAME> (zj2249) Github: jiangzl2016 A. AbstractWe compare the merits of n different recommendation algorithms side by side.* Biased Baseline* Collaborative Filtering (CF - Baseline)* Collective Matrix Factorization (CMF)* Content-Based Filtering (CBF)* Field-aware Factorization Machine (FFM) * Deep Learning Model* Hybrid Approach Comparing the results of the models against our baseline shows that Collective Matrix Factorization Algorithm, DeepFM Algorithm, and Wide and Deep Algorithms outperform the baseline model. The strength of deep learning approaches were showcased as we increased the size of the dataset, beating the baselines initially by .02 RMSE in 20% subsampled data but later beating them by .04 RMSE in 100% of the data. Another key advantage of the deep learning approaches were the short training time. Using 100% of the dataset, they finished training in just 800 seconds on local environment while the CMF took an equivalent amount of time for dataset subsampled down to 50% size. Additionally, the weaknesses of the baseline models were exposed when metrics that measure user experiences such as ranking, coverage, and novelty were investigated. While our predictive algorithms didn’t provide drastic improvements in accuracy, our top deep learning methods providing about 3% improvement versus the baseline, given the fast training time as well as its ability to achieve superior measures on metrics that affect positive user experiences such as coverage, ranking, and novelty metrics, there is a strong reason to believe that deep learning models can provide value to Yelp users in a meaningful manner. In this report, we discuss the merits of various algorithms we explored and finally propose a complete recommendation system, that employs an ensemble of data retrieval system based on user location, cold-start recommendations using the bias-based model as well as DeepFM for general recommendation.  B. Business Case and ObjectiveOur goal through this exercise is to figure out the best methodology for recommending restaurants to our users. This will be largely measured by using RMSE, which measures our forecast’s error relative to the ground truth as well as ranking metrics. Additionally, we want to measure metrics that can help us assess the sanity of our recommendation and experiences of the users. We measure these using coverage and novelty metrics to ensure that there is enough diversity in our recommendations and that our model isn’t resorting to recommending the most popular restaurants as opposed to creating a truly personalized experience.  There are additional considerations such as model computation time, which affects long term overhead of maintenance of the product, and model comprehension. These are important considerations for the business as inefficient solutions will add technology debt and burden to the company’s systems. Also, employing models that are well understood can help the team in getting buy-in from stakeholders. Therefore, these factors will be critically analyzed when assessing the merits of our algorithms.   C. DataWe use the publicly available yelp dataset which consists roughly of 7 million reviews from 1.6 million users across 190 thousand businesses in 10 metropolitan areas. We are additionally provided with business and user metadata as well as check-in information, tips and photos provided by the reviewers.<jupyter_code># !pip install tqdm
import pandas as pd
import tarfile
from tqdm import tqdm, tqdm_notebook, tnrange
import json
import numpy as np
import time
from copy import deepcopy
zf = tarfile.open('yelp_dataset.tar') 
zf.extract('review.json')
line_count = len(open("review.json").readlines())
user_ids, business_ids, stars, dates, texts = [], [], [], [], []
with open("review.json") as f:
    for line in tqdm(f, total=line_count):
        blob = json.loads(line)
        user_ids += [blob["user_id"]]
        business_ids += [blob["business_id"]]
        stars += [blob["stars"]]
        dates += [blob["date"]]
        texts += [blob["text"]]
ratings_ = pd.DataFrame(
    {"user_id": user_ids, "business_id": business_ids, "rating": stars, "date": dates, "text": texts}
)
user_counts = ratings_["user_id"].value_counts()
active_users = user_counts.loc[user_counts >= 5].index.tolist()
ratings_ = ratings_.loc[ratings_.user_id.isin(active_users)]
ratings_.head()
n_users = len(ratings_.user_id.unique())
n_restaurants = len(ratings_.business_id.unique())
print('Unique Users: {0}, unique restaurants: {1}'.format(n_users, n_restaurants))<jupyter_output>Unique Users: 286130, unique restaurants: 185723<jupyter_text>Load in User attributes set and Restaurant attributes set<jupyter_code>users_ = pd.read_csv('active_users.csv')
business_ = pd.read_csv('businesses.csv')
users_.head()
business_.head()<jupyter_output><empty_output><jupyter_text>E. Subsampling and Train-Test Split E.1. FIltering, Undersampling and Dataset Size ConsiderationsThe idea behind undersampling is to develop a smaller dataset that is representable, or partially representative of the entire dataset, and expedites the model development cycle. We utilize 20% dataset size generally across the board for fast development iteration and for computation intensive tasks such as hyperparameter tuning. However, it is critical to analyze model performance across dataset sizes since the models’ capability to handle large dataset is an important consideration if it were to be utilized in production. Therefore we investigate two other dataset sizes as well, 50% and 100% of the data size. Whereas 20% of the dataset size is used generally, larger datasets are employed to analyze certain models’ capability to handle large data as well as measure how their performance changes with larger datasets. Finally, we filter our dataset to only active users who have at least 5 reviews since there needs to be at least some data about the user for the model to perform. If a user does not have at least 5 reviews, we will be building out recommendations using cold-start methods. E.2 Train Test SplitTrain-Test split is done in a very straightforward way. We take all the users that made through our filters as described above. Then we simply take the last two reviews of the users. The most recent review becomes our test set and the other second most recent review becomes our validation set.<jupyter_code>SAMPLING_RATE = 1/5
# Downsample by users
user_id_unique = ratings_.user_id.unique()
user_id_sample = pd.DataFrame(user_id_unique, columns=['unique_user_id']) \
                    .sample(frac= SAMPLING_RATE, replace=False, random_state=1)

ratings_sample = ratings_.merge(user_id_sample, left_on='user_id', right_on='unique_user_id') \
                    .drop(['unique_user_id'], axis=1)
print(ratings_sample.head())
print(ratings_sample.shape)
# hold out last review
ratings_user_date = ratings_sample.loc[:, ['user_id', 'date']]
ratings_user_date.date = pd.to_datetime(ratings_user_date.date)
index_holdout = ratings_user_date.groupby(['user_id'], sort=False)['date'].transform(max) == ratings_user_date['date']
ratings_holdout_ = ratings_sample[index_holdout]
ratings_traincv_ = ratings_sample[~index_holdout]

ratings_user_date = ratings_traincv_.loc[:, ['user_id', 'date']]
index_holdout = ratings_user_date.groupby(['user_id'], sort=False)['date'].transform(max) == ratings_user_date['date']
ratings_cv_ = ratings_traincv_[index_holdout]
ratings_train_ = ratings_traincv_[~index_holdout]

# remove the user that has fake reviews 

cv_users_del = set(ratings_cv_.user_id) - set(ratings_train_.user_id)
holdout_users_del = set(ratings_holdout_.user_id) - set(ratings_train_.user_id)
ratings_cv_ = ratings_cv_[~ratings_cv_.user_id.isin(cv_users_del)]
ratings_holdout_ = ratings_holdout_[~ratings_holdout_.user_id.isin(holdout_users_del)]

# ratings_cv_ = ratings_cv_[~ratings_cv_.user_id.isin(['HiT9sg9pvDiEVMFHJYihXg'])]
# ratings_holdout_ = ratings_holdout_[~ratings_holdout_.user_id.isin(['HiT9sg9pvDiEVMFHJYihXg'])]

print('There are {0} rows, {1} columns in training set.'.format(ratings_train_.shape[0], ratings_train_.shape[1]))
print('There are {0} rows, {1} columns in training set.'.format(ratings_cv_.shape[0], ratings_cv_.shape[1]))
print('There are {0} rows, {1} columns in holdout set.'.format(ratings_holdout_.shape[0], ratings_holdout_.shape[1]))
# check if we have a enough user sample size (> 50000)
number_of_unique_users = len(ratings_train_.user_id.unique())
print(number_of_unique_users)<jupyter_output>57223<jupyter_text>F. Evaluation MetricsA list of evaluation metrics the team uses are: F.1. Regression MetricsRegression metrics measure We present four different regression metrics that are measured but we primarily use the RMSE. Root Mean Square Error (RMSE) calculates square rooted sum of square residuals of predictions. It measures numerical difference between all ground-truth ratings and actual ratings in test set.Mean Absolute Error (MAE) calculates sum of absolute residuals. It measures numerical difference between all ground-truth ratings and actual ratings in test set, and more robust to outliers.R-squared measures what percentage of variance in target that is explained by predictions. The higher the value, the better the predictions. F.2. Ranking MetricsWe first make top 10 recommendations of restaurants for each user and see how relevant the rankings were. First we discuss the metrics then discuss a few caveats to the approach we took to measure them.Inclusion of Last Review in Top 10 Recommendations: We train our model on the training set and make top 10 recommendations to the users. We then look at the test set and see if the user’s latest review was included in that top 10 recommendations. We calculate the proportion of users who received such recommendations. In other words we have,$$\frac{1}{n} * \sum_j^n \sum_{i \in j's top 10}^{10}Rel(i)$$where a recommendation is relevant if it is the latest visited restaurant of the user and n is the total number of users measured. Average Ranking of Latest Restaurant: We train our model on the training set and make a prediction on every single business  and user combination. We then measure the average rankings on that prediction of the latest business that the user visited. In other words we have,$$\frac{1}{n} * \sum_j^n\sum_i^m Rank(i)(1_{i=latest business of j})$$Where the indicator function is one if the restaurant is the last visited restaurant of user j, n is the total number of users and m is the total number of businesses.Some caveats of our approach is that instead of making a prediction on the entire business universe, we make predictions on the businesses that are in the same city as the business of the user’s last review. This makes sense since it would be futile to recommend a restaurant in Los Angeles to a user who is in New York City. This also allows us to reduce the computation time of our recommendation, which is another key advantage we want to have when we serve our model to the users. Additionally, we measure the above two metrics on a subsample of users as opposed to all the users to save computation time, and we also only take subsample of ratings that were positive ratings since we want to measure how good our recommendations are.  F.3. CoverageWe measure the coverage of our recommendations by looking at the proportion of our recommendations that are distinct. In other words, we measure, $$\frac{number of distinct businesses in all recommendations to the subsample} {number of all recommendations to the subsample}$$  F.4. NoveltyWe measure the novelty of our recommendations by simply taking the proportion of businesses in our top ten recommendations that the user hasn’t been to. This is crucial since we don’t want our recommendations to be filled with restaurants that the user has already been to. We measure novelty as simply.  F.5. Runtime We measure the runtime of the model’s training as well as its prediction time on validation set using Python’s time library.   Metrics on Coverage and SerendipityFirst subsample a group of users that we will measure these metrics fromMethodology:    We sample 5 users from each city where the user made the latest review.    These cities must have at least 100 unique businesses    These users must also have made a postive review(above their historical average)to those restaurants.        1. We recommend 10 restaurants to each user        2. We see if their latest restaurant makes it into the top 10 list (Ranking Metric)        3. We see for those 10 x 5 recommendations, how many of them are distinct businesses (Coverage)        4. We see for those top 10 recommendations, how many of them are restaurants they have not visited (Serendipity)        Additionally, we measure what our ranking was for the latest restaurant that the user visited(Ranking Metric    Criteria:<jupyter_code>def process(df):
    df['date']  = pd.to_datetime(df['date'])
    df['week_day'] = df['date'].dt.weekday
    df['month'] = df['date'].dt.month
    df['hour'] = df['date'].dt.hour
    df = df.merge(users_, on = 'user_id')
    df = df.merge(business_, on = 'business_id')
    rename_dict = {'business_longitude': 'longitude', 'business_latitude': 'latitude',
                  'business_state':'state','business_city':'city', 'business_address': 'address'}
    df = df.rename(columns = rename_dict)
    return df

ratings_train = process(ratings_train_.copy())
ratings_holdout = process(ratings_holdout_.copy())
ratings_val = process(ratings_cv_.copy())
ratings_train_final = ratings_train.append(ratings_val)
ratings_entire_df = ratings_train.append(ratings_val).append(ratings_holdout)
ratings_holdout.columns
unique_city_businesses = ratings_entire_df[['city','business_id']].drop_duplicates()
unique_cities = unique_city_businesses.groupby('city').count()['business_id']
unique_cities = unique_cities[unique_cities > 100]
out = pd.DataFrame()
for city in unique_cities.index:
    tmp = ratings_holdout[(ratings_holdout['city'] ==city) &
                              (ratings_holdout['rating'] >ratings_holdout['average_stars'])]
    if len(tmp['user_id'].unique())>4:
        
        ###this weird sampling technique is to ensure we dont' sample the same user twice in a same city
        five_users = np.random.choice(tmp['user_id'].unique(),5, replace = False)
        row = tmp[tmp['user_id'].isin(five_users)].groupby('user_id', group_keys=False).apply(lambda df: df.sample(1))
        out = out.append(row)
predict_df = out[['user_id','city','state']]
predict_df = predict_df.merge(unique_city_businesses, on = 'city')
predict_df.to_csv('data/metric_sample.csv')
# predict_df['predictions'] = 2.5
def get_all_metrics(predict_df, validation_subsample, ratings_train_final):
    top_10_recs = predict_df.groupby(['user_id','city'])['predictions'].nlargest(10).reset_index()
    out = validation_subsample
    cnt =0
    serendipity = 0
    
    
    for row in out.iterrows():
        row_values = row[1]
        top_10 = predict_df.loc[top_10_recs[top_10_recs['user_id'] == row_values['user_id']].level_2]['business_id']
        ###In top 10
        if row_values['business_id'] in top_10.values:
            cnt+=1
        user_history = ratings_train_final[ratings_train_final['user_id'] == row_values['user_id']]    
        been_there = [i for i in top_10.values if i in  user_history.business_id.values]
        serendipity += 1-len(been_there)/10
    
    top_10 = cnt/len(out)
    serendipity = serendipity/len(out)
    
    predict_df = predict_df.reset_index()
    
    analysis_df = predict_df.merge(top_10_recs, left_on = ['user_id','city','index'], \
                                   right_on = ['user_id','city','level_2'])
    
    coverage = (analysis_df.groupby('city')['business_id'].nunique()/50).values.mean()
    
    predict_df['rankings']=predict_df.groupby(['city','user_id'])['predictions']. \
                                                        rank(method="first",ascending = False)
    running_rankings =0
    for row in out.iterrows():
        row_values = row[1]
        user_recs = predict_df[(predict_df['user_id']==row_values['user_id'])
                            &(predict_df['city']==row_values['city'])
                             & (predict_df['business_id']==row_values['business_id'])
                              ]
        assert len(user_recs)==1
        running_rankings += user_recs['rankings'].sum()

    avg_rank = running_rankings / len(out)
    print(top_10, coverage, serendipity, avg_rank)
    
    return top_10, coverage, serendipity, avg_rank<jupyter_output><empty_output><jupyter_text>F. MethodsA list of models that the team attempts are: * Bias Baseline* Collaborative Filtering Baseline: SVD* Collective Matrix Factorization (CMF)* Content-Based Filtering (CBF)* Field-aware Factorization Machine (FFM) * Deep Learning Model_**Note**_: the team has run algorithms on **20%, 50%, 100%** training data respectively. But for readability purpose, only result on 20% data is displayed in this notebook. For full result, please refer to a result table enclosed in **pdf report**. Baseline models  Baseline 1: Bias Baseline$\sum_{r_{ui} \in R_{train}} \left(r_{ui} - (\mu + b_u + b_i)\right)^2 +\lambda \left(b_u^2 + b_i^2 \right)$.  Hyperparameter Tuning<jupyter_code>from surprise import SVD
from surprise import accuracy
from surprise import Reader
from surprise.model_selection import GridSearchCV
from surprise import Dataset
from surprise import BaselineOnly
bsl_options = {'method': 'als', 'n_epochs':3}
bias_baseline = BaselineOnly(bsl_options)
algo.fit(train_sr_20)
predictions = algo.test(val_sr_20)
accuracy.rmse(predictions)
bsl_options = {'method': 'als', 'n_epochs':5}
bias_baseline = BaselineOnly(bsl_options)
algo.fit(train_sr_20)
predictions = algo.test(val_sr_20)
accuracy.rmse(predictions)
bsl_options = {'method': 'als', 'n_epochs':7}
bias_baseline = BaselineOnly(bsl_options)
algo.fit(train_sr_20)
predictions = algo.test(val_sr_20)
accuracy.rmse(predictions)
bsl_options = {'method': 'als', 'n_epochs':9}
bias_baseline = BaselineOnly(bsl_options)
algo.fit(train_sr_20)
predictions = algo.test(val_sr_20)
accuracy.rmse(predictions)<jupyter_output>RMSE: 1.3492<jupyter_text>**Observation**: It seems different hyperparameters all performs the same result; the team just uses default.  Evaluation**Note**: the team has evaluated this and the following algorithms on **20%, 50%, 100%** training data respectively. For readability, only result on 20% data is displayed. For full result, please refer to a result table enclosed.<jupyter_code># 20%
start_time = time.time()
bias_baseline.fit(train_sr_20)
print("--- %s seconds ---" % (time.time() - start_time))
# 20%
bbase_p = bias_baseline.test(test_sr_20)
start_time = time.time()
bbase_20_df = pd.DataFrame(bbase_p, columns = ['userId','itemId','rating','pred_rating','x'])
accuracy.rmse(bbase_p)
print('R^2 (with 20% data): ', r2_score(bbase_20_df.rating , bbase_20_df.pred_rating))
print('MAE (with 20% data): ', mean_absolute_error(bbase_20_df.rating, bbase_20_df.pred_rating))
print("--- %s seconds ---" % (time.time() - start_time))
algo = BaselineOnly()
eval_before_50 = eval_20.build_full_trainset()
eval_sr_20 = eval_before_20.build_testset()
algo.fit(train_sr_20)
eval_pred_20 = algo.test(eval_sr_20)
#accuracy.rmse(predictions_50)
baseline_20 = pd.DataFrame(eval_pred_20, columns = ['userId','itemId','rating','pred_rating','x'])
top_10, coverage, serendipity, avg_rank = get_all_metrics(predict_df_20, out_20, ratings_train_final_20)<jupyter_output>0.10714285714285714 0.503095238095238 0.9697619047619042 528.3333333333334<jupyter_text>Baseline 2: Collaborative Filtering via SVDMatrix factorization is a class of collaborative filtering algorithms. The general idea behind matrix factorization is that there can exist a lower dimensional latent space of features in which users and items can be represented such that the interaction between them can be obtained by simply dot producing the corresponding dense vectors in that space. In short, it decomposes a m*n user-item interaction matrix into two m*k and k*n matrices, sharing a joint latent vector space, where m represents the number of users, and n represents the number of items. In terms of its outcome, we are likely to observe that close users in terms of preferences as well as close items in terms of characteristics can have close representations in the latent space.The mathematical overview is as follows:Given a n*m matrix, such that . X is the user matrix where rows represent the n users and Y is the item matrix where rows represent the m items. We want to search for the dot product of matrices X and Y that best approximate the existing interactions; i.e., we want to find X and Y that minimize the “rating reconstruction error”:$$ (X,Y) = argmin_{X,Y} \sum_{(i,j) \in E} [(X_i)(Y_j)^T − M_{ij}]^2$$Adding a regularization term, we can also get:$$(X,Y) = argmin_{X,Y} ½ \sum_{(i,j) \in E} [(X_i)(Y_j)^T − M_{ij}]^2 + \lambda/2(\sum_{i,k}(X_{ik})^2 + \sum_{j,k}(Y_{jk})^2)$$In general, we obtain the matrices X and Y following a gradient descent optimization process. And once the matrices are obtained, we can predict the ratings simply by multiplying the user vector by any item vector.In this Yelp Rating Challenge, we used the python surprise package to implement MF. The MF algorithm there uses the SVD approach, which is essentially $$ P_{m * n} = U_{m * m} \sum_{m * n} V_{n * n}$$There, the prediction is $$\hat(r_{ui}) = \mu + b_u + b_i + (q_i)^T p_u $$ and the regularized squared error that needs to be minimized is $$\sum_{r_{ui} \in R_{train}} (r_{ui} − \hat(r_{ui}))^2 + \lambda(b^2_{i} + b^2_{u} + ||q_i||^2 + ||p_u||^2)$$As the way the package is designed, we tuned on n_epochs, lr_all and leg_all to get an optimal hyperparameter set, where n_epochs is the number of iterations of the SGD (stochastic gradient descent) procedure, lr_all is the learning rate for all parameters, and reg_all is the regularization term for all parameters.<jupyter_code>#!pip install cmfrec
# !pip install scikit-surprise
from cmfrec import CMF
from surprise import KNNBasic

import matplotlib.pyplot as plt
import json
from tqdm import tqdm
def process(df):
    df['date']  = pd.to_datetime(df['date'])
    df['week_day'] = df['date'].dt.weekday
    df['month'] = df['date'].dt.month
    df['hour'] = df['date'].dt.hour
    df = df.merge(users_, on = 'user_id')
    df = df.merge(business_, on = 'business_id')
    rename_dict = {'business_longitude': 'longitude', 'business_latitude': 'latitude',
              'business_state':'state','business_city':'city', 'business_address': 'address'}
    df = df.rename(columns = rename_dict)
    return df
ratings_train = process(ratings_train_.copy())
ratings_test = process(ratings_holdout_.copy())
ratings_val = process(ratings_cv_.copy())
ratings_test = ratings_test.loc[ratings_test.business_id.isin(ratings_train.business_id)]
ratings_val = ratings_val.loc[ratings_val.business_id.isin(ratings_train.business_id)]
trainset = ratings_train.loc[:,['user_id', 'business_id', 'rating']]
trainset.columns = ['userID', 'itemID','rating']
valset = ratings_val.loc[:, ['user_id', 'business_id', 'rating']]
valset.columns = ['userID', 'itemID','rating']
testset = ratings_holdout.loc[:, ['user_id', 'business_id', 'rating']]
testset.columns = ['userID', 'itemID','rating']
reader = Reader(rating_scale = (0.0, 5.0))
train_data = Dataset.load_from_df(trainset[['userID','itemID','rating']], reader)
val_data = Dataset.load_from_df(valset[['userID','itemID','rating']], reader)
test_data = Dataset.load_from_df(testset[['userID','itemID','rating']], reader)

train_sr = train_data.build_full_trainset()
val_sr_before = val_data.build_full_trainset()
val_sr = val_sr_before.build_testset()
test_sr_before = test_data.build_full_trainset()
test_sr = test_sr_before.build_testset()<jupyter_output><empty_output><jupyter_text>Hyperparameter Tuning<jupyter_code>RMSE_tune = {}
n_epochs = [5, 7, 10]  # the number of iteration of the SGD procedure
lr_all = [0.002, 0.003, 0.005] # the learning rate for all parameters
reg_all =  [0.4, 0.5, 0.6] # the regularization term for all parameters

for n in n_epochs:
    for l in lr_all:
        for r in reg_all:
            algo = SVD(n_epochs = n, lr_all = l, reg_all = r)
            algo.fit(train_sr)
            predictions = algo.test(val_sr)
            RMSE_tune[n,l,r] = accuracy.rmse(predictions)
import operator
min(RMSE_tune.items(), key=operator.itemgetter(1))[0]<jupyter_output><empty_output><jupyter_text>**Observation**: The best combination is when n_epochs = 10, lr_all = 0.005, reg_all = 0.4, and the RMSE score is 1.3496  Evaluation<jupyter_code># train and test on the optimal parameter
start_time = time.time()
algo_real = SVD(n_epochs = 10, lr_all = 0.005, reg_all = 0.4)
algo_real.fit(train_sr)
predictions = algo_real.test(test_sr)
print("--- %s seconds ---" % (time.time() - start_time))
accuracy.rmse(predictions)
accuracy.mae(predictions)
predict_df_20 = pd.read_csv('data/metric_sample.csv', index_col=0)
predict_df_20.columns
# To evaluate coverage and serendipity metrics, use evaluation set created earlier.
predict_df_20 = pd.read_csv('data/metric_sample.csv', index_col=0)
predict_df_20['predictions'] = 2.5 # fill in this value temporally
eval_20 = Dataset.load_from_df(predict_df_20[['user_id','business_id','predictions']], reader)
eval_before_20 = eval_20.build_full_trainset()
eval_sr_20 = eval_before_20.build_testset()
eval_pred_20 = algo_real.test(eval_sr_20)

baseline_20 = pd.DataFrame(eval_pred_20, columns = ['userId','itemId','rating','pred_rating','x'])
predict_df_20['predictions'] = baseline_20.pred_rating
top_10, coverage, serendipity, avg_rank = get_all_metrics(predict_df_20, out, ratings_train_final)<jupyter_output>0.10930232558139535 0.4932558139534884 0.9762790697674414 518.7511627906977<jupyter_text>Models (i) Collective Matrix Factorization (CMF)Collective Matrix Factorization method decomposes two matrices $X$ and $Y$ into three matrices $U$, $V$, and $Z$ such that $X \approx f(UV^T)$ and $ Y \approx f(VZ^T)$, where f is either the identity or sigmoid function. This allows us to include additional features of our users or items that might help optimize the result of personalization. To beat the baseline (which in our case, is Matrix Factorization), we first tried the Collective Matrix Factorization technique. This means that, in addition to the ratings, we also want to include additional features on either the items or the users, to see how the model perform. Before we discussed what additional features we included, we might say a few words on the method and its mathematical intuition.Similar to Matrix Factorization, Collective Matrix Factorization also aims to decompose, but in this time, we are decomposing two matrices $X$ and $Y$ into three matrices $U$, $V$, and $Z$, such that $X \approx f(UV^T)$ and $Y \approx f(VZ^T)$. Here, X can be the same matrix we see in the case of Matrix Factorization, i.e., a simple user-item matrix filled by ratings. And matrix Y is the matrix of additional features. It can be on user or on item. And the features can be one-hot encoded (i.e., categorical) or numerical. For example, in our case, we both involved state location for businesses (which is a categorical feature) and the average rating (which is a numerical feature).Collective Matrix Factorization also has a function to minimize, and we will explain the function through the lens of the python cmfrec package, which is the package that we used to implement Collective Matrix Factorization. The function to minimize is as follows:$ argmin_{A, B, C, D, U_b, I_b} \lVert (X − \mu − U_b − I_b − AB^T)I_{x} \lVert^2 + \lVert U − AC^T \lVert^2 + \lVert I − BD^T \lVert^2 + \lambda (\lVert A\lVert^2 + \lVert B \lVert^2 + \lVert C \lVert^2 + \lVert D \lVert^2  + \lVert U_b \lVert^2 + \lVert I_b \lVert^2)$Where X is the ratings matrix, I is the item-attribute matrix, U is the user-attribute matrix, and A,B,C,D are lower-dimensional matrices. $ |X| $, $|I|$, $|U|$ are the number of non-missing entries in each matrix. And $ Ubias_{u}$ and $Ibias_{i}$ are user and item biases. Thus, for such a reason, when tuning, we tune w_main and w_item when we are including additional item features and we tune w_main and w_user when we are including additional user features.Now, with CMF, we implemented **3** approaches in total- **state average rating, state location, and user average rating**. Please refer to detail as follows:<jupyter_code>users = users_
businesses = business_

def process(df):
    df['date']  = pd.to_datetime(df['date'])
    df['week_day'] = df['date'].dt.weekday
    df['month'] = df['date'].dt.month
    df['hour'] = df['date'].dt.hour
    df = df.merge(users, on = 'user_id')
    df = df.merge(businesses, on = 'business_id')
    rename_dict = {'business_longitude': 'longitude', 'business_latitude': 'latitude',
                  'business_state':'state','business_city':'city', 'business_address': 'address'}
    df = df.rename(columns = rename_dict)
    return df
X_train_20 = trainset_20
X_train_20.columns = ['UserId','ItemId','Rating']

X_test_20 = testset_20
X_test_20.columns = ['UserId','ItemId','Rating']<jupyter_output><empty_output><jupyter_text>**Approach 1: CMF on User, Business and State Average Rating**We first included state average rating as an item additional feature. Since our objective here is to predict the last rating for each active user, we wanted to see if location can be a useful means to bring closer the predicted ratings to the actual ones. To better utilize the location feature though, we came up with two approaches. One is to calculate the state average rating, and the other is to use state location as a categorical feature. With the state average rating, we wanted to observe if some states can have a higher average rating than others. This means that either the restaurants in that state are significantly better or the users in that state are more lenient and friendly. From EDA alone, the average rating seems to make some sense, as we observed that California has the highest state average rating (though with much fewer data observations) and New York has the lowest (same, with much fewer data observations than AZ and NV). This might suggest that, even though New York (especially manhattan area) is known as a food hub, the yelp users here can be a little picky and critical, whereas the users in CA can be more friendly. By all means though, we wanted to see if this “secondary” information that we generated for ourselves can help better predict the result.<jupyter_code># get state average rating
state_avg_20 = pd.DataFrame(ratings_train_20.groupby("state").rating.mean())
state_avg_20.columns = ['state_avg']
train_state_avg_20 = ratings_train_20.merge(state_avg_20, on = "state")
# item additional info: state average
item_avg_20 = train_state_avg_20.loc[:,['business_id','state_avg']]
item_avg_20.columns = ['ItemId','state_avg']<jupyter_output><empty_output><jupyter_text>1.1 Hyperparameter Tuning<jupyter_code>tune = {}
w_main = [0.5, 5.0, 10.0] # weight assign to the MRSE in factorization of the ratings matrix
w_item = [0.5, 5.0, 10.0][::-1] # weight assign to the MRSE in factorization of the item attributes matrix
# tuning
for m in w_main:
    for i in w_item:
        model = CMF(w_main = m, w_item = i, random_seed = 1)
        model.fit(ratings = deepcopy(X_train_20), item_info = deepcopy(item_avg_20))
        prediction = model.predict(X_val_20.UserId, X_val_20.ItemId)
        X_val_20['pred_rating'] = prediction
        tune[m,i] = np.sqrt(np.mean((X_val_20.pred_rating - X_val_20.Rating)**2))
tune<jupyter_output><empty_output><jupyter_text>1.2 Results<jupyter_code>### Best param: w_main = 5.0, w_item = 5.0
# 20%
model = CMF(w_main = 5.0, w_item = 5.0, random_seed = 1)
start_time = time.time()
model.fit(ratings = deepcopy(X_train_20), item_info = deepcopy(item_avg_20))
print("--- %s seconds ---" % (time.time() - start_time))
X_test_20 = X_test_20.loc[X_test_20.ItemId.isin(X_train_20.ItemId)]
X_test_50 = X_test_50.loc[X_test_50.ItemId.isin(X_train_50.ItemId)]
X_test_100 = X_test_100.loc[X_test_100.ItemId.isin(X_train_100.ItemId)]
state_prediction_20 = model.predict(X_test_20.UserId, X_test_20.ItemId)
X_test_20['pred_rating'] = state_prediction_20
print('RMSE (with 20% data): ', np.sqrt(np.mean((X_test_20.pred_rating - X_test_20.Rating)**2)))
print('R^2 (with 20% data): ', r2_score(X_test_20.Rating ,X_test_20.pred_rating))
print('MAE (with 20% data): ', mean_absolute_error(X_test_20.Rating, X_test_20.pred_rating))<jupyter_output>RMSE (with 20% data):  1.3495188130540272
R^2 (with 20% data):  0.18677513147304292
MAE (with 20% data):  1.1202827761356042<jupyter_text>1.3 Evaluate<jupyter_code>predict_df_20.head()
model = CMF(w_main = 5.0, w_item = 5.0, random_seed = 1)
model.fit(ratings = deepcopy(X_train_20), item_info = deepcopy(item_avg_20))
predict_df_20['predictions'] = model.predict(predict_df_20.user_id, predict_df_20.business_id)
top_10, coverage, serendipity, avg_rank = get_all_metrics(predict_df_20, out_20, ratings_train_final_20)<jupyter_output>0.13095238095238096 0.2609523809523809 0.9695238095238087 484.93809523809523<jupyter_text>**Approach 2: CMF on State Location as feature**With the second approach on the location feature, we simply fed in the one-hot encoded columns. This time, we just want to see if the location itself, as a categorical feature, can bring us any closer to the actual value.  2.1 Hyperparameter Tuning<jupyter_code># one-hot encode
df_state_20 = pd.get_dummies(ratings_train_20.state)

state_loc_20 = ratings_train_20.loc[:,['business_id']]
state_loc_20.columns = ['ItemId']
state_loc_20 = state_loc_20.join(df_state_20)
tune_2 = {}

w_main = [0.5, 5.0, 10.0] # weight assign to the MRSE in factorization of the ratings matrix
w_item = [0.5, 5.0, 10.0][::-1] # weight assign to the MRSE in factorization of the item attributes matrix

# tuning
for m in w_main:
    for i in w_item:
        model = CMF(w_main = m, w_item = i, random_seed = 1)
        model.fit(ratings = deepcopy(X_train_20), item_info = deepcopy(state_loc_20),\
            cols_bin_item=[cl for cl in state_loc_20.columns if cl != 'ItemId'])
        prediction = model.predict(X_val_20.UserId, X_val_20.ItemId)
        X_val_20['pred_rating'] = prediction
        tune_2[m,i] = np.sqrt(np.mean((X_val_20.pred_rating - X_val_20.Rating)**2))<jupyter_output>INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
  Objective function value: 1.502233
  Number of iterations: 48
  Number of functions evaluations: 59
INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
  Objective function value: 1.345407
  Number of iterations: 49
  Number of functions evaluations: 61
INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
  Objective function value: 0.946406
  Number of iterations: 46
  Number of functions evaluations: 58
INFO:tensorflow:Optimization terminated with:
  Message: b'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT'
  Objective function value: 6.707548
  Number of iterations: 1000
  Number of functions evaluations: 1050
INFO:tensorflow:Optimization terminated with:
  Message: b'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT'
  Objective function value: 6.544841
  Number of iter[...]<jupyter_text>2.2 Results<jupyter_code># get the best param: w_main = 5.0, w_item = 5.0
tune_2
# 20%
X_test_20 = X_test_20.loc[X_test_20.ItemId.isin(X_train_20.ItemId)]
model = CMF(w_main = 5.0, w_item = 5.0, random_seed = 1)
start_time = time.time()
model.fit(ratings = deepcopy(X_train_20), item_info = deepcopy(state_loc_20),\
            cols_bin_item=[cl for cl in state_loc_20.columns if cl != 'ItemId'])
print("--- %s seconds ---" % (time.time() - start_time))
start_time = time.time()
loc_prediction_20 = model.predict(X_test_20.UserId, X_test_20.ItemId)
X_test_20['pred_rating'] = loc_prediction_20
print('RMSE (with 20% data): ', np.sqrt(np.mean((X_test_20.pred_rating - X_test_20.Rating)**2)))
print('R^2 (with 20% data): ', r2_score(X_test_20.Rating , X_test_20.pred_rating))
print('MAE (with 20% data): ', mean_absolute_error(X_test_20.Rating , X_test_20.pred_rating))
print("--- %s seconds ---" % (time.time() - start_time))<jupyter_output>RMSE (with 20% data):  1.3496651740914456
R^2 (with 20% data):  0.18659872653728782
MAE (with 20% data):  1.1201697481001782<jupyter_text>2.3 Evaluate<jupyter_code>model = CMF(w_main = 5.0, w_item = 5.0, random_seed = 1)
model.fit(ratings = deepcopy(X_train_20), item_info = deepcopy(state_loc_20),\
            cols_bin_item=[cl for cl in state_loc_20.columns if cl != 'ItemId'])
predict_df_20['predictions'] = model.predict(predict_df_20.user_id, predict_df_20.business_id)
top_10, coverage, serendipity, avg_rank = get_all_metrics(predict_df_20, out_20, ratings_train_final_20)<jupyter_output>0.13095238095238096 0.2602380952380952 0.9692857142857138 473.73571428571427<jupyter_text>**Approach 3: CMF on User Average Rating as feature**Lastly, we also did a user average rating for the user info. This is similar to the state average rating approach, but we just wanted to see if by directly analyzing the users’ behaviors, that normally how critical they are with the restaurants, can help us better understand them and have a better predicted result than other approaches. To notice here though, we didn’t use the given average rating, since that would include the last rating that we were trying to predict; thus, we had to hand-calculated the average rating again on the training set. We also wanted to include the yelp spending time, that is the time they been using yelp and see if this can help us better predict the result. But due to the time constraint, we are not able to implement.<jupyter_code># get state average rating
user_avg_20 = pd.DataFrame(ratings_train_20.groupby("user_id").rating.mean())
user_avg_20.columns = ['user_avg']
user_avg_20 = ratings_train_20.merge(user_avg_20, on = "user_id")

user_avg_50 = pd.DataFrame(ratings_train_50.groupby("user_id").rating.mean())
user_avg_50.columns = ['user_avg']
user_avg_50 = ratings_train_50.merge(user_avg_50, on = "user_id")

user_avg_100 = pd.DataFrame(ratings_train_100.groupby("user_id").rating.mean())
user_avg_100.columns = ['user_avg']
user_avg_100 = ratings_train_100.merge(user_avg_100, on = "user_id")
# user additional info: user average
user_info_20 = user_avg_20.loc[:,['user_id','user_avg']]
user_info_20.columns = ['UserId','state_avg']

user_info_50 = user_avg_50.loc[:,['user_id','user_avg']]
user_info_50.columns = ['UserId','state_avg']

user_info_100 = user_avg_100.loc[:,['user_id','user_avg']]
user_info_100.columns = ['UserId','state_avg']<jupyter_output><empty_output><jupyter_text>3.1 Tuning<jupyter_code>tune_3 = {}
w_main = [0.5, 5.0, 10.0] # weight assign to the MRSE in factorization of the ratings matrix
w_user = [0.5, 5.0, 10.0][::-1] # weight assign to the MRSE in factorization of the user attributes matrix
# tuning
for m in w_main:
    for i in w_user:
        model = CMF(w_main = m, w_user = i, random_seed = 1)
        model.fit(ratings = deepcopy(X_train_20), user_info = deepcopy(user_info_20))
        prediction = model.predict(X_val_20.UserId, X_val_20.ItemId)
        X_val_20['pred_rating'] = prediction
        tune_3[m,i] = np.sqrt(np.mean((X_val_20.pred_rating - X_val_20.Rating)**2))<jupyter_output>INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
  Objective function value: 0.912842
  Number of iterations: 463
  Number of functions evaluations: 530
INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
  Objective function value: 0.912821
  Number of iterations: 378
  Number of functions evaluations: 436
INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
  Objective function value: 0.912564
  Number of iterations: 131
  Number of functions evaluations: 147
INFO:tensorflow:Optimization terminated with:
  Message: b'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT'
  Objective function value: 6.144692
  Number of iterations: 1000
  Number of functions evaluations: 1085
INFO:tensorflow:Optimization terminated with:
  Message: b'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT'
  Objective function value: 6.126369
  Number of [...]<jupyter_text>3.2 Results<jupyter_code># get the best param: w_main = 5.0, w_user = 10.0
tune_3
testset_50 = ratings_holdout_50.iloc[:, 0:3]
testset_50.columns = ['userID', 'itemID','rating']

X_test_50 = testset_50
X_test_50.columns = ['UserId','ItemId','Rating']
X_test_20 = X_test_20.loc[X_test_20.ItemId.isin(X_train_20.ItemId)]
X_test_50 = X_test_50.loc[X_test_50.ItemId.isin(X_train_50.ItemId)]
X_test_100 = X_test_100.loc[X_test_100.ItemId.isin(X_train_100.ItemId)]
# 20%
model = CMF(w_main = 5.0, w_user = 10.0, random_seed = 1)
start_time = time.time()
model.fit(ratings = deepcopy(X_train_20), user_info = deepcopy(user_info_20))
print("--- %s seconds ---" % (time.time() - start_time))
start_time = time.time()
user_prediction_20 = model.predict(X_test_20.UserId, X_test_20.ItemId)
X_test_20['pred_rating'] = user_prediction_20
print('RMSE (with 20% data): ', np.sqrt(np.mean((X_test_20.pred_rating - X_test_20.Rating)**2)))
print('R^2 (with 20% data): ', r2_score(X_test_20.Rating , X_test_20.pred_rating))
print('MAE (with 20% data): ', mean_absolute_error(X_test_20.Rating , X_test_20.pred_rating))
print("--- %s seconds ---" % (time.time() - start_time))<jupyter_output>--- 0.562385082244873 seconds ---<jupyter_text>3.3 Evaluate<jupyter_code>model = CMF(w_main = 5.0, w_user = 10.0, random_seed = 1)
model.fit(ratings = deepcopy(X_train_20), user_info = deepcopy(user_info_20))
predict_df_20['predictions'] = model.predict(predict_df_20.user_id, predict_df_20.business_id)
top_10, coverage, serendipity, avg_rank = get_all_metrics(predict_df_20, out_20, ratings_train_final_20)<jupyter_output>0.1357142857142857 0.2580952380952381 0.9699999999999993 470.53333333333336<jupyter_text>(ii) Content Based Model **Approach 1: Predict rating via user reviews**Implemented according to **CF-MCM (User-Item based)** approach in this following paper: https://arxiv.org/pdf/1607.00024.pdfThe idea of this model is to predict the rating of a review given by user _U_ to restaurant _R_ by comparing the review to the ones written by other users who have also reviewed restaurant _R_. The predicted rating is calculated by taking a weighted average of other users' ratings, where the weight is calculated based on similarity of reviews. Here is the detail: Two users'ratings are compared in the following manner: first aggregate all reviews user _u_ and user _a_ have written; then bucket them according to ratings given (from r = 1.0 to 5.0); finally find the max of all 5 pairs of similarity.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
import nltk
nltk.download('stopwords')

from nltk.corpus import stopwords
from nltk.tokenize import WordPunctTokenizer
from nltk.stem.porter import PorterStemmer
from nltk.tokenize import word_tokenize
import string
import re
ratings_train = ratings_train_.copy()
ratings_cv = ratings_cv_.copy()
ratings_holdout = ratings_holdout_.copy()
def clean_text(text):
    # substitute some irregular context 
    text = re.sub(r"what's", "what is ", text)
    text = re.sub(r"\'s", " ", text)
    text = re.sub(r"\'ve", " have ", text)
    text = re.sub(r"n't", " not ", text)
    text = re.sub(r"i'm", "i am ", text)
    text = re.sub(r"\'re", " are ", text)
    text = re.sub(r"\'d", " would ", text)
    text = re.sub(r"\'ll", " will ", text)
    text = re.sub(r",", " ", text)
    text = re.sub(r"\.", " ", text)
    text = re.sub(r"!", " ! ", text)
    text = re.sub(r"\/", " ", text)
    text = re.sub(r"\^", " ^ ", text)
    text = re.sub(r"\+", " + ", text)
    text = re.sub(r"\-", " - ", text)
    text = re.sub(r"\=", " = ", text)
    text = re.sub(r"'", " ", text)
    text = re.sub(r"(\d+)(k)", r"\g<1>000", text)
    text = re.sub(r":", " : ", text)
    text = re.sub(r"\s{2,}", " ", text)
    # remove numbers
    text = re.sub(r"\d+", "", text)
    
    ## Convert words to lower case and split them
    text = text.lower().split()
    
    # remove punctuations from each word
    table = str.maketrans('','', string.punctuation)
    text = [w.translate(table) for w in text]
    
    
    ## Remove stop words
    stops = set(stopwords.words("english"))
    text = [w for w in text if not w in stops and len(w) >= 3]
    
    # Replace slang terms
    # Word stemming
    
#     porter = PorterStemmer()
#     text = [porter.stem(word) for word in text]
    
    text = " ".join(text)
    
    return text

ratings_train['text'] = ratings_train['text'].apply(clean_text)
ratings_train.head()
# concatnate all reviews for each business for each rating value
ratings_train_sub = ratings_train[['user_id', 'business_id', 'rating', 'text']]
# ratings_train_sub['ct'] = 0
userid_df = ratings_train_sub.groupby(['user_id', 'rating']).agg({'text': ' '.join, 'business_id': 'count'})
# retain a copy of indexed dataset
businessid_userid_df = ratings_train_sub.set_index(['user_id', 'business_id', 'rating'])

## Learn TF-IDF vector representation for each concatenated review
# which will be used for similarity calculation
userid_vectorizer = TfidfVectorizer(tokenizer = WordPunctTokenizer().tokenize, analyzer='word', \
                             stop_words='english', max_features=50)
userid_tfidf_fit = userid_vectorizer.fit(userid_df['text'])
# Algorithm implementation 

def predict_approach_1(selected_uid, selected_bid): 
    # cosine similarity
    def text_similarity(v1, v2):
        return np.dot(v1, v2.T).toarray()[0][0]
    
    pred_rating = -1

    users_reviews = userid_df

    selected_business = businessid_userid_df.xs(selected_bid, level='business_id').reset_index()
    users_who_review_business = selected_business.user_id

    user_reviews_userID = users_reviews.xs(selected_uid, level='user_id')

    w_ui_u_vec = []
    for ui in users_who_review_business:
        user_reviews_ui = users_reviews.xs(ui, level='user_id')
        rating_range = [1.0, 2.0, 3.0, 4.0, 5.0]
        valid_rating_range_1 = user_reviews_userID.reset_index()['rating'].tolist()
        valid_rating_range_2 = user_reviews_ui.reset_index()['rating'].tolist()
        w_ui_u = -100

        for r in rating_range:
            if (r not in valid_rating_range_1) or (r not in valid_rating_range_2):
                continue
            else:
                user_reviews_r = user_reviews_userID.loc[r, 'text']
                ui_reviews_r = user_reviews_ui.loc[r, 'text']
                user_reviews_r_vec = userid_tfidf_fit.transform([user_reviews_r])
                ui_reviews_r_vec = userid_tfidf_fit.transform([ui_reviews_r])
                # similarity_r = text_similarity(user_reviews_r, ui_reviews_r)
                similarity_r = text_similarity(user_reviews_r_vec, ui_reviews_r_vec)
                if similarity_r > w_ui_u:
                    w_ui_u = similarity_r
        w_ui_u_vec.append(w_ui_u)

    tmp = user_reviews_userID.reset_index()
    r_u_bar = sum(tmp.rating * tmp.business_id) / sum(tmp.business_id) # business_id is number of ratings 
                                                                        # that has correpsonding value 
    if sum(w_ui_u_vec) == 0:
        pred_rating = r_u_bar + sum(w_ui_u_vec \
                                * (selected_business.rating - np.mean(selected_business.rating))) \
                                * 1.0 / sum(w_ui_u_vec)
    else:
        pred_rating = r_u_bar
    return pred_rating
tmp_df = ratings_cv[ratings_cv['user_id'].isin(ratings_train.user_id)].reset_index()
ratings_cv_filtered = tmp_df[tmp_df['business_id'].isin(ratings_train.business_id)]

tmp_df = ratings_holdout[ratings_holdout['user_id'].isin(ratings_train.user_id)].reset_index()
ratings_holdout_filtered = tmp_df[tmp_df['business_id'].isin(ratings_train.business_id)]
approach_1_start = time.time()

predict_df_1 = ratings_cv_filtered.loc[:, ['user_id', 'business_id', 'rating']]
predict_df_1['pred_rating'] = -1

for (u, b) in tqdm(predict_df_1.iterrows(), position=0, leave=True):
    try:
        pred = predict_approach_1(b.user_id, b.business_id)
        predict_df_1.loc[u, 'pred_rating'] = pred 
    except AttributeError:
        if sum(predict_df_1.pred_rating == -1) == 0:
            print('Finished')
        else:
            print('Error: Not Finised')
            print('Currently at u = {}'.format(u))
approach_1_duration = (time.time() - approach_1_start) * 1.0 / 60 
print('The approach 1 takes {} minutes to run.'.format(approach_1_duration))
predict_df_1.head()
predict_df_1.shape
# Remove rows without predictions
predict_df_1_NA_removed = predict_df_1.loc[~predict_df_1.pred_rating.isnull(), :]
predict_df_1_NA_removed.shape
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
mse, mae, r2 = mean_squared_error(predict_df_1_NA_removed.rating, predict_df_1_NA_removed.pred_rating), \
                mean_absolute_error(predict_df_1_NA_removed.rating, predict_df_1_NA_removed.pred_rating), \
                r2_score(predict_df_1_NA_removed.rating, predict_df_1_NA_removed.pred_rating)
print('MSE: {0:.4f}, MAE: {1:.4f}, R2: {2:.4f}'.format(mse, mae, r2))<jupyter_output>MSE: 2.1025, MAE: 1.1308, R2: 0.0218<jupyter_text>**Approach 2: (Implementation please see Appendix)**Build User Profile & Business Profile based on business categories information. As the algorithm takes too long to run, the code is attached in appendix.  (iii) Field-Aware Factorization Machine<jupyter_code># !pip install CMake
# Installation guide https://linxid.github.io/2018/05/10/Xlearn/
# !pip install xlearn
# !pip install regex
ratings_train_.head()
import xlearn as xl
import networkx as nx
pd.options.mode.chained_assignment = None
import regex as re
import math

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.cluster import KMeans
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
ratings_train.columns
def process(df):
    df['date']  = pd.to_datetime(df['date'])
    df['week_day'] = df['date'].dt.weekday
    df['month'] = df['date'].dt.month
    df['hour'] = df['date'].dt.hour
    df = df.merge(users_, on = 'user_id')
    df = df.merge(business_, on = 'business_id')
    rename_dict = {'business_longitude': 'longitude', 'business_latitude': 'latitude',
                  'business_state':'state','business_city':'city', 'business_address': 'address'}
    df = df.rename(columns = rename_dict)
    return df

ratings_train = process(ratings_train_.copy())
ratings_holdout = process(ratings_holdout_.copy())
ratings_val = process(ratings_cv_.copy())
def _convert_to_ffm(path, df, type, target, numerics, categories, features, encoder):
    # Flagging categorical and numerical fields
    print('convert_to_ffm - START')
    for x in numerics:
        if(x not in encoder['catdict']):
            print(f'UPDATING CATDICT: numeric field - {x}')
            encoder['catdict'][x] = 0
    for x in categories:
        if(x not in encoder['catdict']):
            print(f'UPDATING CATDICT: categorical field - {x}')
            encoder['catdict'][x] = 1

    nrows = df.shape[0]
    with open(path + str(type) + "_ffm.txt", "w") as text_file:

        # Looping over rows to convert each row to libffm format
        for n, r in enumerate(range(nrows)):
            datastring = ""
            datarow = df.iloc[r].to_dict()
            datastring += str(int(datarow[target]))  # Set Target Variable here

            # For numerical fields, we are creating a dummy field here
            for i, x in enumerate(encoder['catdict'].keys()):
                if(encoder['catdict'][x] == 0):
                    # Not adding numerical values that are nan
                    if math.isnan(datarow[x]) is not True:
                        datastring = datastring + " "+str(i)+":" + str(i)+":" + str(datarow[x])
                else:

                    # For a new field appearing in a training example
                    if(x not in encoder['catcodes']):
                        print(f'UPDATING CATCODES: categorical field - {x}')
                        encoder['catcodes'][x] = {}
                        encoder['currentcode'] += 1
                        print(f'UPDATING CATCODES: categorical value for field {x} - {datarow[x]}')
                        encoder['catcodes'][x][datarow[x]] = encoder['currentcode']  # encoding the feature

                    # For already encoded fields
                    elif(datarow[x] not in encoder['catcodes'][x]):
                        encoder['currentcode'] += 1
                        print(f'UPDATING CATCODES: categorical value for field {x} - {datarow[x]}')
                        encoder['catcodes'][x][datarow[x]] = encoder['currentcode']  # encoding the feature

                    code = encoder['catcodes'][x][datarow[x]]
                    datastring = datastring + " "+str(i)+":" + str(int(code))+":1"

            datastring += '\n'
            text_file.write(datastring)

    return encoder<jupyter_output><empty_output><jupyter_text>Feature Generation In the following steps, different sets of features will be generated as input to FFM.(i) Create user review **text embeddings** based on training data only<jupyter_code>rating_texts = ratings_train.groupby(['user_id'])['text'].apply(lambda x: ','.join(x)).reset_index()
rating_texts['text'] = rating_texts['text'].str.lower()
rating_texts['text'] = rating_texts['text'].str.replace(r"[^a-zA-Z ]+", " ").str.strip()

vectorizer = TfidfVectorizer(lowercase=True, stop_words = 'english', strip_accents = 'ascii')
vectorizer.fit(rating_texts['text'])
vector = vectorizer.transform(rating_texts['text'])
tsv = TruncatedSVD(n_components=50)
tsv.fit(vector)
transformed_tsv = tsv.transform(vector)

'''
wcss=[]
for i in range(77, 100):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(transformed_tsv)
    wcss.append(kmeans.inertia_)
plt.plot(range(2, 100), wcss)
plt.title('Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('WCSS')
plt.show()
'''

kmeans = KMeans(n_clusters=110, init='k-means++', max_iter=300, n_init=10, random_state=0)
kmeans.fit(transformed_tsv)
text_cluster = kmeans.predict(transformed_tsv)

rating_texts.loc[:,'text_cluster'] = text_cluster
rating_texts_features = rating_texts[['user_id','text_cluster']]
ratings_train= ratings_train.merge(rating_texts_features[['user_id','text_cluster']], on ='user_id', how = 'left')
ratings_holdout = ratings_holdout.merge(rating_texts_features[['user_id','text_cluster']], on ='user_id', how = 'left')
ratings_val = ratings_val.merge(rating_texts_features[['user_id','text_cluster']], on ='user_id', how = 'left')
ratings_train.head(1)<jupyter_output><empty_output><jupyter_text>(ii) Create **graph** features based on user friends attribute.<jupyter_code>train_users = ratings_train[['user_id','friends']].drop_duplicates()
train_users_dict = train_users.set_index('user_id').T.to_dict('list')

g = nx.Graph()
g.add_nodes_from(train_users_dict.keys())

for k, v in train_users_dict.items():
    for i in v[0].split(','):
        g.add_edge(k,i.strip())    

sub_graphs = nx.connected_component_subgraphs(g)

sgs =[]
for i, sg in enumerate(sub_graphs):
    sgs += [sg]
fin_sgs = []
for i in sgs:
    if len(i.nodes()) >=5:
        fin_sgs +=[i]

graph_user_ids, graph_ids = [],[]
num_id = 0
for graph in fin_sgs:
    for node in graph.nodes():
        graph_user_ids += [node]
        graph_ids += [num_id]
    num_id += 1
social_graphs = pd.DataFrame(
    {"user_id": graph_user_ids, "graph_cluster": graph_ids}
)

ratings_train= ratings_train.merge(social_graphs, on ='user_id', how = 'left')
ratings_holdout = ratings_holdout.merge(social_graphs, on ='user_id', how = 'left')
ratings_val = ratings_val.merge(social_graphs, on ='user_id', how = 'left')<jupyter_output><empty_output><jupyter_text>(iii) Create **Location** Features: business longitude and latitude<jupyter_code>X = ratings_train[['longitude','latitude']]

''' 
EXPLORATORY ANALYSIS TO DETERMINE NUMBER OF CLUSTERS. DON'T RUN
HERE USING THE EBLOW METHOD WE CHOOSE NCLUSTERs OF 10 

X = ratings_train[['longitude','latitude']]
wcss = []
for i in range(2, 100):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(X)
    wcss.append(kmeans.inertia_)
plt.plot(range(2, 50), wcss[0:48])
plt.title('Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('WCSS')
plt.show()
'''

kmeans = KMeans(n_clusters=10, init='k-means++', max_iter=300, n_init=10, random_state=0)
kmeans.fit(X)

ratings_train.loc[:,'location_cluster'] = kmeans.predict(ratings_train[['longitude','latitude']])
ratings_holdout.loc[:,'location_cluster'] = kmeans.predict(ratings_holdout[['longitude','latitude']])
ratings_val.loc[:,'location_cluster'] = kmeans.predict(ratings_val[['longitude','latitude']])

ratings_train.text_cluster.fillna('999', inplace=True)
ratings_holdout.text_cluster.fillna('999', inplace=True)
ratings_val.text_cluster.fillna('999', inplace=True)

ratings_train.location_cluster.fillna('999', inplace=True)
ratings_holdout.location_cluster.fillna('999', inplace=True)
ratings_val.location_cluster.fillna('999', inplace=True)

ratings_train.graph_cluster.fillna('999', inplace=True)
ratings_holdout.graph_cluster.fillna('999', inplace=True)
ratings_val.graph_cluster.fillna('999', inplace=True)
class FFMModel:

    def __init__(self, train, test, config, suffix = None):
        self.train_df = train
        self.test_df = test
        self.model = xl.create_ffm()
        self.suffix = suffix
        self.config = config
        self.preds = None 
          
    def __configure(self):
        destination = self.config['destination']
        label = self.config['label']
        numerical_columns  = self.config['numerical_columns']
        categorical_columns  = self.config['categorical_columns']
        all_columns  = numerical_columns + categorical_columns
              
        encoder = {"currentcode": len(self.config['numerical_columns']),
           "catdict": {},
           "catcodes": {}}
        
        encoder = _convert_to_ffm(destination, self.train_df , 'train', label, numerical_columns, \
                                  categorical_columns, all_columns, encoder)
        encoder = _convert_to_ffm(destination, self.test_df , 'test', label, numerical_columns, \
                                  categorical_columns, all_columns, encoder)
        
    def train(self, params = None):
        encoder = self.__configure()
        self.model.setTrain(self.config['destination']+'train_ffm.txt')
        self.model.setValidate(self.config['destination']+'test_ffm.txt')
        self.model.setTest(self.config['destination']+'test_ffm.txt')
        
        if not params:
            params = {'task': 'reg',
                     'lr': 0.2,
                     'lambda': 0.002,
                     'metric': 'auc'}
        
        self.model.fit(params, self.config['model_destination']+self.config['model_name']+'.out')

    def evaluate_on_val(self):
        self.model.predict(self.config['model_destination'] + self.config['model_name']+'.out', \
                           self.config['output_destination']+'predictions.txt')
        preds = pd.read_csv(self.config['output_destination']+'predictions.txt', sep=" ", header=None)
        preds.columns = ['prediction']
        preds.prediction = np.clip(preds.prediction,0.0,5.0)
        self.preds = preds
        return preds            
    
    def get_regression_metrics(self):
        if self.preds is None:
            self.evaluate_on_val()
        
        predictions = self.evaluate_on_val()
        test_df = self.test_df.copy()
        test_df['preds']  = np.clip(predictions.values,0.0,5.0)
        
        r2 = r2_score(test_df['rating'], test_df['preds'])
        mse = mean_squared_error(test_df['rating'], test_df['preds'])
        mae = mean_absolute_error(test_df['rating'], test_df['preds'])
        
        print('R2: ', r2, ' MSE: ',mse, ' MAE: ', mae)
        return r2,mse,mae<jupyter_output><empty_output><jupyter_text>Model Fitting and EvaluationNow run FFM on different combinations of features and calculate R2, mean squared error (MSE) and mean absolute error (MAE) on them. Metrics with just userid and business id<jupyter_code>baseline_config = {   'destination': 'data/',
    'categorical_columns':['business_id','user_id'],
    'numerical_columns' : [],
    'label' : 'rating',
    'model_destination' : 'trained_models/',
    'model_name' : 'baseline',
    'output_destination': 'output/'
}
baseline_FFM = FFMModel(ratings_train, ratings_val, baseline_config)
%%capture
baseline_FFM.train()
r2, mse, mae = baseline_FFM.get_regression_metrics()<jupyter_output>R2:  0.1873866119640475  MSE:  1.7830047851238464  MAE:  1.0886505079082125<jupyter_text>Metrics with Baseline + Business Location<jupyter_code>user_business_config = {   'destination': 'data/',
    'categorical_columns':['business_id','user_id','city','state'],
    'numerical_columns' : [],
    'label' : 'rating',
    'model_destination' : 'trained_models/',
    'model_name' : 'user_business',
    'output_destination': 'output/'
}
user_business_FFM = FFMModel(ratings_train, ratings_val, user_business_config)
%%capture
user_business_FFM.train()
r2, mse, mae = user_business_FFM.get_regression_metrics()<jupyter_output>R2:  0.1964585848489021  MSE:  1.763099414005971  MAE:  1.0615091977315227<jupyter_text>Metrics with Baseline + Location Cluster (has best MSE)<jupyter_code>location_config = {   'destination': 'data/',
    'categorical_columns':['business_id','user_id','location_cluster','city','state'],
    'numerical_columns' : [],
    'label' : 'rating',
    'model_destination' : 'trained_models/',
    'model_name' : 'location',
    'output_destination': 'output/'
}
location_FFM = FFMModel(ratings_train, ratings_val, location_config)
%%capture
location_FFM.train()
r2, mse, mae = location_FFM.get_regression_metrics()<jupyter_output>R2:  0.19744277000305088  MSE:  1.760939953104722  MAE:  1.0666155431063107<jupyter_text>Metrics with User and Business Information + Text Cluster<jupyter_code>text_config = {   'destination': 'data/',
    'categorical_columns':['business_id','user_id','text_cluster','city','state'],
    'numerical_columns' : [],
    'label' : 'rating',
    'model_destination' : 'trained_models/',
    'model_name' : 'text',
    'output_destination': 'output/'
}
text_FFM = FFMModel(ratings_train, ratings_val, text_config)
%%capture
text_FFM.train()
r2, mse, mae = text_FFM.get_regression_metrics()<jupyter_output>R2:  0.18948085767652612  MSE:  1.778409672390595  MAE:  1.064447769604502<jupyter_text>Metrics with User and Business Information + Graph Cluster<jupyter_code>graph_config = {   'destination': 'data/',
    'categorical_columns':['business_id','user_id','graph_cluster','city','state'],
    'numerical_columns' : [],
    'label' : 'rating',
    'model_destination' : 'trained_models/',
    'model_name' : 'graph',
    'output_destination': 'output/'
}
graph_FFM = FFMModel(ratings_train, ratings_val, graph_config)
%%capture
graph_FFM.train()
r2, mse, mae = graph_FFM.get_regression_metrics()<jupyter_output>R2:  0.19572440525321522  MSE:  1.7647103224053686  MAE:  1.072474083328964<jupyter_text>Metrics All clusters<jupyter_code>all_config = {   'destination': 'data/',
    'categorical_columns':['business_id','user_id','graph_cluster','location_cluster','text_cluster'],
    'numerical_columns' : [],
    'label' : 'rating',
    'model_destination' : 'trained_models/',
    'model_name' : 'all_clusters',
    'output_destination': 'output/'
}
all_FFM = FFMModel(ratings_train, ratings_val, all_config)
%%capture
all_FFM.train()
r2, mse, mae = all_FFM.get_regression_metrics()<jupyter_output>R2:  0.1862330235045122  MSE:  1.7855359441887817  MAE:  1.065033877918174<jupyter_text>**Observation**: The baseline with location cluster performs best as it has lowest MSE. Therefore, this model will be used. The following analysis is done on this model.  Run Time Analysis FFM runs in linear time which leads to fast runtime. Our model takes just two minutes to run.<jupyter_code>%%capture
import time
start_time = time.time()
location_FFM = FFMModel(ratings_train, ratings_val, location_config)
location_FFM.train()
print("--- %s seconds ---" % (time.time() - start_time))
%%capture
start_time = time.time()
location_FFM.get_regression_metrics()
print("--- %s seconds ---" % (time.time() - start_time))<jupyter_output>--- 0.16828489303588867 seconds ---<jupyter_text>Coverage and Serendipity metrics<jupyter_code>predictions = location_FFM.evaluate_on_val()
predict_df['predictions'] = predictions['prediction']
top_10, coverage, serendipity, avg_rank = get_all_metrics(predict_df, out, ratings_train_final)<jupyter_output>0.011627906976744186 0.8576470588235293 0.9990697674418605 72.29767441860466<jupyter_text>(iv) Deep Learning models<jupyter_code>!pip install -U deepctr[cpu]<jupyter_output>Collecting deepctr[cpu]
  Downloading https://files.pythonhosted.org/packages/cb/46/2a3291f804c56ba0e5ba31d6c825c23bb1a2c7d8f6f066db28f8ed5f374a/deepctr-0.7.0-py3-none-any.whl (79kB)
[K    100% |████████████████████████████████| 81kB 2.6MB/s ta 0:00:011
[?25hCollecting requests (from deepctr[cpu])
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
[K    100% |████████████████████████████████| 61kB 10.2MB/s ta 0:00:01
[?25hCollecting h5py (from deepctr[cpu])
  Downloading https://files.pythonhosted.org/packages/60/06/cafdd44889200e5438b897388f3075b52a8ef01f28a17366d91de0fa2d05/h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9MB)
[K    100% |████████████████████████████████| 2.9MB 418kB/s eta 0:00:01
[?25hRequirement already up-to-date: tensorflow!=1.7.*,!=1.8.*,>=1.4.0; extra == "cpu" in /home/jzljohn18/anaconda3/lib/python3.6/site-packages (from deepctr[cpu])
Collecti[...]<jupyter_text>Deep Learning Models DeepFM![DeepFM Model Architecture](https://d2l.ai/_images/rec-deepfm.svg)It is a model which integrates the feature representation learning of a neural network with factorization machines.Adding nonlinear transformation layers to factorization machines gives it the capability to model both low-order feature combinations and high-order feature combinations. Moreover, non-linear inherent structures from inputs can also be captured with neural networks. Wide and Deep Learning![Wide and Deep Model](https://2.bp.blogspot.com/-wkrmRibw_GM/V3Mg3O3Q0-I/AAAAAAAABG0/Jm3Nl4-VcYIJ44dA5nSz6vpTyCKF2KWQgCKgB/s640/image03.png)The Wide part of the model tries to capture the co-occurrence of a query-item feature pair correlates with the target label. The Deep model generalizes the query-item interactions.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')#, force_remount=True)
!pip install -U deepctr[gpu]
!pip install -U scikit-learn<jupyter_output>Requirement already up-to-date: deepctr[gpu] in /usr/local/lib/python3.6/dist-packages (0.7.0)
Requirement already satisfied, skipping upgrade: h5py in /usr/local/lib/python3.6/dist-packages (from deepctr[gpu]) (2.8.0)
Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from deepctr[gpu]) (2.21.0)
Requirement already satisfied, skipping upgrade: tensorflow-gpu!=1.7.*,!=1.8.*,>=1.4.0; extra == "gpu" in /usr/local/lib/python3.6/dist-packages (from deepctr[gpu]) (2.0.0)
Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from h5py->deepctr[gpu]) (1.12.0)
Requirement already satisfied, skipping upgrade: numpy>=1.7 in /usr/local/lib/python3.6/dist-packages (from h5py->deepctr[gpu]) (1.17.4)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->deepctr[gpu]) (2.8)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1[...]<jupyter_text>The GPU being used for the deep learning models is a Tesla P100<jupyter_code>!nvidia-smi<jupyter_output>Fri Dec 20 14:14:03 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P0    25W / 250W |      0MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU  [...]<jupyter_text>Importing packages and reading the dataset ...<jupyter_code>## data handling
# setup libraries and env
import os
import shutil
import sys

import numpy as np
from scipy import sparse

import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sn
sn.set()

import pandas as pd
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

from deepctr.models import DeepFM, CCPM, FNN, PNN, WDL, MLR, NFM, AFM, DCN, DIN, DIEN, DSIN, xDeepFM, AutoInt, NFFM, FGCNN, FiBiNET
from deepctr.inputs import SparseFeat,get_feature_names

import itertools as it

training =  pd.read_csv('/content/drive/My Drive/final_project_datasets/ratings_sample_train_20.csv', index_col = 0)
validation = pd.read_csv('/content/drive/My Drive/final_project_datasets/ratings_sample_validation_20.csv', index_col = 0)
test = pd.read_csv('/content/drive/My Drive/final_project_datasets/ratings_sample_test_20.csv', index_col = 0)

businesses = pd.read_csv('/content/drive/My Drive/final_project_datasets/businesses.csv')
users = pd.read_csv('/content/drive/My Drive/final_project_datasets/active_users.csv')
# training.dropna(inplace = True)
# validation.dropna(inplace = True)
# test.dropna(inplace = True)<jupyter_output><empty_output><jupyter_text>Include cities as a feature in the deep learning models<jupyter_code>businesses['business_city_state'] = businesses['business_city'] + businesses['business_state']
print(training.shape, validation.shape, test.shape)
training = training.merge(right = businesses[['business_id', 'business_city_state']], how = 'left', on = 'business_id')
validation = validation.merge(right = businesses[['business_id', 'business_city_state']], how = 'left', on = 'business_id')
test = test.merge(right = businesses[['business_id', 'business_city_state']], how = 'left', on = 'business_id')
column_sequence = ['user_id', 'business_id', 'business_city_state', 'text', 'rating', 'date']
training = training[column_sequence]
validation = validation[column_sequence]
test = test[column_sequence]
# convert object to datetime
# training.date = pd.to_datetime(training.date)
# validation.date = pd.to_datetime(validation.date)
# test.date = pd.to_datetime(test.date)

# find hour from datetime
# training['hour'] = training.date.dt.hour
# validation['hour'] = validation.date.dt.hour
# test['hour'] = test.date.dt.hour

test.head()<jupyter_output><empty_output><jupyter_text>Data Quality check<jupyter_code># quality check
print(len(set(test.user_id) - set(training.user_id)))
print(len(set(validation.user_id) - set(training.user_id)))
test = test.loc[test.business_id.isin(training.business_id)]
validation = validation.loc[validation.business_id.isin(training.business_id)]
# map each user_id, business_id to an index
# user_mapping = {}
# for n,i in enumerate(training.user_id.unique()):
#   user_mapping[i] = n

# business_mapping = {}
# for n,i in enumerate(training.business_id.unique()):
#   business_mapping[i] = n
# for training
# training['user_id'] = training['user_id'].map(user_mapping)
# training['business_id'] = training['business_id'].map(business_mapping)
# for validation
# validation['user_id'] = validation['user_id'].map(user_mapping)
# validation['business_id'] = validation['business_id'].map(business_mapping)
# for test
# test['user_id'] = test['user_id'].map(user_mapping)
# test['business_id'] = test['business_id'].map(business_mapping)
test.head()
# validation = validation.loc[~validation.business_id.isin(['WpC53SqwoCY5AuYIFr_1eA'])]<jupyter_output><empty_output><jupyter_text>Preparation for input into deep learning models<jupyter_code># 1.Label Encoding for sparse features,and do simple Transformation for dense features
training = training.loc[training.business_city_state.apply(type) != float]
training_deep = training.copy()
validation_deep = validation.copy()
test_deep = test.copy()

sparse_features = ["user_id", "business_id", "business_city_state"]#, "hour"]
target = ['rating']
for feat in sparse_features:
  lbe = LabelEncoder()
  training_deep[feat] = lbe.fit_transform(training_deep[feat])
  validation_deep[feat] = lbe.transform(validation_deep[feat])
  test_deep[feat] = lbe.transform(test_deep[feat])<jupyter_output><empty_output><jupyter_text>Grid Search - Hyperparameter TuningWe have tuned 3 parameters for both the models,1. Embedding dimension - 8, 16, 322. Hidden Units - (128, 128), (256, 256), (256, 128)3. Dropout - 0.1, 0.3, 0.5  Grid Search for DeepFM<jupyter_code>params = {
    'embedding_dim' : [8, 16, 32],
    'dnn_hidden_units': [(128, 128), (256, 256), (256, 128)],
    'dnn_dropout': [0.1, 0.3, 0.5]
}
allNames = sorted(params)
combinations = it.product(*(params[Name] for Name in allNames))

best_params = None
best_mse = 1000

# grid search for DeepFM
for i in list(combinations):
  dropout, dnn_hidden_units, embedding_dim = i

  # 2.count #unique features for each sparse field
  fixlen_feature_columns = [SparseFeat(feat, training_deep[feat].nunique(), embedding_dim = embedding_dim) for feat in sparse_features]
  linear_feature_columns = fixlen_feature_columns
  dnn_feature_columns = fixlen_feature_columns
  feature_names = get_feature_names(linear_feature_columns + dnn_feature_columns)
  
  # 3.define model inputs   
  train_model_input = {name:training_deep[name].values for name in feature_names}
  valid_model_input = {name:validation_deep[name].values for name in feature_names}
  test_model_input = {name:test_deep[name].values for name in feature_names}

  # 4.Define Model,train,predict and evaluate
  model = DeepFM(linear_feature_columns, dnn_feature_columns, fm_group=['business_city_state'], task='regression', dnn_hidden_units = dnn_hidden_units, dnn_dropout = dropout, l2_reg_embedding=1e-05, l2_reg_dnn=1e-05, l2_reg_linear=1e-05, seed = 42)
  model.compile("adam", "mse", metrics=['mse'], )
  # only one epoch because it overfits after the first epoch
  history = model.fit(train_model_input, training_deep[target].values, batch_size=256, epochs=1, verbose=2, validation_data= (valid_model_input, validation[target].values))

  validation_predictions = model.predict(valid_model_input, batch_size=256)
  val_mse = mean_squared_error(validation_deep[target].values, validation_predictions)
  print("validation MSE", round(val_mse, 4))
  if val_mse < best_mse:
    best_mse = val_mse
    best_params = [dropout, dnn_hidden_units, embedding_dim]
best_mse, best_params<jupyter_output><empty_output><jupyter_text>Grid Search for WDL<jupyter_code>params = {
    'embedding_dim' : [8, 16, 32],
    'dnn_hidden_units': [(128, 128), (256, 256), (256, 128)],
    'dnn_dropout': [0.1, 0.3, 0.5]
}
allNames = sorted(params)
combinations = it.product(*(params[Name] for Name in allNames))

best_params_wdl = None
best_mse_wdl = 1000

# grid search for WDL
for i in list(combinations):
  dropout, dnn_hidden_units, embedding_dim = i

  # 2.count #unique features for each sparse field
  fixlen_feature_columns = [SparseFeat(feat, training_deep[feat].nunique(), embedding_dim = embedding_dim) for feat in sparse_features]
  linear_feature_columns = fixlen_feature_columns
  dnn_feature_columns = fixlen_feature_columns
  feature_names = get_feature_names(linear_feature_columns + dnn_feature_columns)
  
  # 3.define model inputs   
  train_model_input = {name:training_deep[name].values for name in feature_names}
  valid_model_input = {name:validation_deep[name].values for name in feature_names}
  test_model_input = {name:test_deep[name].values for name in feature_names}

  # 4.Define Model,train,predict and evaluate
  model = WDL(linear_feature_columns, dnn_feature_columns, task='regression', dnn_hidden_units = dnn_hidden_units, dnn_dropout = dropout, l2_reg_embedding=1e-05, l2_reg_dnn=1e-05, l2_reg_linear=1e-05, seed = 42)
  model.compile("adam", "mse", metrics=['mse'], )
  # only one epoch because it overfits after the first epoch
  history = model.fit(train_model_input, training_deep[target].values, batch_size=256, epochs=1, verbose=2, validation_data= (valid_model_input, validation[target].values))

  validation_predictions = model.predict(valid_model_input, batch_size=256)
  val_mse = mean_squared_error(validation_deep[target].values, validation_predictions)
  print("validation MSE", round(val_mse, 4))
  if val_mse < best_mse_wdl:
    best_mse_wdl = val_mse
    best_params_wdl = [dropout, dnn_hidden_units, embedding_dim]
best_mse_wdl, best_params_wdl<jupyter_output><empty_output><jupyter_text>Refitting the model on the best parameters for each of the models<jupyter_code># 1.Label Encoding for sparse features,and do simple Transformation for dense features
training_combined = pd.concat([training, validation], axis = 0)
training_combined_deep = training_combined.copy()
test_deep = test.copy()

sparse_features = ["user_id", "business_id", "business_city_state"]#, "hour"]
target = ['rating']

for feat in sparse_features:
  lbe = LabelEncoder()
  training_combined_deep[feat] = lbe.fit_transform(training_combined_deep[feat])
  test_deep[feat] = lbe.transform(test_deep[feat])

# best DeepFM Model
dropout_deepfm = 0.1
dnn_hidden_units_deepfm = (128, 128)
embedding_dim_deepfm = 8

# 2.count #unique features for each sparse field
fixlen_feature_columns = [SparseFeat(feat, training_combined_deep[feat].nunique(), embedding_dim = embedding_dim_deepfm) for feat in sparse_features]
linear_feature_columns = fixlen_feature_columns
dnn_feature_columns = fixlen_feature_columns
feature_names = get_feature_names(linear_feature_columns + dnn_feature_columns)

# 3.define model inputs   
train_model_input = {name:training_combined_deep[name].values for name in feature_names}
test_model_input = {name:test_deep[name].values for name in feature_names}

# 4.Define Model,train,predict and evaluate
model_deepfm = DeepFM(linear_feature_columns, dnn_feature_columns, fm_group=['business_city_state'], task='regression', dnn_hidden_units = dnn_hidden_units_deepfm, dnn_dropout = dropout_deepfm, l2_reg_embedding=1e-05, l2_reg_dnn=1e-05, l2_reg_linear=1e-05, seed = 42)
model_deepfm.compile("adam", "mse", metrics=['mse'], )
# only one epoch because it overfits after the first epoch
history = model_deepfm.fit(train_model_input, training_combined_deep[target].values, batch_size=256, epochs=1, verbose=2)

test_predictions_deepfm = model_deepfm.predict(test_model_input, batch_size=255)
test_mse_deepfm = mean_squared_error(test_deep[target].values, test_predictions_deepfm)


# best WDL Model
dropout_wdl = 0.3
dnn_hidden_units_wdl = (256, 256)
embedding_dim_wdl = 8

# 2.count #unique features for each sparse field
fixlen_feature_columns = [SparseFeat(feat, training_combined_deep[feat].nunique(), embedding_dim = embedding_dim_wdl) for feat in sparse_features]
linear_feature_columns = fixlen_feature_columns
dnn_feature_columns = fixlen_feature_columns
feature_names = get_feature_names(linear_feature_columns + dnn_feature_columns)

# 3.define model inputs   
train_model_input = {name:training_combined_deep[name].values for name in feature_names}
test_model_input = {name:test_deep[name].values for name in feature_names}

# 4.Define Model,train,predict and evaluate
model_wdl = WDL(linear_feature_columns, dnn_feature_columns, task='regression', dnn_hidden_units = dnn_hidden_units_wdl, dnn_dropout = dropout_wdl, l2_reg_embedding=1e-05, l2_reg_dnn=1e-05, l2_reg_linear=1e-05, seed = 42)
model_wdl.compile("adam", "mse", metrics=['mse'], )
# only one epoch because it overfits after the first epoch
history = model_wdl.fit(train_model_input, training_combined_deep[target].values, batch_size=256, epochs=1, verbose=2)

test_predictions_wdl = model_wdl.predict(test_model_input, batch_size=256)
test_mse_wdl = mean_squared_error(test_deep[target].values, test_predictions_wdl)
test_mse_deepfm, test_mse_wdl<jupyter_output><empty_output><jupyter_text>R2 Score<jupyter_code>print("R2 Score: %0.3f" %r2_score(y_true = test_deep[target], y_pred = test_predictions_deepfm))
print("R2 Score: %0.3f" %r2_score(y_true = test_deep[target], y_pred = test_predictions_wdl))<jupyter_output>R2 Score: 0.204
R2 Score: 0.203<jupyter_text>Mean Absolute Error<jupyter_code>print("Mean Absolute Error: %0.3f" %mean_absolute_error(y_true = test_deep[target], y_pred = test_predictions_deepfm))
print("Mean Absolute Error: %0.3f" %mean_absolute_error(y_true = test_deep[target], y_pred = test_predictions_wdl))<jupyter_output>Mean Absolute Error: 1.082
Mean Absolute Error: 1.079<jupyter_text>Mean Squared Error<jupyter_code>print("Root Mean Square Error: %0.3f" %mean_squared_error(y_true = test_deep[target], y_pred = test_predictions_deepfm, squared = False))
print("Root Mean Square Error: %0.3f" %mean_squared_error(y_true = test_deep[target], y_pred = test_predictions_wdl, squared = False))<jupyter_output>Root Mean Square Error: 1.335
Root Mean Square Error: 1.336<jupyter_text>Rest of the metrics<jupyter_code>def process(df):
    # df = df.drop(df.columns[0], axis =1)
    df['date']  = pd.to_datetime(df['date'])
    df['week_day'] = df['date'].dt.weekday
    df['month'] = df['date'].dt.month
    df['hour'] = df['date'].dt.hour
    df = df.merge(users, on = 'user_id')
    df = df.merge(businesses, on = 'business_id')
    return df
ratings_train = process(training.copy())
ratings_validation = process(validation.copy())
ratings_test = process(test.copy())
ratings_train_final = ratings_train.append(ratings_validation)
unique_city_businesses = ratings_train_final[['business_city','business_id']].drop_duplicates()
unique_cities = unique_city_businesses.groupby('business_city').count()['business_id']
unique_cities = unique_cities[unique_cities > 100]
out = pd.DataFrame()
for city in unique_cities.index:
    tmp = ratings_train_final[(ratings_train_final['business_city'] ==city) &
                              (ratings_train_final['rating'] >ratings_train_final['average_stars'])]
    if len(tmp['user_id'].unique())>4:
        np.random.seed(42)
        ###this weird sampling technique is to ensure we dont' sample the same user twice in a same city
        five_users = np.random.choice(tmp['user_id'].unique(),5, replace = False)
        row = tmp[tmp['user_id'].isin(five_users)].groupby('user_id', group_keys=False).apply(lambda df: df.sample(1))
        out = out.append(row)
all(out.groupby('business_city').count()['user_id']==5)
predict_df = out[['user_id','business_city','business_state']]
predict_df = predict_df.merge(unique_city_businesses, on = 'business_city')
all(predict_df.groupby('business_city')['user_id'].nunique()==5)
# remove businesses not in training
predict_df = predict_df.loc[predict_df.business_id.isin(training.business_id)]
predict_df[['user_id', 'business_id']]
predict_df['business_city_state'] = predict_df['business_city'] + predict_df['business_state']
metric_test_deep = predict_df[['user_id', 'business_id', 'business_city_state']].copy()

for feat in sparse_features:
  lbe = LabelEncoder()
  lbe.fit(training[feat])
  metric_test_deep[feat] = lbe.transform(metric_test_deep[feat])
  
metric_test_input = {name:metric_test_deep[name].values for name in feature_names}
metric_test_predictions_deepfm = model_deepfm.predict(metric_test_input, batch_size=256)
metric_test_predictions_wdl = model_wdl.predict(metric_test_input, batch_size=256)
predict_df['predictions'] = metric_test_predictions_wdl
top_10_recs = predict_df.groupby(['user_id','business_city'])['predictions'].nlargest(10).reset_index()
all(top_10_recs.groupby('business_city')['user_id'].count()==50)
cnt =0
serendipity = 0
for row in out.iterrows():
    row_values = row[1]
    top_10 = predict_df.loc[top_10_recs[top_10_recs['user_id'] == row_values['user_id']].level_2]['business_id']
    ###In top 10
    if row_values['business_id'] in top_10.values:
        cnt+=1
    user_history = ratings_train_final[ratings_train_final['user_id'] == row_values['user_id']]    
    been_there = [i for i in top_10.values if i in  user_history.business_id.values]
    serendipity += 1-len(been_there)/10<jupyter_output><empty_output><jupyter_text>Inclusion of Last Review in Top 10 Recommendations<jupyter_code>cnt/len(out)<jupyter_output><empty_output><jupyter_text>Novelty(% of new restaurants in top 10 recommendations)<jupyter_code>serendipity/len(out)
predict_df = predict_df.reset_index()
predict_df.columns
top_10_recs.head(1)
predict_df.head(1)
analysis_df = predict_df.merge(top_10_recs, left_on = ['user_id','business_city','index'], right_on = ['user_id','business_city','level_2'])
all(analysis_df.groupby('business_city')['business_id'].count() ==50)<jupyter_output><empty_output><jupyter_text>Coverage(% of unique recommendations)<jupyter_code>(analysis_df.groupby('business_city')['business_id'].nunique()/50).values.mean()
predict_df['rankings']=predict_df.groupby(['business_city','user_id'])['predictions'].rank("first",ascending = False)
running_rankings =0
for row in out.iterrows():
    row_values = row[1]
    user_recs = predict_df[(predict_df['user_id']==row_values['user_id'])
                        &(predict_df['business_city']==row_values['business_city'])
                         & (predict_df['business_id']==row_values['business_id'])
                          ]
    assert len(user_recs)==1
    running_rankings += user_recs['rankings'].sum()<jupyter_output><empty_output><jupyter_text>Average Ranking of Last Positive Review<jupyter_code>running_rankings / len(out)<jupyter_output><empty_output><jupyter_text>Conclusion and Result ComparisonPlease refer to the report!  Appendix<jupyter_code># Select restaurants in training set
print(business.shape)
train_business_id = np.unique(ratings_train.business_id)
business = business.loc[business.business_id.isin(train_business_id), :]
print(business.shape)

### Business Profile

tmp_series = pd.Series(business.category.str.split(', ').tolist(), index=business.business_id)
business_df = pd.get_dummies(tmp_series.apply(pd.Series).stack()).sum(level=0).reset_index()

business_df.head()

business_df_normalized = business_df.set_index('business_id') \
                            .div(business_df.set_index('business_id').sum(axis=1)**0.5, axis=0)

business_df_normalized.sort_values('business_id', inplace=True)

business_df_normalized.head()

### User Profile

# check if all restaurants that users have rated exist in restuarant profile
print('The number of all rated restaurants: {}'.format(len(np.unique(ratings_train[['business_id']]))))
print('The number of resturants existing profile: {}'.format(len(np.unique(business_df.business_id))))

# **Caveat**: Even for a subsample (ratings_train is a subsample of full ratings dataset), there are restaurants rated by users but not **do not exist** in business profile dataset. This needs to be handled when building recommender system. 

# Build uer profile
users_id = np.unique(ratings_train.user_id)
users_df = pd.DataFrame(columns = business_df_normalized.columns)

approach_2_start = time.time()
# create a for loop to create profile vector for each user
for i in tqdm(range(len(users_id))):
    selected_user_id = users_id[i]
    # select business that user has rated
    selected_rating = ratings_train.loc[ratings_train.user_id == selected_user_id, ['business_id', 'rating']]. \
                                        set_index('business_id')
    ind = selected_rating.index
    # To calculate user profile vector, 
    # multiply weight (normalized frequency) by ratings across all rated business and take mean of the result
    try: 
        # only need to extract rated restaurants because the rest of term will be 
        w = business_df_normalized.loc[ind, :]
    except KeyError:
        business_id_existed_profile = business_df.business_id
        filtered_ind = pd.Index([j for j in ind if j in business_id_existed_profile.tolist()])    
        w = business_df_normalized.loc[filtered_ind, :]
    except:
        print('Some other errors occured during iteration i = {}'.format(i))
    if (w.shape[0] == 0):
        users_df.loc[selected_user_id] = 0
    else:
        profile_vec = w.mul(selected_rating.loc[:, 'rating'], axis=0).mean(axis=0)
        users_df.loc[selected_user_id] = profile_vec
        
approach_2_duration = (time.time() - approach_2_start) * 1.0 / 60 
print('The approach 2 takes {} minutes to run.'.format(approach_2_duration)) 

users_df.shape

### CHECK
# IDF vector  
DF = business_df_normalized.sum(axis=0)
IDF = (business_df.shape[0]/DF).apply(np.log) #log inverse of DF
# The dot product of business profile vector and IDF vector gives
# weighted score of each business
IDF_business_df_normalized = business_df_normalized.mul(IDF.values)
IDF_business_df_normalized.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11885774792362927, 'fraction_numerical': 0.03157848295335473, 'mean_word_length': 4.857225918987015, 'pattern_counts': {'":': 15, '<': 199, '<?xml version=': 0, '>': 208, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 5, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '53', 'dir': 'jupyter-structured-clean-dedup', 'id': '377659', 'license': 'MIT', 'n_tokens_mistral': 31546, 'n_tokens_neox': 29349, 'n_words': 7692, 'path': 'yelp_rating_master_notebook.ipynb', 'repo_name': 'yjs1210/yelp-recommender', 'size': '90683'}