<jupyter_start><jupyter_text>Restaurant Recommendation System Notebook by [<NAME>]("https://github.com/xuwenyihust") My own solution to Harvard CS 109 HW4 Import libraries & some preparation works<jupyter_code>%matplotlib inline
from collections import defaultdict
import json

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import pandas as pd

from matplotlib import rcParams
import matplotlib.cm as cm
import matplotlib as mpl
from operator import itemgetter

#colorbrewer2 Dark2 qualitative color table
dark2_colors = [(0.10588235294117647, 0.6196078431372549, 0.4666666666666667),
                (0.8509803921568627, 0.37254901960784315, 0.00784313725490196),
                (0.4588235294117647, 0.4392156862745098, 0.7019607843137254),
                (0.9058823529411765, 0.1607843137254902, 0.5411764705882353),
                (0.4, 0.6509803921568628, 0.11764705882352941),
                (0.9019607843137255, 0.6705882352941176, 0.00784313725490196),
                (0.6509803921568628, 0.4627450980392157, 0.11372549019607843)]

rcParams['figure.figsize'] = (10, 6)
rcParams['figure.dpi'] = 150
#rcParams['axes.prop_cycle'] = dark2_colors
rcParams['lines.linewidth'] = 2
rcParams['axes.facecolor'] = 'white'
rcParams['font.size'] = 14
rcParams['patch.edgecolor'] = 'white'
rcParams['patch.facecolor'] = dark2_colors[0]
rcParams['font.family'] = 'StixGeneral'


def remove_border(axes=None, top=False, right=False, left=True, bottom=True):
    """
    Minimize chartjunk by stripping out unnecesasry plot borders and axis ticks
    
    The top/right/left/bottom keywords toggle whether the corresponding plot border is drawn
    """
    ax = axes or plt.gca()
    ax.spines['top'].set_visible(top)
    ax.spines['right'].set_visible(right)
    ax.spines['left'].set_visible(left)
    ax.spines['bottom'].set_visible(bottom)
    
    #turn off all ticks
    ax.yaxis.set_ticks_position('none')
    ax.xaxis.set_ticks_position('none')
    
    #now re-enable visibles
    if top:
        ax.xaxis.tick_top()
    if bottom:
        ax.xaxis.tick_bottom()
    if left:
        ax.yaxis.tick_left()
    if right:
        ax.yaxis.tick_right()
        
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)<jupyter_output><empty_output><jupyter_text>Load the dataset<jupyter_code>fulldf=pd.read_csv("https://raw.githubusercontent.com/xuwenyihust/Restaurant-Recommendation-System/master/bigdf.csv")
fulldf.head(2)<jupyter_output><empty_output><jupyter_text>Q1. Writing a simple "global" recommenderAssume that every users perceive the similarity between items in the same way. (**global similarity** instead of user-specific similarity)Calculate the similarity between items by their entire common user support.**common user support:** use the set of common raters of both items to measure the similarity between items. Exploratory Data Analysis**1.1** Visualize the sparsity of the full data set by plotting two histograms of the review count grouped by the user_id and business_id respectively. Are there more users or more businesses?We can see from the plots that users with 0~3 reviews are the most,and that businesses with 0~5 reviews are the most.<jupyter_code># Number of review_ids for each user_id
review_per_user = fulldf.groupby(["user_id"])["review_id"].count()
print(review_per_user[:5])

bin_cnt = len(set(review_per_user.tolist()))

fig, ax = plt.subplots()
ax.set(xlabel="Review count", ylabel="User count", title="Distribution of review count per user")
remove_border(ax)
#ax.axis('off')
plt.hist(review_per_user, bins=int(bin_cnt/3), log=True, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()
# Number of review_ids for each business_id
review_per_business = fulldf.groupby(["business_id"])["review_id"].count()
print(review_per_business[:5])

bin_cnt = len(set(review_per_business.tolist()))

fig, ax = plt.subplots()
ax.set(xlabel="Review count", ylabel="Business count", title="Distribution of review count per business")
remove_border(ax)
#ax.axis('off')
plt.hist(review_per_business, bins=int(bin_cnt/5), log=True, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()<jupyter_output>business_id
--5jkZ3-nUPZxUvtcbr8Uw    11
--BlvDO_RG2yElKu9XA1_g    36
-0QBrNvhrPQCaeo7mTo0zQ     3
-0bUDim5OGuv8R0Qqq6J4A     6
-1bOb2izeJBZjHC7NWxiPA    47
Name: review_id, dtype: int64<jupyter_text>Compare the number of users & businesses.<jupyter_code>user_cnt = len(set(fulldf["user_id"].tolist()))
business_cnt = len(set(fulldf["business_id"].tolist()))
print("More users than businesses.") if user_cnt > business_cnt else print("More users than businesses.")<jupyter_output>More users than businesses.<jupyter_text>**1.2** Compute the average rating of reviews in the data set and a histogram of all the ratings in the dataset.<jupyter_code># The average rating of all reviews
avg_rating = fulldf["stars"].mean()
print("Average rating: ",avg_rating)

# Histogram of all the ratings
fig, ax = plt.subplots()
ax.set(xlabel="Rating", ylabel="Count", title="Distribution of ratings")
remove_border(ax)
plt.hist(fulldf["stars"], bins=5, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()<jupyter_output>Average rating:  3.7414126802349332<jupyter_text>The following function is used to re-compute review counts and averages whenever you subset a reviews data frame. We'll use it soon to construct a smaller, more computationally tractable data frame.<jupyter_code>def recompute_frame(ldf):
    """
    takes a dataframe ldf, makes a copy of it, and returns the copy
    with all averages and review counts recomputed
    this is used when a frame is subsetted.
    """
    ldfu=ldf.groupby('user_id')
    ldfb=ldf.groupby('business_id')
    user_avg=ldfu.stars.mean()
    user_review_count=ldfu.review_id.count()
    business_avg=ldfb.stars.mean()
    business_review_count=ldfb.review_id.count()
    nldf=ldf.copy()
    nldf.set_index(['business_id'], inplace=True)
    nldf['business_avg']=business_avg
    nldf['business_review_count']=business_review_count
    nldf.reset_index(inplace=True)
    nldf.set_index(['user_id'], inplace=True)
    nldf['user_avg']=user_avg
    nldf['user_review_count']=user_review_count
    nldf.reset_index(inplace=True)
    return nldf<jupyter_output><empty_output><jupyter_text>**1.3** Create a smaller data set in dataframe smalldf by looking for those businesses with more than 150 reviews and those users with more than 60 reviews. Include all the columns that were there in the parent dataframe. Since you have created a subset of the data set, use the method provided above to recalculate the averages. Print the number of unique users and items in this data set.Note that while this cut makes sure we have prolific users, the cut on businesses restores sparsity by reducing the number of reviews per user.<jupyter_code># This can be easily done cause tese information have already been included as columns
smalldf=fulldf[(fulldf.user_review_count > 60) & (fulldf.business_review_count > 150)]
smalldf=recompute_frame(smalldf)<jupyter_output><empty_output><jupyter_text>How does this compare to the parent data set, in terms of size and sparsity? Once again, plot histograms of the review count grouped by user, and by the review count grouped by business, respectively, and describe the results<jupyter_code>review_per_user = smalldf.groupby(["user_id"])["review_id"].count()
print(review_per_user[:5])

bin_cnt = len(set(review_per_user.tolist()))

fig, ax = plt.subplots()
ax.set(xlabel="Review count", ylabel="User count", title="Distribution of review count per user")
remove_border(ax)
#ax.axis('off')
plt.hist(review_per_user, bins=int(bin_cnt/5), log=False, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()
review_per_business = smalldf.groupby(["business_id"])["review_id"].count()
print(review_per_business[:5])

bin_cnt = len(set(review_per_business.tolist()))

fig, ax = plt.subplots()
ax.set(xlabel="Review count", ylabel="Business count", title="Distribution of review count per business")
remove_border(ax)
#ax.axis('off')
plt.hist(review_per_business, bins=int(bin_cnt/5), log=False, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()<jupyter_output>business_id
-4A5xmN21zi_TXnUESauUQ    25
-AAig9FG0s8gYE4f8GfowQ    43
-bd26a1QEEpqUZjBmtBUiQ    26
-sC66z4SO3tR7nFCjfQwuQ    79
0K634ehk41ZRc4kj3NTAEQ    25
Name: review_id, dtype: int64<jupyter_text>After the dataset filtering, the size of the new dataset is smaller, and the **sparsity is becoming less** too.**1.4** Compute histograms of the average user rating in the smaller data set, and the average business rating in the smaller data set. Print the overall mean.<jupyter_code># Average user rating
fig, ax = plt.subplots()
ax.set(xlabel="Average user rating", ylabel="Count", title="Distribution of average user rating")
remove_border(ax)

avg_ratings_by_user=smalldf.groupby('user_id').stars.mean()
print("avg_ratings_by_user: ", avg_ratings_by_user[:5])
plt.hist(avg_ratings_by_user, bins=int(bin_cnt/5), log=False, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()

# Average business rating
fig, ax = plt.subplots()
ax.set(xlabel="Average business rating", ylabel="Count", title="Distribution of average business rating")
remove_border(ax)

avg_ratings_by_business=smalldf.groupby('business_id').stars.mean()
print("avg_ratings_by_business: ", avg_ratings_by_business[:5])
plt.hist(avg_ratings_by_business, bins=int(bin_cnt/5), log=False, color="Green")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()

# The overall mean:
print("The overall mean: ", smalldf["stars"].mean())<jupyter_output>avg_ratings_by_user:  user_id
--65q1FpAL_UQtVZ2PTGew    4.218750
-9mljS0O9mwdOnGv-Oea3Q    2.333333
-F32Vl8Rk4dwsmk0f2wRIw    4.029412
04xW5mhSBaCzpjbcf2VSZg    3.400000
0bNXP9quoJEgyVZu9ipGgQ    3.246154
Name: stars, dtype: float64<jupyter_text>Common SupportLets now make a histogram of the common user support (the number of common reviewers) of each pair of restaurants on the smaller set, and print the mean. Pay attention to the code, as you will use parts of it later. (This code takes a bit of time to run, so be patient).The common support is an important concept, as for each pair of restaurants, its the number of people who reviewed both. It will be used to modify similarity between restaurants. If the common support is low, the similarity is less believable.<jupyter_code>restaurants=smalldf["business_id"].unique()
print("Unique restaurants: \n", restaurants[:5])

# common_user_support[0]: rest0 & rest1
# common_user_support[1]: rest0 & rest2
# ...
common_user_support = []
# Choose the first rest used in common_user_support
for i in range(len(restaurants)-1):
    # Choose the second rest used in common_user_support
    for j in range(i+1, len(restaurants)):
        rest0_user = smalldf[smalldf["business_id"] == restaurants[i]]["user_id"].unique()
        rest1_user = smalldf[smalldf["business_id"] == restaurants[j]]["user_id"].unique()
        common_users = set(rest0_user).intersection(set(rest1_user))
        common_user_support.append(len(common_users))
        
fig, ax = plt.subplots()
remove_border(ax)
#ax.set(xlabel="Average business rating", ylabel="Count of common users", title="Distribution of common user support")
plt.hist(common_user_support, color='Green')
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.show()

# Mean of the common user support
print("Mean of the common user support: ", np.mean(common_user_support))<jupyter_output>Unique restaurants: 
 ['eIxSLxzIlfExI6vgAbn2JA' '9ziO3NpoNTKHvIKCBFB_fQ' 'nMHhuYan8e3cONo3PornJA'
 'Zx8_4zKdDBSO3qGrkukBIA' 'bc-lE-wGVAsUrX-kJhtY-Q']<jupyter_text>Most common support between restaurant pairs are less than 10. Calculating SimilarityUsers rate restaurants on a scale of 1-5. Even though this rating is integer valued, for the purposes of this assignment we shall treat it as a real number.Even though each reviewer uses the same 5-star scale when rating restaurants, comparing two users by comparing their raw user ratings can be problematic. Consider a user whose average rating is 2. This is a curmudgeonly user. Consider another whose average rating is 4. This is a rather enthusiastic one. How should we compare a 3 rating by the curmudgeonly one to a 5 rating of the enthusiastic one?It is for this purpose that we must subtract the average rating of the user from the actual rating of the restaurants in computing the similarity of two restaurants. This makes the above ratings by the two users comparable. We do this in the function pearson_sim defined below.If there is no common support (n_common=0), we have no basis for making a similarity estimate, and so we set the similarity to 0. In the case that the individual restaurant rating variance is 0, such as in the case where there is only one common reviewer (n_common=1), we return the NaN that the scipy pearsonr returns. We will deal with it soon.<jupyter_code>from scipy.stats.stats import pearsonr
def pearson_sim(rest1_reviews, rest2_reviews, n_common):
    """
    Given a subframe of restaurant 1 reviews and a subframe of restaurant 2 reviews,
    where the reviewers are those who have reviewed both restaurants, return 
    the pearson correlation coefficient between the user average subtracted ratings.
    The case for zero common reviewers is handled separately. Its
    ok to return a NaN if any of the individual variances are 0.
    """
    if n_common==0:
        rho=0.
    else:
        diff1=rest1_reviews['stars']-rest1_reviews['user_avg']
        diff2=rest2_reviews['stars']-rest2_reviews['user_avg']
        rho=pearsonr(diff1, diff2)[0]
    return rho<jupyter_output><empty_output><jupyter_text>The function get_restaurant_reviews defined below takes a restaurant business_id and a set of users, and returns the reviews of that restaurant by those users. You will use this function in calculating a similarity function, in 1.5.<jupyter_code>def get_restaurant_reviews(restaurant_id, df, set_of_users):
    """
    given a resturant id and a set of reviewers, return the sub-dataframe of their
    reviews.
    """
    mask = (df.user_id.isin(set_of_users)) & (df.business_id==restaurant_id)
    reviews = df[mask]
    reviews = reviews[reviews.user_id.duplicated()==False]
    return reviews<jupyter_output><empty_output><jupyter_text>**1.5** Write a function calculate_similarity that operates between two restaurants and calculates a similarity for them, taking a dataframe and a similarity function similarity_func. An example of the similarity_func is the pearson_sim we defined above. calculate_similarity operates as follows:1. For each of the two restaurants, get the set of reviewers who have reviewed the restaurant and compute the intersection of these two sets. Also compute the number of common reviewers n_common.2. Use the function get_restaurant_reviews defined below to get the reviews for each restaurant as made by these common reviewers. Notice that get_restaurant_reviews returns a sub data frame of reviews.3. Calculate the similarity using similarity_func which takes the two reviews dataframes from part 2 and the number of common reviewers n_common as arguments4. Return the similarity and n_common in a tuple (sim, n_common). If the similarity is a NaN, set the similarity to 0.<jupyter_code>"""
Function
--------
calculate_similarity

Parameters
----------
rest1 : string
    The id of restaurant 1
rest2 : string
    The id of restaurant 2
df : DataFrame
  A dataframe of reviews, such as the smalldf above
similarity_func : func
  A function like pearson_sim above which takes two dataframes of individual
  restaurant reviews made by a common set of reviewers, and the number of
  common reviews. This function returns the similarity of the two restaurants
  based on the common reviews.
  
Returns
--------
A tuple
  The first element of the tuple is the similarity and the second the
  common support n_common. If the similarity is a NaN, set it to 0
"""

def calculate_similarity(rest1, rest2, df, similarity_func):
    rest1_user = smalldf[smalldf["business_id"] == rest1]["user_id"].unique()
    rest2_user = smalldf[smalldf["business_id"] == rest2]["user_id"].unique()
    common_user = set(rest1_user).intersection(set(rest2_user))
    n_common = len(common_user)
    
    rest1_reviews = get_restaurant_reviews(rest1, df, common_user)
    rest2_reviews = get_restaurant_reviews(rest2, df, common_user)
    similarity = similarity_func(rest1_reviews, rest2_reviews, n_common)
    if np.isnan(similarity):
        return 0, n_common
    return similarity, n_common<jupyter_output><empty_output><jupyter_text>Making a database of similaritiesWe now move to calculating a global database of pairwise restaurant similarities. We provide you here with a function to make a database of the similarities for each pair of restaurants in the database. The class Database is initialized in its constructor by taking as arguments a dataframe of reviews. The method populate_by calculating iterates over every possible pair of business_id's in the dataframe and populates the database with similarities and common supports. It takes as arguments a function the similarity function similarity_func like pearson_sim (calculate_similarity then uses this to calculate the similarity). The get method on the database can be used to retrieve the similarity for two business ids.(See Thu Oct 17th's class video for information about classes)<jupyter_code>class Database:
    "A class representing a database of similaries and common supports"
    
    def __init__(self, df):
        "the constructor, takes a reviews dataframe like smalldf as its argument"
        database={}
        self.df=df
        self.uniquebizids={v:k for (k,v) in enumerate(df.business_id.unique())}
        keys=self.uniquebizids.keys()
        l_keys=len(keys)
        self.database_sim=np.zeros([l_keys,l_keys])
        self.database_sup=np.zeros([l_keys, l_keys], dtype=np.int)
        
    def populate_by_calculating(self, similarity_func):
        """
        a populator for every pair of businesses in df. takes similarity_func like
        pearson_sim as argument
        """
        items=self.uniquebizids.items()
        for b1, i1 in items:
            for b2, i2 in items:
                if i1 < i2:
                    sim, nsup=calculate_similarity(b1, b2, self.df, similarity_func)
                    self.database_sim[i1][i2]=sim
                    self.database_sim[i2][i1]=sim
                    self.database_sup[i1][i2]=nsup
                    self.database_sup[i2][i1]=nsup
                elif i1==i2:
                    nsup=self.df[self.df.business_id==b1].user_id.count()
                    self.database_sim[i1][i1]=1.
                    self.database_sup[i1][i1]=nsup
                    

    def get(self, b1, b2):
        "returns a tuple of similarity,common_support given two business ids"
        sim=self.database_sim[self.uniquebizids[b1]][self.uniquebizids[b2]]
        nsup=self.database_sup[self.uniquebizids[b1]][self.uniquebizids[b2]]
        return (sim, nsup)<jupyter_output><empty_output><jupyter_text>Lets run make_database and store the result in the global variable db. Lets print out an example entry. Running this function will take a bit of time.<jupyter_code>db=Database(smalldf)
db.populate_by_calculating(pearson_sim)
db.get("z3yFuLVrmH-3RJruPEMYKw", "zruUQvFySeXyEd7_rQixBg")<jupyter_output><empty_output><jupyter_text>K-Nearest restaurants (in similarity)We are now going to find the k-nearest restaurants to a given restaurant based on the database of similarities that we calculated. But we have a problem.Consider the two cases where there is just one common reviewer, and where there are 40. In the former case, we might get a artificially high similarity based on the tastes of just this user, and thus we must reduce its importance in the nearest-neighbor calculation. In the latter case, we would get a much more unbiased estimator of the similarity of the two restaurants.To control the effect of small common supports, we can **shrink** our pearson co-efficients. We shall do this by using the "regularization" parameter `reg`:$$s_{mj} = \frac{N_{common}\, \rho_{mj}}{N_{common}+reg} $$where $N_{common}$ (`n_common`) is the common reviewer support and $\rho_{ij}$ is the pearson co-relation coefficient.Recall the notions of regularization introduced in class. We want to reduce the variance in our estimates, so we pull our estimates in toward a conservative point in a way that strongly corrals in estimates when there is very little data, but allows the data to speak when there is a lot. This can be shown as equivalent to adding in a `reg` amount of bayesian prior, as Joe has alluded to in class. A good value of the regularizer is intuitively one that doesn't affect the similarity when the common support is high ~10, but has a large effect when the support is small. In this case, values of 2-4 are good. Usually, the value of `reg` is determined using cross-validation, but for the sake of simplicity we will generally set it to 3.We define a function `shrunk_sim` which takes the `sim` and `n_common` obtained from the database, and shrinks the similarity down using the regularizer `reg`.<jupyter_code>def shrunk_sim(sim, n_common, reg=3.):
    "takes a similarity and shrinks it down by using the regularizer"
    ssim=(n_common*sim)/(n_common+reg)
    return ssim<jupyter_output><empty_output><jupyter_text>**1.6** Now we can move to writing a `knearest` function, which finds the `k` nearest neighbors of a given restaurant based on the shrunk similarities we calculate. Note that as defined here, the nearest neighbors are global over the entire set of restaurants, as opposed to being restricted to the restaurants a user has reviewed(we shall do that in the next problem). Thus, this is an expensive function!Write a `knearest` that returns a *k-length sorted list* of 3-tuples each corresponding to a restaurant. The tuple structure is `(business_id, shrunken similarity score, common support)` where the similarity score and common support are with respect to the restaurant whose neighbors we are finding, and the `business_id` is the id of the "nearby" restaurant found. The nearby restaurants are found from a supplied numpy array of restaurants `set_of_restaurants`. The spec for the function is given below. HINT: use `itemgetter` from the `operator` module to do the sorting.<jupyter_code>"""
Function
--------
knearest

Parameters
----------
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
set_of_restaurants : array
    The set of restaurants from which we want to find the nearest neighbors
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.
    
  
Returns
--------
A sorted list
    of the top k similar restaurants. The list is a list of tuples
    (business_id, shrunken similarity, common support).
"""
def knearest(restaurant_id, set_of_restaurants, dbase, k, reg):
    similars=[]
    for other_rest_id in set_of_restaurants:
        if other_rest_id!=restaurant_id:
            sim, nc=dbase.get(restaurant_id, other_rest_id)
            ssim=shrunk_sim(sim, nc, reg=reg)
            similars.append((other_rest_id, ssim, nc ))
    similars=sorted(similars, key=itemgetter(1), reverse=True)
    return similars[:k]<jupyter_output><empty_output><jupyter_text>Ok it's time to recommend!Lets choose the two very different businesses in the dataframe<jupyter_code>testbizid="eIxSLxzIlfExI6vgAbn2JA"
testbizid2="L-uPZxooP_ziXCtRrWi8Pw"<jupyter_output><empty_output><jupyter_text>We provide functions to look up a business name given a business id, and a username given a user id.<jupyter_code>def biznamefromid(df, theid):
    return df['biz_name'][df['business_id']==theid].values[0]
def usernamefromid(df, theid):
    return df['user_name'][df['user_id']==theid].values[0]
print(testbizid, biznamefromid(smalldf,testbizid))
print(testbizid2, biznamefromid(smalldf, testbizid2))<jupyter_output>eIxSLxzIlfExI6vgAbn2JA Lobbys Beef Burgers Dogs
L-uPZxooP_ziXCtRrWi8Pw Café Monarch<jupyter_text>Get top matchesIts now time to answer the question: "if you liked this, you might also like these". We use our `testbizid` and `testbizid2` to compute the `k=7` nearest neighbors with a regularization of `3.` . We print these top 7 matches names, along with their similarity coefficient and common support.<jupyter_code>tops=knearest(testbizid, smalldf.business_id.unique(), db, k=7, reg=3.)
print("For ",biznamefromid(smalldf, testbizid), ", top matches are:")
for i, (biz_id, sim, nc) in enumerate(tops):
    print(i,biznamefromid(smalldf,biz_id), "| Sim", sim, "| Support",nc)
tops2=knearest(testbizid2, smalldf.business_id.unique(), db, k=7, reg=3.)
print("For ",biznamefromid(smalldf, testbizid2), ", top matches are:")
for i, (biz_id, sim, nc) in enumerate(tops2):
    print(i,biznamefromid(smalldf,biz_id), "| Sim", sim, "| Support",nc)<jupyter_output>For  Café Monarch , top matches are:
0 Postino Arcadia | Sim 0.625 | Support 5
1 The Main Ingredient Ale House And Café | Sim 0.571428571429 | Support 4
2 Brio Tuscan Grille | Sim 0.571428571429 | Support 4
3 Kazimierz World Wine Bar | Sim 0.5 | Support 3
4 Harlow's Cafe | Sim 0.5 | Support 3
5 The Fry Bread House | Sim 0.5 | Support 3
6 Cien Agaves Tacos & Tequila | Sim 0.5 | Support 3<jupyter_text>We can see that these two restaurants are in somewhat different orbits :-). Lets now turn our attention to another question: what are the top recommendations for a user? To answer this we must find the user's top rated restaurants, find the nearest neighbors of these restaurants, merge these lists while removing the duplicates and the ones that the user has already rated, and sort by the restaurant's average rating. We provide the code to get the user's top choices in a subset data frame.<jupyter_code>def get_user_top_choices(user_id, df, numchoices=5):
    "get the sorted top 5 restaurants for a user by the star rating the user gave them"
    udf=df[df.user_id==user_id][['business_id','stars']].sort_values(['stars'], ascending=False).head(numchoices)
    return udf
testuserid="7cR92zkDv4W3kqzii6axvg"
print("For user", usernamefromid(smalldf,testuserid), "top choices are:")
bizs=get_user_top_choices(testuserid, smalldf)['business_id'].values
[biznamefromid(smalldf, biz_id) for biz_id in bizs]<jupyter_output>For user Vern top choices are:<jupyter_text>Get top recommendations for user. **1.7** Its your job now to write a function `get_top_recos_for_user` which takes as arguments a `userid`, the `n` top choices for the user, the dataframe, `k`, and a regularizer, and returns the top recommendations obtained from combining the restaurants that are neighbors of each of the `n` choices, in the way described in the previous paragraph. This returned list is a list of tuples `(restaurant_id, business_avg)` sorted by `business_avg` where `business_avg` is the average rating of the restaurant over the dataframe.<jupyter_code>"""
Function
--------
get_top_recos_for_user

Parameters
----------
userid : string
    The id of the user for whom we want the top recommendations
df : Dataframe
    The dataframe of restaurant reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businesses. e.g. dbase.get(rid1,rid2)
n: int
    the n top choices of the user by star rating
k : int
    the number of nearest neighbors desired, default 8
reg: float
    the regularization.
    
  
Returns
--------
A sorted list
    of the top recommendations. The list is a list of tuples
    (business_id, business_avg). You are combining the k-nearest recommendations 
    for each of the user's n top choices, removing duplicates and the ones the user
    has already rated.
"""
def get_top_recos_for_user(userid, df, dbase, n, k, reg):
    recommendations = []
    # get_user_top_choices(): df[df.user_id==user_id][['business_id','stars']]
    tops = get_user_top_choices(userid, df, n)['business_id'].tolist()
    for top in tops:
        # For each top rated biz by this user, get a knearest list
        # knearest returns: [(business_id, shrunken similarity, common support) ...]
        top_knearest = knearest(top, smalldf["business_id"].unique(), db, k, reg)
        
        for nearest in top_knearest:
            business_id = nearest[0]
            business_avg = smalldf[smalldf["business_id"] == business_id]["stars"].mean()
            # Must recommend a restaurant that hasn't been rated by this user
            if (business_id, business_avg) not in recommendations and business_id not in df[df.user_id==userid].business_id.values:
                recommendations.append((business_id, business_avg))
                
    recommendations_sorted = sorted(recommendations, key=itemgetter(1), reverse=True)[:n]
    return recommendations_sorted<jupyter_output><empty_output><jupyter_text>Lets print the top recommendations for `testuserid`, with a regularization of 3.<jupyter_code>print("For user", usernamefromid(smalldf,testuserid), "the top recommendations are:")
toprecos=get_top_recos_for_user(testuserid, smalldf, db, n=5, k=7, reg=3.)
for biz_id, biz_avg in toprecos:
    print(biznamefromid(smalldf,biz_id), "| Average Rating |", biz_avg)<jupyter_output>For user Vern the top recommendations are:
Rokerij | Average Rating | 4.379310344827586
Wildfish Seafood Grille | Average Rating | 4.294117647058823
Defalco's Italian Grocery | Average Rating | 4.232558139534884
Cornish Pasty Company | Average Rating | 4.206896551724138
Pappadeaux Seafood Kitchen | Average Rating | 4.185185185185185<jupyter_text>Problem2: A user based recommender with predicted ratingsFurtherly personalize. Defining the predicted rating**2.1** Write a function knearest_amongst_userrated, analogous to the knearest function we defined above, to find the nearest k neighbors to a given restaurant from the restaurants that the user has already rated. This function will take as arguments the restaurant_id, the user_id, the dataframe of reviews, the database, the k, and the regularizer reg. Just like before, return a k-length sorted list of 3-tuples each corresponding to a restaurant. HINT: use the knearest function you defined earlier<jupyter_code>"""
Function
--------
knearest_amongst_userrated

Parameters
----------
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
user_id : string
    The id of the user, in whose reviewed restaurants we want to find the neighbors
df: Dataframe
    The dataframe of reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.
    
  
Returns
--------
A sorted list
    of the top k similar restaurants. The list is a list of tuples
    (business_id, shrunken similarity, common support).
"""
def knearest_amongst_userrated(restaurant_id, user_id, df, dbase, k, reg):
    rated_restaurants = df[df["user_id"]==user_id]["business_id"].unique()
    return knearest(restaurant_id, rated_restaurants, dbase, k, reg)<jupyter_output><empty_output><jupyter_text>**2.2** Now write a function that returns the predicted rating for a user and an item using the formula at the beginning of this problem. Include code to deal with the possibility that the sum of scores that goes in the denominator is 0: return an predicted rating of the baseline portion of the formula in that case. This function rating takes as arguments the dataframe, the database, the wanted restaurant_id and user_id, and k as well as the regularizer.<jupyter_code>"""
Function
--------
rating

Parameters
----------
df: Dataframe
    The dataframe of reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
user_id : string
    The id of the user, in whose reviewed restaurants we want to find the neighbors
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.
    
  
Returns
--------
A float
    which is the impued rating that we predict that user_id will make for restaurant_id
"""
def rating(df, dbase, restaurant_id, user_id, k, reg):
    # Get the k nearest restaurants rated by this user to the rest for prediction
    # knearest_amongst_userrated returns (business_id, shrunken similarity, common support) tuples
    knearest_rests = knearest_amongst_userrated(restaurant_id, user_id, df, dbase, k, reg)
    # Filter out the reviews by this user
    users_reviews = df[df["user_id"] == user_id]
    user_avg = users_reviews["user_avg"].values[0]
    rest_avg = df[df["business_id"]==restaurant_id]["business_avg"].values[0]
    all_avg = df["stars"].mean()

    numertaor = 0
    denominator = 0

    # Predict based on the k nearest restaurants
    # Each in the k nearest restaurants' rating will contribute to the predicting rating using its similarity to that restaurant as a weight
    for n_business_id, sim, common_sup in knearest_rests:
        denominator += sim
        # this user, this restaurant
        r_reviews_row=users_reviews[users_reviews['business_id']==n_business_id]
        r_stars=r_reviews_row.stars.values[0]
        r_avg=r_reviews_row.business_avg.values[0]
        
        rminusb=(r_stars - (r_avg + user_avg - all_avg))
        numertaor=numertaor+sim*rminusb
    
    baseline = all_avg + (user_avg - all_avg) + (rest_avg - all_avg)
    
    if denominator > 0.:
        val =  numertaor/denominator + baseline
    else:
        val=baseline
    return val<jupyter_output><empty_output><jupyter_text>For the top-recommendations in the variable toprecos from the previous section, we compute the predicted rating and compare it with the average rating over all users available inside the tuples that make up toprecos. We use a k of 7 and regularization 3. For comparision we also print this users' average rating. Do you notice anything interesting about how the order has changed from when we did this with the global similarities? (for you to think, not to answer)<jupyter_code>print("User Average", smalldf[smalldf.user_id==testuserid].stars.mean(),"for",usernamefromid(smalldf,testuserid))
print("Predicted ratings for top choices calculated earlier:")
for biz_id,biz_avg in toprecos:
    print(biznamefromid(smalldf, biz_id),"|",rating(smalldf, db, biz_id, testuserid, k=7, reg=3.),"|","Average",biz_avg)<jupyter_output>User Average 3.5652173913043477 for Vern
Predicted ratings for top choices calculated earlier:
Rokerij | 4.71714023074 | Average 4.379310344827586
Wildfish Seafood Grille | 4.27594504172 | Average 4.294117647058823
Defalco's Italian Grocery | 3.822223075 | Average 4.232558139534884
Cornish Pasty Company | 4.62810510121 | Average 4.206896551724138
Pappadeaux Seafood Kitchen | 4.08845573953 | Average 4.185185185185185<jupyter_text>Testing the ratingsLet us compare the predicted ratings with a user's ratings. Note that we are doing this on the same set that we constructed the predictions with, so this is not a validation of the procedure, but simply a check of the procedure's fit. We first write a helper function to return the user score for a restaurant, and the restaurant's average score over all users.<jupyter_code>def get_other_ratings(restaurant_id, user_id, df):
    "get a user's rating for a restaurant and the restaurant's average rating"
    choice=df[(df.business_id==restaurant_id) & (df.user_id==user_id)]
    users_score=choice.stars.values[0]
    average_score=choice.business_avg.values[0]
    return users_score, average_score<jupyter_output><empty_output><jupyter_text>For the user testuserid, we loop over the variable bizs (which is a set of restaurants the user has rated) and print the predicted rating, and the actual rating and restaurant average rating obtained using the function above. We again use k=7 and a regularization of 3.<jupyter_code>print("for user",usernamefromid(smalldf,testuserid), 'avg', smalldf[smalldf.user_id==testuserid].stars.mean())
for biz_id in bizs:
    print("----------------------------------")
    print(biznamefromid(smalldf, biz_id))
    print("Predicted Rating:",rating(smalldf, db, biz_id, testuserid, k=7, reg=3.)) 
    u,a=get_other_ratings(biz_id, testuserid, smalldf)
    print("Actual User Rating:",u,"Avg Rating",a)<jupyter_output>for user Vern avg 3.5652173913043477
----------------------------------
Tee Pee Mexican Food
Predicted Rating: 3.52640184162
Actual User Rating: 5 Avg Rating 3.04347826087
----------------------------------
Local Breeze
Predicted Rating: 4.2280987611
Actual User Rating: 5 Avg Rating 4.0
----------------------------------
Carly's Bistro
Predicted Rating: 3.99008654065
Actual User Rating: 5 Avg Rating 3.5
----------------------------------
District American Kitchen and Wine Bar
Predicted Rating: 3.80281696528
Actual User Rating: 4 Avg Rating 3.55263157895
----------------------------------
Sonora Mesquite Grill
Predicted Rating: 4.08382083034
Actual User Rating: 4 Avg Rating 4.38461538462<jupyter_text>Error AnalysisThis next function takes a set of actual ratings, and a set of predicted ratings, and plots the latter against the former. We can use a graph of this kind to see how well or badly we do in our predictions. Since the nearest neighbor models can have alternating positive and negative similarities (the sum of similarity weights in the denominator can get large), the ratings can get very large. Thus we restrict ourselves to be between -10 and 15 in our ratings and calculate the fraction within these bounds. We also plot the line with unit slope, line sehments joining the means, and a filled in area representing one standard deviation from the mean.The first argument to compare_results is a numpy array of the actual star ratings obtained from the dataframe, while the second argument is the numpy array of the predicted ones. (Feel free to improve this function for your display)<jupyter_code>def compare_results(stars_actual, stars_predicted, ylow=-10, yhigh=15, title=""):
    """
    plot predicted results against actual results. Takes 2 arguments: a
    numpy array of actual ratings and a numpy array of predicted ratings
    scatterplots the predictions, a unit slope line, line segments joining the mean,
    and a filled in area of the standard deviations."
    """
    fig=plt.figure()
    df=pd.DataFrame(dict(actual=stars_actual, predicted=stars_predicted))
    ax=plt.scatter(df.actual, df.predicted, alpha=0.2, s=30, label="predicted")
    plt.ylim([ylow,yhigh])
    plt.plot([1,5],[1,5], label="slope 1")
    xp=[1,2,3,4,5]
    yp=df.groupby('actual').predicted.mean().values
    plt.plot(xp,yp,'k', label="means")
    sig=df.groupby('actual').predicted.std().values
    plt.fill_between(xp, yp - sig, yp + sig, 
                 color='k', alpha=0.2)
    plt.xlabel("actual")
    plt.ylabel("predicted")
    plt.legend(frameon=False)
    remove_border()
    plt.grid(False)
    plt.title(title)
    print("fraction between -15 and 15 rating", np.mean(np.abs(df.predicted) < 15))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09265887417715817, 'fraction_numerical': 0.029860085600580684, 'mean_word_length': 4.535328345802161, 'pattern_counts': {'":': 0, '<': 108, '<?xml version=': 0, '>': 110, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '127546', 'license': 'MIT', 'n_tokens_mistral': 13184, 'n_tokens_neox': 12131, 'n_words': 4311, 'path': 'harvard_cs _09_hw4/restaurant_recommendation_system.ipynb', 'repo_name': 'xuwenyihust/Restaurant-Recommendation-System-', 'size': '39955'}