<jupyter_start><jupyter_text>Collaborative FilteringAlso called User-User Filtering which uses other users to recommend items to the input user. It finds users that have similar preferences and opinions as the input and then recommends items that they have liked  to the input. Persona correlation is one of the methods that can be used to find similar users. The process is as follows :- Select a user with the movies the user has watched- Based on his rating to movies, find the top X neighbors- Get the watched movie record of the user for each neighbor- Calculate the similarity score using some method- Recommend the items with the highest score<jupyter_code>userInput = [
            {'title':'Breakfast Club, The', 'rating':5},
            {'title':'Toy Story', 'rating':3.5},
            {'title':'Jumanji', 'rating':2},
            {'title':"Pulp Fiction", 'rating':5},
            {'title':'Akira', 'rating':4.5}
         ] 
inputMovies = pd.DataFrame(userInput)
inputMovies
inputId = moviesdf[moviesdf['title'].isin(inputMovies['title'].tolist())]
inputMovies = pd.merge(inputId, inputMovies)
inputMovies = inputMovies.drop('year', 1)
inputMovies
# filtering out users that have watched movies that the input has watched and storing it
userSubset = ratingsdf[ratingsdf['movieId'].isin(inputMovies['movieId'].tolist())]
userSubset.head()
userSubsetGroup = userSubset.groupby(['userId'])
userSubsetGroup.get_group(1130)
# sorting these groups so that users that share the most movies in common with the input have higher priority which provides richer recommendations
userSubsetGroup = sorted(userSubsetGroup, key = lambda x:  len(x[1]), reverse = True)
userSubsetGroup[0:3]<jupyter_output><empty_output><jupyter_text>Similarity of usersHow similar each user is to the input can be found using the Pearson Correlation Coefficient which is used to measure the strength of a linear association between two variables. The formula for finding this coefficient between sets X and Y with N values can be given as:![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/bd1ccc2979b0fd1c1aec96e386f686ae874f9ec0 "Pearson Correlation")Pearson Correlation is invariant to scaling i.e. multiplying all elements by a nonzero constant or adding any constant to all elements. For example, if there are two vectors X and Y, then pearson(X, Y) == pearson(X, 2 * Y + 3). This is important for recommendation systems because for example two users might rate two series of items totally different in terms of absolute rates, but they would be similar users (i.e with simialar ideas) with similar rates in various scales.The values given by the formula vary from r = -1 to  r = 1, where 1 forms a direct correlation between the two entities(perfect positive correlation) and -1 forms a perfect negative correlation. 1 means that the two users have similar tastes while a -1 means the opposite.<jupyter_code>userSubsetGroup = userSubsetGroup[0:100]

# store the pearson correlation in a dictionary where the key is the user ID and the value is the coefficient
pearsonDict = {}

# for every user group in our subset
for name, group in userSubsetGroup:
    group = group.sort_values(by = 'movieId')
    # N for the formula
    nRatings = len(group)
    # get review scores for the  movies that they both have in common
    tempdf = inputMovies[inputMovies['movieId'].isin(group['movieId'].tolist())]
    # store the review scores in a temporary buffer variable in a list format to facilitate future calculations
    tempRatingList = tempdf['rating'].tolist()
    tempGroupList = group['rating'].tolist()
    # calculating the pearson correlation between two users
    Sxx = sum([i**2 for i in tempRatingList]) - pow(sum(tempRatingList), 2) / float(nRatings)
    Syy = sum([i**2 for i in tempGroupList]) - pow(sum(tempGroupList), 2) / float(nRatings)
    Sxy = sum( i*j for i, j in zip(tempRatingList, tempGroupList)) - sum(tempRatingList) * sum(tempGroupList) / float(nRatings)
    
    # if the denominator is different than zero, then divide else  0 correlation
    if Sxx != 0 and Syy != 0:
        pearsonDict[name] = Sxy / sqrt(Sxx * Syy)
    else:
        pearsonDict[name] = 0
        
pearsonDict.items()
pearsonDf = pd.DataFrame.from_dict(pearsonDict, orient = 'index')
pearsonDf.columns = ['similarityIndex']
pearsonDf['userId'] = pearsonDf.index
pearsonDf.index = range(len(pearsonDf))
pearsonDf.head()
# top 50 users that are most similar to the input
topUsers = pearsonDf.sort_values(by = 'similarityIndex', ascending = False)[0:50]
topUsers.head()<jupyter_output><empty_output><jupyter_text>Movies will be recommended by taking the weighted average of the ratings of the movies using Pearson Correlation as the weight. First we get the movies watched by the users in the pearsonDf from the ratings dataframe and then store their correlation in a new column called _similarityIndex achieved by merging the two tables<jupyter_code>topUserRating = topUsers.merge(ratingsdf, left_on = 'userId', right_on = 'userId', how = 'inner')
topUserRating.head()
# multiply the movie rating by its weight(similarity index) then sum up the new ratings and divide it by the sum of the weights
# multiply the similarity by the user's rating
topUserRating['weightedRating'] = topUserRating['similarityIndex'] * topUserRating['rating']
topUserRating.head()
# applies a sum to the topUsers after grouping it by user ID
tempTopUserRating = topUserRating.groupby('movieId').sum()[['similarityIndex', 'weightedRating']]
tempTopUserRating.columns = ['sumSimilarityIndex', 'sumWeightedRating']
tempTopUserRating.head()
# creates an empty dataframe
recommendationDf = pd.DataFrame()
# taking the weighted average
recommendationDf['weighted average recommendation score'] = tempTopUserRating['sumWeightedRating'] / tempTopUserRating['sumSimilarityIndex']
recommendationDf['movieId'] = tempTopUserRating.index
recommendationDf.head()
# top 20 movies recommended by the algorithm
recommendationDf  =recommendationDf.sort_values(by = 'weighted average recommendation score', ascending = False)
recommendationDf.head(20)
moviesdf.loc[moviesdf['movieId'].isin(recommendationDf.head(20)['movieId'].tolist())]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08735521235521236, 'fraction_numerical': 0.010939510939510939, 'mean_word_length': 5.16765873015873, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '416400', 'license': 'BSD-4-Clause-UC', 'n_tokens_mistral': 1869, 'n_tokens_neox': 1719, 'n_words': 677, 'path': 'collaborativeFiltering.ipynb', 'repo_name': 'siddharthchd/Recommendation-Systems', 'size': '6216'}