<jupyter_start><jupyter_text>Recommender SystemsDATA 512 Final Project  <NAME>  10th December, 2017   IntroductionA recommender system is a software that analyzes large amounts of data and extracts personal characteristics and features to present users with relevant content.If we go to Amazon today and look for "Machine learning with the Tensor flow" book, we would get recommendations like "Customer who bought this item also bought this item"; "Items Frequently brought together"; "More items to consider"; "Recommendation for you in books" and many more variations of recommendations. The analysis of recommender systems is interesting and important as it involves business-centric, engineering-centric and human-centric approaches to design RS. It solves a real-world problem and tackles unresolved research questions. Business-centric - The fundamental reason why many companies seem to care about recommender systems is for money and business as they generate significant user engagement and revenue. But, Advertisements biases the recommendation on platforms like Amazon, Facebook, Netflix, IMDB, TMDB, Spotify etc. Hence, using the traditional approaches, we built an unbiased recommender system. Engineering-centric - Also, building a recommender system is a true data science problem. It is a true representation and intersection between Software Engineering, Machine Learning, Statistics and Human-Centered aspects as building it requires all these skills.Human-centric - But most importantly Recommender System is a human-centric problem statement as the system is for the humans and the results and performance complete depends on Human Recommender interaction. We built models whose end product or outcome can be easily interpreted. Further, these models are build taking into account the audience, purpose, and context. BackgroundAlmost every company is working and using recommender systems in today's era. It is not at all surprising to us if we sign-in to some application and get recommendations the very next minute in our inbox. Since the work in this field is rapidly growing and the techniques are improving tremendously, there comes a big question regarding the human recommender interaction and how these algorithms are positively affecting the user's choices. <NAME>, <NAME>, <NAME>, and <NAME> conducted a research in 2014 on User perception of differences in recommender algorithms [10]. Further studies have been conducted in understanding the human-centered aspect for such sophisticated algorithms. <NAME>, <NAME>, and <NAME> published in 2006 a research on "Don't look stupid: avoiding pitfalls when recommending research papers" [16]. The research that closely relates or connects to what we are doing in this project is by <NAME>, <NAME>, and <NAME>. 2006. Making recommendations better: an analytic model for human-recommender interaction. Hypotheses/ research questionsFor our research project, we deliver two recommendation algorithms from scratch without using the already built libraries for such algorithms like MLlib.Further, we tried to answer two hypotheses -Hypotheses 1 - Compare the recommender personality of content-based and collaborative filtering RSHypotheses 2 - Why should a user trust that the algorithm understands who they are, what they like and what they are doing?How does this research inform your hypotheses, your analysis, or your system design?Human-centered data science fills the gap between or it is the bridge between quantitative and qualitative research. This bridge opens new opportunities, for instance, Data Scientist role is also one of the many outcomes from the amalgamation of a deep connection between qualitative methods and formally defined data-driven research technique.The project is a true representation of what we call amalgamation of a deep connection between qualitative methods and formally defined data-driven research technique and what we studied during the lectures and the take-home readings. From this project, we cover important aspects using the hypothesis like the interpretability, data privacy, data ethics and understand the bridge of HCDS between qualitative and quantitative methods. Data Brief description about the dataset We have taken the dataset from Kaggle datasets - https://www.kaggle.com/rounakbanik/the-movies-dataset. The dataset on Kaggle consists of multiple files but we are using the following files:movies_metadata.csv:  Contains information on 45,000 movies featured in the Full MovieLens dataset[6]. Fields in movies_metadata.csv include: adult  belongs_to_collection  budget  genres  homepage  id  imdb_id  original_language  original_title  overview  popularity  poster_path  production_companies  production_countries  release_date  revenue  runtime  spoken_languages  status  tagline  title  video  vote_average  vote_count  links_small.csv: Contains the TMDB and IMDB IDs of a small subset of 9,000 movies of the Full Dataset. Fields in links_small.csv include:movieId  imdbId  tmdbId  ratings_small.csv: Consist of rating for movies by anonymous users. The subset of 100,000 ratings from 700 users on 9,000 movies. Fields in ratings_small.csv include:userId  movieId  rating  timestamp   License of the data on KaggleReleased Under CC0: Public Domain License link: https://creativecommons.org/publicdomain/zero/1.0/ Is it legal to use this data set?This dataset is an aggregation of data collected from TMDB and GroupLens (MovieLens dataset). The Movie Details, Credits, and Keywords have been collected from the TMDB Open API. We can use the MovieLens dataset but must acknowledge the use of the dataset in publications or project report [7]. We have acknowledged the use of the dataset in the reference section.  Methods Analytical methods Collaborative FilteringContent-based recommender systems concentrate on the characteristics of the items and give recommendations based on the similarity between them, that is if you like an item then you will also like a “similar” item. For calculating the similarity between movies, we are using two features - overview and taglines. We are calculating the cosine similarity between movies by converting the textual features into numerical feature vectors using tf-idf vectorizer from sci-kit learn package. Content-based FilteringCollaborative filtering is an unsupervised learning algorithm which produces recommendations based on the knowledge of user’ attitude to items, that is it uses the “wisdom of the crowd” and "past behavior" to recommend items. For calculating the similarity between movies, we are using Pearson correlation coefficient.  Hybrid ApproachThe amalgamation of Collaborative filtering and content-based approach is called a hybrid approach. It is very powerful technique as it takes advantages of both the approaches and eliminates the disadvantages. What methods are we using and why these methods?We are concentrating on Content-based and Collaborative Filtering algorithm as these are traditional approaches to build a recommendation engine. Both the approaches are mutually exclusive and the results depend on what we want to achieve. In Collaborative filtering, recommender system tries to consider the similarities between users on the platform and based on these similarities, movies that similar neighbors of the target user showed interest in are equally recommended to him or her. This approach gives room for more diverse or serendipitous recommendations.On the other hand, in Content-based approach, recommendations are made to the user based on their movie interests. Movies are only recommended to the user if he had shown interest in that genre, tags or related attribute previously. This approach lacks some diversity and serendipity but may recommend movies that the user may like now just because he had liked movies with similar attributes before. Our hypothesis includes the involvement of human aspect. Fortunately, research in the field of recommender systems still remains powerful and alive. There have been great advancements and development in these approaches. Moreover, Information filtering can be more effective when users are involved in the algorithmic process genuinely. We believe that to be true now more than ever because working on transparency of the recommender systems, and enabling ever-improving suggestions from users is both meaningful and challenging. Code / Implementation of methods  Import all the libraries<jupyter_code># Import all the libraries
import pandas as pd
import time

import sklearn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
import numpy as np
import re
import json
import scipy<jupyter_output><empty_output><jupyter_text>Content Based filtering The analysis is done on a smaller data set comprising of 9099 movies. Since, content based filtering concentrate on the characteristics of the items, hence we have used overview and tagline of the movies to find the top ten similar movies for each movie.<jupyter_code># Content based method 

# This function generates recommendation using content based filtering approach using the description of the movie
def generate_similar_data(ds, smooth_idf=True, norm='l2', sublinear_tf=True, min_df=1, max_df=1.0, min_n=1, max_n=2,
                          max_features=None):
    start = time.time()

    tf = TfidfVectorizer(analyzer='word', sublinear_tf=sublinear_tf, ngram_range=(min_n, max_n), stop_words='english',
                         norm=norm, smooth_idf=smooth_idf, max_df=max_df, min_df=min_df, max_features=max_features)
    tfidf_matrix = tf.fit_transform(ds['description'].values.astype('U'))
    idf = tf.idf_
    feature_list = []
    ids_list = []
    
    cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)
    # normalizing the cosine similarities
    cosine_similarities = sklearn.preprocessing.normalize(cosine_similarities, norm='l2')

    similar_movie_list = []
    movie_name_list = []
    similarity_measure = []
    # Number of recommendation to select for each movie 
    n = 12
    for idx, row in ds.iterrows():
        similar_indices = cosine_similarities[idx].argsort()[:-n:-1]
        similar_items = [(cosine_similarities[idx][i], ds['id'][i]) for i in similar_indices]

        for similar_item in similar_items[1:]:
            similar_movie_list.append(similar_item[1])
            movie_name_list.append(row.title)
            similarity_measure.append(similar_item[0])

    di = {}
    for idx, row in ds.iterrows():
        di[row.id] = row.title
        
    df = pd.DataFrame({'movie_name': movie_name_list, 'similar_movie_name': similar_movie_list, 'similarity_measure': similarity_measure})
    df["similar_movie_name"].replace(di, inplace=True)
    
    print("Engine trained in %s seconds." % (time.time() - start))
    return df


# Read Movies data
main_df = pd.read_csv('movies_metadata.csv')
main_df = main_df[main_df.id.apply(lambda x: x.isnumeric())]
main_df['id'] = main_df['id'].astype('int')

# Link movie data with the links data 
links_small_dataset = pd.read_csv('links_small.csv')
links_small_tmdbId = pd.DataFrame({'id':links_small_dataset[links_small_dataset['tmdbId'].notnull()]['tmdbId'].astype('int')})

# merge dataframes
new_df = pd.merge(main_df, links_small_tmdbId, on=['id'], how='inner')
new_df['description'] = new_df['overview'] + ' ' + new_df['tagline']  

similarity_data_content_based = generate_similar_data(new_df)
# This function get the movie recommendation using content based approach for any movie in the dataset. 
def get_movie_recommendations_Content_Based(movies, n = 5):  
    similarities = similarity_data_content_based[similarity_data_content_based.movie_name.isin([movies])]
    similarities = similarities.sort_values(by=['similarity_measure'], ascending=False)[:n]
    return similarities<jupyter_output><empty_output><jupyter_text>Collaborative filteringThere are two types of collaborative filtering approaches - Item-item collaborative filtering - In the item-based approach a rating (u1, i1), from user u1 for item i1, is produced by looking at the set of items similar to i1, then the ratings by u1 of similar items are combined into a predicted rating.User-user collaborative filtering - In the user-based approach, for user u1, a score for an unrated item is produced by combining the ratings of users similar to u1.We will be using Item-item collaborative filtering as it makes more sense to compare the recommender personality of content based and item-item collaborative filtering. Since, both the techiques use items to get the similarity measure.<jupyter_code># read ratings file 
ratings = pd.read_csv('ratings_small.csv')
# merge dataframes
ratings = pd.merge(ratings, links_small_dataset, on=['movieId'], how='inner')
ratings = ratings.dropna(axis=0, how='any')
ratings['tmdbId'] = ratings['tmdbId'].astype('int')
movie_titles_with_ids = pd.DataFrame({'tmdbId':new_df['id'], 'title':new_df['title']})
ratings = pd.merge(ratings, movie_titles_with_ids, on=['tmdbId'], how='inner')
del ratings['timestamp']
ratings.head()<jupyter_output><empty_output><jupyter_text>Convert the ratings dataframe to a matrix with a movie per column and a user per row.<jupyter_code>ratings_mtx_df = ratings.pivot_table(values='rating', index='userId', columns='title')  
ratings_mtx_df.fillna(0, inplace=True)

movie_index = ratings_mtx_df.columns

ratings_mtx_df.head()<jupyter_output><empty_output><jupyter_text>Measure the similarity between movies, that is, find correlation between movies and use that correlation to find similar movies to those the users have liked.<jupyter_code>corr_matrix = np.corrcoef(ratings_mtx_df.T)  
# This function get the movie recommendation using CF approach for any movie in the dataset. 
def get_movie_recommendations_CF(movies, n = 5):  
    movie_idx = list(movie_index).index(movies)
    sum_movie_similarities = corr_matrix[movie_idx]
    similarities = pd.DataFrame({
        'title': movie_index,
        'similarity_sum': sum_movie_similarities
        })
    similarities = similarities.sort_values(by=['similarity_sum'], ascending=False)[1:n+1]
    return similarities<jupyter_output><empty_output><jupyter_text>Findings and research  Hypothesis 1 Comparing the recommender personality of content based and item based collaborative filtering<jupyter_code># Content based 
sample_movie = 'The Dark Knight Rises'
recommendations = get_movie_recommendations_Content_Based(sample_movie)
recommendations

# Collaborative filtering
sample_movie = 'The Dark Knight Rises'
recommendations = get_movie_recommendations_CF(sample_movie)
recommendations<jupyter_output><empty_output><jupyter_text>Comparing the personality of both the recommenders based on the results above -1. Freshness    1.1. The content-based approach gives old movies recommendations mostly from 90's movies - like Batman Forever, Batman Returns, Batman: Mask of the Phantasm.    1.2. Collaborative filtering whereas give fresh recommendations like Captain America: The First Avenger (2011), Limitless (2011), The Hunger Games: Catching Fire (2013), Inception (2010), and Inception (2012)  2. Diversity       2.1. Collaborative filtering recommends diverse set of movies from the same or similar genre, whereas content based only recommend movies which are based on or related to Batman.  3. Degree of personalization       3.1. The content-based algorithm is based on the items attribute and hence they are not personalized as per the users' requirement. They are personalized to the item which the user recently saw.     3.2. Collaborative filtering algorithm is based on the past history of the user and hence the recommendations are highly personalized based on what the user saw in the past. This approach adds novelty factor as well to the recommendation personality as we explore the items/movies which a user might see in future based on similar user analysis.   We know from several readings and resources that that satisfaction positively dependent on diversity and negatively dependent on novelty. Hence, both the techniques have pros and cons, for instance, collaborative filtering has higher diversity and freshness in the results but then it also has high novelty. Whereas results from content-based are quite the opposite of collaborative filtering.   Hypothesis 2Why should a user trust that the algorithm understands who they are, what they like and what they are doing?<jupyter_code># Content based 
sample_movie = 'Harry Potter and the Prisoner of Azkaban'
recommendations = get_movie_recommendations_Content_Based(sample_movie, n = 4)
recommendations<jupyter_output><empty_output><jupyter_text>The results we got from the content-based approach, we can see that they are apt as all the recommended movies are Harry Potter movies and this is because of the content in the tagline and overview. We as Data Scientist/ Engineers understand why we got such results. But, such results might creep out the users if they got an email/ or notification on the dashboard to view the recommended movies because the users have no clue why, and from where they got such recommendation. Therefore, to gain or improve the user trust that the algorithm understands who they are, what they like and what they are doing, we should make sure that we follow the ethics while recommending the movies/items - 1. Awareness - Let members be aware of how we are adapting to their tastes. 2. Explanations - Not recommending it because it suits our business needs, but because it matches the information we have from you: your explicit taste preferences and ratings, your viewing history.Hence, if we adhere to a format for the content based recommender algorithm APIs while recommending movies which gives an explanation and add awareness, then we can reflect the recommendations to the users in a positive and non-creepy way. For instance,<jupyter_code>features = ['movies\'s tag line', 'movies\'s overview']
sample_movie = 'H<NAME> and the Prisoner of Azkaban'
recommendations = get_movie_recommendations_Content_Based(sample_movie, n = 4)

print('Because you saw', sample_movie, 'we recommend you \n', 
      recommendations['similar_movie_name'].to_string(index=False), 
      '\nbased on there similarity in the following features \n', features )<jupyter_output>Because you saw <NAME> and the Prisoner of Azkaban we recommend you 
 <NAME>ter and the Chamber of Secrets
Harry Potter and the Deathly Hallows: Part 1
         Harry Potter and the Goblet of Fire
Harry Potter and the Deathly Hallows: Part 2 
based on there similarity in the following features 
 ["movies's tag line", "movies's overview"]<jupyter_text>The above template captures both awareness and explaination for the personalized recommendations from content based approach.<jupyter_code># Collaborative filtering
sample_movie = '<NAME> and the Prisoner of Azkaban'
recommendations = get_movie_recommendations_CF(sample_movie, n = 4)
recommendations<jupyter_output><empty_output><jupyter_text>For collaborative filtering approach the recommendations are provided based on the wisdom of the crowd. The only difference is whether the wisdom is captured from the item - item matrix based on user count or user-user matrix based on item count. Hence, for collaborative filtering we could use the following template -<jupyter_code>sample_movie = 'Harry Potter and the Prisoner of Azkaban'
recommendations = get_movie_recommendations_CF(sample_movie, n = 4)

print('Viewer who saw', sample_movie, 'also saw the following movies \n', recommendations['title'].to_string(index=False))<jupyter_output>Viewer who saw <NAME> and the Prisoner of Azkaban also saw the following movies 
 Harry Potter and the Goblet of Fire
          Harry Potter and the Chamber of Secrets
         Harry Potter and the Philosopher's Stone
Pirates of the Caribbean: The Curse of the Bla...
{'fraction_non_alphanumeric': 0.05386393327303789, 'fraction_numerical': 0.006532006833484072, 'mean_word_length': 5.241141423643776, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '93446', 'license': 'MIT', 'n_tokens_mistral': 5193, 'n_tokens_neox': 4812, 'n_words': 2410, 'path': 'Data-512-Final-Project.ipynb', 'repo_name': 'niharikasharma/data-512-final-project', 'size': '20036'}