<jupyter_start><jupyter_text>CS 143 Final Project Recommendation Algorithm ImplementationFor this project we are using a kNN model to serve as a baseline recommendation algorithm. All predictions will be based off of the MovieLens Data set that contains 27,000,000 ratings and 1,100,000 tag applications applied to 58,000 movies by 280,000 users. Recommendations will be built off of movie ratings. References:(1) https://github.com/KevinLiao159/MyDataSciencePortfolio/blob/master/movie_recommender/movie_recommendation_using_KNN.ipynb(2) https://grouplens.org/datasets/movielens/latest/(3) https://www.geeksforgeeks.org/implementation-k-nearest-neighbors/  1. Upload Data<jupyter_code>import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
# import movie data
movie_data = pd.read_csv("data/movies-small.csv",
    usecols=['movieId', 'title'],
    dtype={'movieId': 'int32', 'title': 'str'})

# import corresponding ratings
rating_data = pd.read_csv("data/ratings-small.csv",
    usecols=['userId', 'movieId', 'rating'],
    dtype={'userId': 'int32', 'movieId': 'int32', 'rating': 'float32'})
print("Number of unique movies: ", len(rating_data['movieId'].unique()))
print("Number of unique users: ", len(rating_data['userId'].unique()))<jupyter_output><empty_output><jupyter_text>2. Clean DataAs with all data sets, there are likely points that do not represent the vast majority of users. In this case, there are likely unpopular movies or raters who provide very few ratings. To avoid any skewing in predictions based off this data, we will remove those individuals from the data.<jupyter_code># determine least popular movies and drop
movies_count = pd.DataFrame(rating_data.groupby('movieId').size(), columns=['count'])
popular_movie_ids = movies_count[movies_count['count'] >= 50].index
ratings_drop_movies = rating_data[rating_data.movieId.isin(popular_movie_ids)]
updated_movie_data = movie_data[movie_data.movieId.isin(popular_movie_ids)]
# determine least active users and drop
ratings_count = pd.DataFrame(rating_data.groupby('userId').size(), columns=['count'])
active_user_ids = ratings_count[ratings_count['count'] >= 50].index
ratings_drop_movies_users = ratings_drop_movies[ratings_drop_movies.userId.isin(active_user_ids)]
print("Original number of ratings: ", rating_data.shape[0])
print("Dropping unpopular movies number of ratings: ", ratings_drop_movies.shape[0])
print("Dropping unpopular movies and inactive users number of ratings: ", ratings_drop_movies_users.shape[0])
print("Number of final unique movies: ", len(ratings_drop_movies_users['movieId'].unique()))
print("Number of final unique users: ", len(ratings_drop_movies_users['userId'].unique()))
# create movie vs user matrix for kNN computations
movie_user_matrix = ratings_drop_movies_users.pivot(index='movieId', columns='userId', values='rating').fillna(0)
movie_user_matrix.shape<jupyter_output><empty_output><jupyter_text>3. Implement kNN Model<jupyter_code># reformat movie_data to be indexed on movie_id
movie_data = movie_data.set_index('movieId') 
def euclidean_distance (x,y):
    return np.linalg.norm(x-y)
def rec_model (data, num_recs, movie_title, movie_mapping):
    
    # get movie_id and data
    movie_id = movie_mapping[movie_mapping["title"] == movie_title].index[0]
    movie_data = data.loc[movie_id]
    
    # drop movie from data set
    data = data.drop(movie_id)
    
    # list to save all distances
    dists = []
    
    # iterate over all points in the data set to calculate distance from inputted val
    # CAN BE PARALLELIZED
    for index, row in data.iterrows():
        dist = euclidean_distance(movie_data, row)
        dists.append((index, dist))
        
    # sort distances in ascending order
    dists.sort(key=lambda x: x[1])
    
    # trim to num_recs recommendations and drop first 
    top_movies = dists[:num_recs]
    
    # map selected movies back to titles
    titles = [movie_mapping.loc[movie[0]]["title"] for movie in top_movies]
    
    return titles
rec_model(movie_user_matrix, 5, 'Pocahontas (1995)', movie_data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10808179162609542, 'fraction_numerical': 0.015092502434274586, 'mean_word_length': 5.648867313915858, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 15, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '142756', 'license': 'MIT', 'n_tokens_mistral': 1360, 'n_tokens_neox': 1270, 'n_words': 346, 'path': 'recommendation/kNN_implementation.ipynb', 'repo_name': 'joshseides/pCoAP', 'size': '4108'}