<jupyter_start><jupyter_text>LAB 2 - <NAME>First, let us import all packages<jupyter_code>import pandas as pd
import numpy as np
import matplotlib as plt
from sklearn.decomposition import PCA<jupyter_output><empty_output><jupyter_text>Then we read the data file into data frame<jupyter_code>ratings = pd.read_csv('ratings.dat', sep="::", header=None, engine='python')
ratings.columns = ['UserID', 'MovieID', 'Rating', 'Timestamp']
ratings.drop(['Timestamp'], axis=1, inplace=True)
print(ratings[:5])<jupyter_output>UserID  MovieID  Rating
0       1     1193       5
1       1      661       3
2       1      914       3
3       1     3408       4
4       1     2355       5<jupyter_text>1. Create m x u matrix with movies as row and users as column<jupyter_code>ratings_pivot = ratings.pivot_table('Rating', index='MovieID', columns='UserID')
print(ratings_pivot.iloc[0:10, 0:10])<jupyter_output>UserID    1   2   3   4    5    6    7    8    9    10
MovieID                                               
1        5.0 NaN NaN NaN  NaN  4.0  NaN  4.0  5.0  5.0
2        NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  5.0
3        NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  NaN
4        NaN NaN NaN NaN  NaN  NaN  NaN  3.0  NaN  NaN
5        NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  NaN
6        NaN NaN NaN NaN  2.0  NaN  4.0  NaN  NaN  NaN
7        NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  4.0
8        NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  NaN
9        NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  NaN
10       NaN NaN NaN NaN  NaN  NaN  NaN  NaN  NaN  NaN<jupyter_text>2. Normalize the matrix (we have to fill NaNs with 0 first, and call it matrix a)<jupyter_code>a = ratings_pivot.fillna(value=0, inplace=True)
a = StandardScaler().fit_transform(ratings_pivot)
print(a.shape)
np.set_printoptions(precision=2)
print(a[0:10, 0:10])
print(type(ratings_pivot))
ratings_pivot.index.values<jupyter_output><class 'pandas.core.frame.DataFrame'><jupyter_text>3. Compute SVD to get U, S and V<jupyter_code>U, Sigma, V = np.linalg.svd(a)<jupyter_output><empty_output><jupyter_text>4. From your V.T select 50 components.<jupyter_code>V_T = V[:50]<jupyter_output><empty_output><jupyter_text>Shape of U ,S and V after reduction:<jupyter_code>print(U.shape)
print(Sigma.shape)
print(V.shape)
print(V_T.shape)<jupyter_output>(3706, 3706)
(3706,)
(6040, 6040)
(50, 6040)<jupyter_text>New, reduced matrix a we can recover by multiplying U ,S and V after reduction:<jupyter_code>a_reduced = V_T.dot(a.T)
a_reduced = a_reduced.T #we have to transpose in order to get movies as rows and components as columns
print(a_reduced.shape)<jupyter_output>(3706, 50)<jupyter_text>In order to have movie names, we have to import movies.dat file.<jupyter_code>movies = pd.read_csv('movies.dat', sep="::", header=None, engine='python')
movies.columns = ['MovieID', 'Title', 'Genres']
movies.drop(['Genres'], axis=1, inplace=True)
print(movies[:5])
movies.loc[1, 'Title'] #the second row is the movie with id 2, but not necessarily<jupyter_output><empty_output><jupyter_text>The function:<jupyter_code>def recommend(movie_id, matrix):

# first we have to check if input is valid:
    if isinstance(movie_id, int) and isinstance(movie_id, np.int) and movie_id > 0 and movie_id <= matrix.shape[0]:
# initialize cos vector:
        cos = np.zeros(matrix.shape[0])
# find which row of a matrix is our movie:
        movie_loc = ratings_pivot.index.get_loc(movie_id)
# calculate cos sim between our movie and every other: 
        for i in range(0, matrix.shape[0]):
            a = np.dot(matrix[movie_loc], matrix[i])
            b = np.linalg.norm(matrix[movie_loc]) * np.linalg.norm(matrix[i])
            cos[i] = abs(a/b)

# sort in descending order:
        order = np.argsort(-cos)
# find ten best fits (the first one is going to be our movie): 
        ten = order[1:11]
        print('Recommended movies are:')
        print(ten)
        ten_most_similar = movies.loc[movies['MovieID'].isin(ten)]

    else:
        print('Invalid movie ID.')
    return(ten_most_similar)<jupyter_output><empty_output><jupyter_text>Let us try with movie ID 2, 'Jumanji (1995)':<jupyter_code>recommend(2, a_reduced)<jupyter_output>Recommended movies are:
[3256  632   58 1980  308 1825  651 1787 3206 1873]<jupyter_text>6. Repeat the same process except now instead of using SVD you will use PCA to get the eigenvectors.7. You will require co-variance matrix as an input to your eig function.Matrix a is already standardized and NaNs are filled with 0.  Eigenvectors and eigenvalues:<jupyter_code>cov_mat = np.cov(a.T)
print(cov_mat.shape)
eig_vals, eig_vecs = np.linalg.eig(cov_mat)
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
eig_pairs_sorted = sorted(eig_pairs, key=lambda x: x[0])

eig_vec_reduced = np.stack([vec for val, vec in eig_pairs[:50]], axis=1)
print(eig_vec_reduced.shape)<jupyter_output>(6040, 50)<jupyter_text>8. Use that same steps after that to get 50 components and use cosine similarity to get the results.<jupyter_code>a_reduced_PCA = np.matmul(a, eig_vec_reduced)
print(a.shape)
print(eig_vec_reduced.shape)
print(a_reduced_PCA.shape)
recommend(2, a_reduced_PCA)<jupyter_output>Recommended movies are:
[3256  632   58 1980  308 1825  651 1787 3206 1873]<jupyter_text>9. Compare the results for SVD and PCA.<jupyter_code>print('10 recommended movies for the movie with id 2, Jumanji (1995) using SVD:')
print(recommend(2, a_reduced))
print('10 recommended movies for the movie with id 2, Jumanji (1995) using PCA:')
print(recommend(2, a_reduced_PCA))<jupyter_output>10 recommended movies for the movie with id 2, Jumanji (1995) using PCA:
[3256  632   58 1980  308 1825  651 1787 3206 1873]
      MovieID                                              Title
57         58                   Postino, Il (The Postman) (1994)
305       308                         Three Colors: White (1994)
627       632        Land and Freedom (Tierra y libertad) (1995)
645       651                              Superweib, Das (1996)
1728     1787  Paralyzing Fear: The Story of Polio in America...
1758     1825                          Player's Club, The (1998)
1804     1873                             Misérables, Les (1998)
1911     1980     Friday the 13th Part VII: The New Blood (1988)
3137     3206                            Against All Odds (1984)
3187     3256                               Patriot Games (1992)
{'fraction_non_alphanumeric': 0.09784979744468682, 'fraction_numerical': 0.06310377064506077, 'mean_word_length': 3.168181818181818, 'pattern_counts': {'":': 2, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '12818', 'license': 'MIT', 'n_tokens_mistral': 2605, 'n_tokens_neox': 2223, 'n_words': 689, 'path': 'SVD movie recommender-lab2.ipynb', 'repo_name': 'MiskovicMilica/Data-science-portfolio', 'size': '6427'}