<jupyter_start><jupyter_text>**Helpful Links: Where the Data Lives**Open Academic Society: [Project Page](https://www.openacademic.ai/oag/)Microsoft Research: [MS Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/)<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Step 1 - Data Import, Clean, and Filter<jupyter_code>model_df = pd.read_json('../machine_learning/data/mag_papers_0/mag_subset20K.txt', lines=True)
model_df.shape

model_df.columns
# filter out non-English articles
# keep abstract, authors, fos, keywords, year, title
model_df = model_df[model_df.lang == 'en'].drop_duplicates(subset = 'title', keep = 'first') \
                                          .drop(['doc_type', 'doi', 'id', 'issue', 'lang', 'n_citation', 'page_end', 
                                                 'page_start', 'publisher', 'references','url', 'venue', 'volume'], axis=1)

model_df.shape
model_df.head(5)<jupyter_output><empty_output><jupyter_text>(1) raw data > algorithm w/ XKCD comic  Content Based Recommendation using cosine similarity How to go about building a recommender system? Let's start simple with a few fields. We'll calculate the cosine similarity between two items, then rank the results to choose a "most similar" paper for each input. We can already see that this dataset will need some wrangling. Lists and dictionaries are good for data storage, but not [tidy](http://vita.had.co.nz/papers/tidy-data.html) or well-suited for machine learning without some unpacking.<jupyter_code>tmp = model_df.fos.fillna('0')
print(type(tmp))
tmp[:10]

## get words from list, then unique, get feature-array finally

unique_fos = sorted(list({ feature
                          for paper_row in model_df.fos.fillna('0')
                          for feature in paper_row }))

unique_year = sorted(model_df['year'].astype('str').unique())

len(unique_fos + unique_year)
print(type(unique_fos))
print(unique_year[-5:])
print(unique_fos[-10:])

model_df.shape[0] - pd.isnull(model_df['fos']).sum()
len(unique_fos)
## since the original list is too large(with size 7604), here only get 15 items of them randomly

import random
[unique_fos[i] for i in sorted(random.sample(range(len(unique_fos)), 15)) ]
def feature_array(x, unique_array):
    row_dict = {}
    for i in x.index:
        var_dict = {}
        
        for j in range(len(unique_array)):
            if type(x[i]) is list:
                if unique_array[j] in x[i]:
                    var_dict.update({unique_array[j]: 1})
                else:
                    var_dict.update({unique_array[j]: 0})
            else:    
                if unique_array[j] == str(x[i]):
                    var_dict.update({unique_array[j]: 1})
                else:
                    var_dict.update({unique_array[j]: 0})
        
        row_dict.update({i : var_dict})
    
    print(str(row_dict[0])[:500])
    feature_df = pd.DataFrame.from_dict(row_dict, dtype='str').T
#     feature_df = pd.DataFrame.from_dict(row_dict, dtype='str')
    
    return feature_df
%time year_features = feature_array(model_df['year'], unique_year)
print(year_features.shape)
print(len(unique_year))
year_features[:5]
%time fos_features = feature_array(model_df['fos'], unique_fos)

from sys import getsizeof
print('Size of year feature array: ', getsizeof(year_features))
print('Size of fos feature array: ', getsizeof(fos_features))
print(fos_features.shape)
print(len(unique_fos))
fos_features[:5]

year_features.shape[1] + fos_features.shape[1]
# now looking at 10399 x  7760 array for our feature space

%time first_features = fos_features.join(year_features).T

first_size = getsizeof(first_features)

print('Size of first feature array: ', first_size)<jupyter_output>CPU times: user 31.2 s, sys: 3.83 ms, total: 31.2 s
Wall time: 31.1 s
Size of first feature array:  4680956578<jupyter_text>Let's see how our current features perform at giving us a good recommendation. We'll define a "good" recommendation as a paper that looks similar to the input. We will start with a simple example of building a recommender with just a few fields, building sparse arrays of available features to calculate for the cosine similary between papers. We will see if reasonably similar papers can be found in a timely manner.<jupyter_code>first_features.shape
first_features.head()
first_features.tail()
from scipy.spatial.distance import cosine

def item_collab_filter(features_df):
    item_similarities = pd.DataFrame(index = features_df.columns, columns = features_df.columns)
    
    for i in features_df.columns:
        for j in features_df.columns:
            item_similarities.loc[i][j] = 1 - cosine(features_df[i], features_df[j])
    
    return item_similarities
first_features = first_features.astype(int)
%time first_items = item_collab_filter(first_features.loc[:, 0:1000])

print(first_items.shape)
first_items<jupyter_output>(515, 515)<jupyter_text>We need to get a better idea of how these features will translate to us getting a good recommendation. Do we have enough observations to move forward? Let's plot a heatmap to see if we have any papers that are similar to each other.<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
%matplotlib inline
sns.set()
ax = sns.heatmap(first_items.fillna(0), 
                 vmin=0, vmax=1, 
                 cmap="YlGnBu", 
                 xticklabels=250, yticklabels=250)
ax.tick_params(labelsize=12)
def paper_recommender(paper_index, items_df):
    print('Based on the paper: \nindex = ', paper_index)
    print(model_df.iloc[paper_index])
    top_results = items_df.loc[paper_index].sort_values(ascending=False).head(4)
    print('\nTop three results: ') 
    order = 1
    for i in top_results.index.tolist()[-3:]:
        print(order,'. Paper index = ', i)
        print('Similarity score: ', top_results[i])
        print(model_df.iloc[i], '\n')
        if order < 5: order += 1
first_items.index
paper_recommender(0, first_items)<jupyter_output>Based on the paper: 
index =  0
title       System and Method for Maskless Direct Write Li...
year                                                     2015
abstract    A system and method for maskless direct write ...
fos         [Electronic engineering, Computer hardware, En...
authors                                                   NaN
keywords                                                  NaN
Name: 0, dtype: object

Top three results: 
1 . Paper index =  260
Similarity score:  0.6
title       Rigorous high-precision computation of the Hur...
year                                                     2015
abstract    We study the use of the Euler-Maclaurin formul...
fos         [Arithmetic zeta function, Mathematical analys...
authors     [{'name': '<NAME>', 'org': 'RISC, J...
keywords    [arbitrary precision arithmetic, power series,...
Name: 498, dtype: object 

2 . Paper index =  788
Similarity score:  0.6
title       The role of the social worker in the high-risk...[...]
{'fraction_non_alphanumeric': 0.10559182514902073, 'fraction_numerical': 0.017030939540164634, 'mean_word_length': 3.204653937947494, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '126088', 'license': 'MIT', 'n_tokens_mistral': 2245, 'n_tokens_neox': 2108, 'n_words': 630, 'path': '09.01-05_[End-to-End_Example]_Recommender_Take_1.ipynb', 'repo_name': 'sondywoo/feature-engineering-book', 'size': '7057'}