<jupyter_start><jupyter_text>**FAISS Document Similarity Search for CORD-19 dataset** This notebook draws from [<NAME>'s work](https://www.kaggle.com/crispyc/coronawhy-task-ties-patient-descriptions) on the Kaggle round 2 competition regarding the CORD-19 dataset.  **Data requirements*** document [embeddings](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge?select=cord_19_embeddings) (currently document embeddings derived from SPECTER)**Input:** (either, not both)* `k` - the number of similar articles desired* an article identifier (`cord_uid`)* an article title**Output:**:* list of IDs of similar articles with length `k`  **Imports & Data Sources**<jupyter_code>#!pip install faiss-cpu
import pandas as pd
import numpy as np
import os
import faiss

#Connecting to MongoDB
# !pip install pymongo
import pymongo
#Setup embeddings filepath
import os
from google.colab import drive

drive.mount('/content/gdrive', force_remount=True)
root = os.getcwd()
download_destination = 'gdrive/My Drive/COVID-19/lit_review_tool'
cwd = os.path.join(root, download_destination)
os.chdir(cwd)
print('Current working directory: ', os.getcwd())
#Import embeddings
def read_cord_19_embeddings(filename):                                      
    # emb_path = '/'.join(('embeddings', filename))
    emb_path = '/'.join((cwd, 'cord_19_embeddings', filename))
    print('emb_path: ', emb_path)
    emb = pd.read_csv(emb_path, header = None, index_col = 0)
    print(emb.head())
    return emb

filename = 'cord_19_embeddings_2020-07-31.csv'
emb = read_cord_19_embeddings(filename)
emb.shape<jupyter_output>emb_path:  /content/gdrive/My Drive/COVID-19/lit_review_tool/cord_19_embeddings/cord_19_embeddings_2020-07-31.csv
               1         2         3    ...       766       767       768
0                                       ...                              
ug7v899j -2.939984 -6.312201 -1.045903  ...  5.811529 -2.932384  0.305205
02tnwd4m  4.688466 -6.284460  1.228246  ...  4.176544 -5.686561  0.512474
ejv2xln0  0.559920 -6.754057  2.271945  ...  4.521635 -2.847652 -1.536697
2b73a28n  2.352540 -3.997074  2.399774  ... -0.573785 -1.402024  3.922583
9785vg6d -1.290922 -5.849864  1.557572  ...  4.226255  1.244299 -4.117685

[5 rows x 768 columns]<jupyter_text>**Get metadata from MongoDB**This connection replaces the original download of metadata.  I don't believe any additional data (e.g. full text, abstracts, etc.) is necessary for fully processing the original notebook.<jupyter_code>#Setup link to CoronaWhy's MongoDB

# Read-only credentials to CoronaWhy MongoDB service
mongouser = 'coronawhyguest'
mongopass = '<PASSWORD>'
cord_version = 'v22'

# mongo_URI = 'mongodb://{}:{}@mongodb.<EMAIL>awhy.<EMAIL>'.format(mongouser, mongopass)
mongo_URI = 'mongodb://cord19-rw:corona<EMAIL>'
client = pymongo.MongoClient(mongo_URI)
db = client.get_database('cord19')
print('Existing collections: ', db.list_collection_names())
collection = db[cord_version]

#Test query
# pd.DataFrame(collection.find({'title': "Nitric oxide: a pro-inflammatory mediator in lung disease?"})).head() #should have 2 results as of 8/3/2020)
# Creating a matrix to store article embeddings 
xb = np.ascontiguousarray(emb).astype(np.float32)
# Assigning dimension for the vector space
d = xb.shape[1]<jupyter_output><empty_output><jupyter_text>**Creating faiss search index**<jupyter_code># Building the index
index = faiss.IndexFlatIP(d) #IndexFlatIP: taking inner product of the vectors
print('Index training complete: ', index.is_trained)

faiss.normalize_L2(xb) #with normalized vectors, the inner product (IP, of IndexFlatIP) becomes cosine similarity
index.add(xb)# Adding vectors to the index
                  
print('Total rows in index: ', index.ntotal)<jupyter_output>Index training complete:  True
Total rows in index:  204823<jupyter_text>**Create example data**For ease of testing.  This could eventually be a feature in a search engine.<jupyter_code>cord_uid_examples = ['02tnwd4m', '8zchiykl', 'gdsfkw1b', 'byp2eqhd', '7gk8uzo0', '0mtmodmo']

title_examples = ['Nitric oxide: a pro-inflammatory mediator in lung disease?',
                  'The 21st International Symposium on Intensive Care and Emergency Medicine, Brussels, Belgium, 20-23 March 2001',
                  'Protein secretion in Lactococcus lactis: an efficient way to increase the overall heterologous protein production',
                  'Immune pathways and defence mechanisms in honey bees Apis mellifera',
                  'Species-specific evolution of immune receptor tyrosine based activation motif-containing CEACAM1-related immune receptors in the dog',
                  'Novel, Divergent Simian Hemorrhagic Fever Viruses in a Wild Ugandan Red Colobus Monkey Discovered Using Direct Pyrosequencing']
#TWO OPTIONS:

#1) Feed in `cord_uid` directly (in list format)
    # No need to search MongoDB for `cord_uid` - feed this straight into similarity search

#2) Feed in title, and get CORD_UID
def get_cord_uid_for_title(study_title):
    cord_uids = list(collection.find({'title': str(study_title)}, {'cord_uid'})) #search by title, return cord_uid
    return cord_uids
#Get `cord_uid` ONLY
result_cord_uids = set([result['cord_uid'] for result in get_cord_uid_for_title(title_examples[0])]) #filters out `_id` column
result_cord_uids
#Prepare query vector
query_vec = np.ascontiguousarray(emb.loc[result_cord_uids]).reshape(1,-1).astype(np.float32)
faiss.normalize_L2(query_vec)<jupyter_output><empty_output><jupyter_text>**Run FAISS search**Remember, this is a document similarity search.<jupyter_code>k = 6
similar_id_list=[]

def document_similarity_search(query_vec, k, return_cord_uid=False, return_metadata=False):
    D, I = index.search(query_vec, k)
    similar_id_list.extend(I.tolist()[0])
    similar_cord_uid_list = [cid for cid in emb.iloc[similar_id_list].index if cid not in result_cord_uids]
    if return_cord_uid:
        print('Articles similar to {}: '.format(result_cord_uids), similar_cord_uid_list, '\n')
    if return_metadata:
        mongo_results = pd.DataFrame(collection.find({'cord_uid': {'$in':similar_cord_uid_list}})) #the last two ('hkrljpn3', 'ocu597fg') are apparently ot present in MongoDB
        return mongo_results
document_similarity_search(query_vec, k, return_cord_uid=True, return_metadata=True) #the last two ('hkrljpn3', 'ocu597fg') are apparently ot present in MongoDB<jupyter_output>Articles similar to {'02tnwd4m'}:  ['bzub2kkv', '6v0y6xsa', 'ka676pli', 'hkrljpn3', 'ocu597fg']
{'fraction_non_alphanumeric': 0.11015780603646391, 'fraction_numerical': 0.05576834686686073, 'mean_word_length': 4.435470441298918, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '474483', 'license': 'MIT', 'n_tokens_mistral': 2428, 'n_tokens_neox': 2154, 'n_words': 574, 'path': 'faiss_document_similarity_search.ipynb', 'repo_name': 'alofgran/coronawhy_lit_review_tool', 'size': '6550'}