<jupyter_start><jupyter_text>Information Retrieval - Artificial Intelligence This notebook retrieves abstracts relevant to AI and then uses topic modeling to analyze the chosen abstracts.  Three info retrieval techniques are used: Literal Term Matching, TF-IDF, and Latent Semantic Indexing.  These are linear algebra techniques.  We use the Scikit-Learn library.<jupyter_code>import pandas as pd
import numpy as np
import pickle
import matplotlib.pyplot as plt
import gensim
import time
import seaborn as sns

from sklearn.decomposition import NMF, TruncatedSVD, LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import pairwise_distances

from gensim.models.phrases import Phrases, Phraser

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
# pull in entire dataframe

df = pd.read_pickle("/home/dab3dj/git/dspg21RnD/data/dspg21RnD/smaller-final-dataset.pkl")

df.reset_index(inplace = True)
#df.rename(columns={'index':'original index'}, inplace=True)
df.head()
# input needed for doc-term matrix creation in Scikit-Learn is one string per document (not a list of strings).  
# Original data 'ABSTRACT' is already in this form, but not the tokens in "final_frqwds_removed"

tokens = df["final_frqwds_removed"]

docs = [] # docs will contain the processed tokens in string form (1 string per abstract)

for abstract in tokens:
    docs.append(" ".join(abstract))
    
docs = pd.Series(docs)<jupyter_output><empty_output><jupyter_text>Functions needed for all info retrieval approaches<jupyter_code># Create query vector 

def create_query(words, terms):
    
    # words: search query words
    # terms: terms in corpus
    
    q = np.zeros(len(terms))  # number of terms

    idx = []
    for word in query_words:
        idx.append(terms.index(word))

    q[idx] = 1
    
    return q
def return_top_abstracts(docs, scores, top_n):
    
    '''
    docs: Series that contains abstract
    scores: scores of abstracts
    top_n: return the top_n abstracts given by idx, if top_n = -1 return all abstracts
    '''
    # sort scores in descending order
    scores_sorted_idx = np.argsort(scores)[::-1]
    
    if top_n == -1:
        n = sum(scores > 0)
        ix = scores_sorted_idx[:n]
    else:
        ix = scores_sorted_idx[:top_n]
    
    print(ix[0:10])
    
    return ix, docs[ix]
    
def create_result_df(abstracts, scores):
    
    df = pd.DataFrame()
    df["abstracts"] = abstracts
    df["scores"] = scores
    
    return df<jupyter_output><empty_output><jupyter_text>Literal Term Matching - Frequency Count Document-Term MatrixThis will return all abstracts in the corpus with exact word matches to the query.  Results will be return in sorted order of how high the query scores with each abstract. A high score means more occurences of the query words in the abstract.<jupyter_code># Note - we are now using the spaCy stopwords list instead of nltk.  It is more comprehensive.
# ONLY USE THIS FUNCTION if using raw text to search

'''
def create_stopwords():
      
    """ creates list of stopwords. stop words include the general English list and any additional we see sneaking 
    through.  """
    
    spacy_stop_words = STOP_WORDS

    # more stop words that do not add meaning to topics
    additional_stopwords = {'addition', 'specifically', 'similar','including', 'particular', 
                            'furthermore','include', 'includes','overall', 'finally', 'specific', 
                            'additional'} 
           
    sw = spacy_stop_words.union(additional_stopwords)
    
    return sw
    
'''<jupyter_output><empty_output><jupyter_text>Create Document-Term Matrix<jupyter_code># Create document-term matrix based on count frequencies

#stop_words = create_stopwords()

vectorizer = CountVectorizer(lowercase=True, min_df=1)
doc_term_matrix = vectorizer.fit_transform(docs)
doc_term_matrix.shape
terms = vectorizer.get_feature_names()<jupyter_output><empty_output><jupyter_text>Create Query Words - list the search termsA query is just a list of words to search for in the corpus.  We will use the same query for all three info retrieval techniques.<jupyter_code>'artificial_intelligence' in terms
# ONLY USE FOR RAW TEXT STRING MATCHING

# tokens with - or strange symbol between words

'''
count = 0
idx = []

for ix, abstract in enumerate(df['ABSTRACT']):
    if 'artificial intelligence' in abstract.lower(): 
        count = count + 1
        idx.append(ix)
    elif 'artificialintelligence' in abstract.lower(): 
        count = count + 1
        idx.append(ix)
    elif 'artificially intelligent' in abstract.lower(): 
        count = count + 1
        idx.append(ix)
    #else: 
        # do nothing

print(count)

'''
# CHANGE QUERY WORDS HERE 

# these words were found by analyzing all tokens with some form of 'artificial intelligence'
query_words = ['artificial_intelligence', 'artificial_intelligence_ai', 
               'artificial_intelligence_machine_learning', 'artificialintelligence', 'artificially_intelligent',
               'artificial_intelligence_aaai', 'artificial_intelligence_ijcai', 'artificialintelligence_ai',
               'artificialintelligent'
              ] 

#'ai', 'artificial', 'intelligence' 
              
q = create_query(query_words, terms)
# calculate the score for each document against the query. Docs with more occurences of the query words 
# will score higher

f_scores = doc_term_matrix.dot(q)
sum(f_scores >0)  # how many abstracts include at least one of the query words

# some are being left off from raw counts
# sort scores in descending order

f_scores_sorted = np.sort(f_scores)[::-1]
f_scores_sorted[:10]
f_idx, f_top_abstracts = return_top_abstracts(docs, f_scores, -1)  # CHANGE NUMBER OF TOP DOCS RETURNED
f_top_abstracts
f_top_abstracts.iloc[0]
df['ABSTRACT'][493912]<jupyter_output><empty_output><jupyter_text>TF-IDF Document-Term MatrixThis approach is similar to Literal Term Matching using frequency counts in the document-term matrix.  However, instead of using frequency counts, the entries of the document-term matrix are weighted using TF-IDF.<jupyter_code># Find doc-term matrix using TF-IDF weighting

tf_idf_vectorizer = TfidfVectorizer(lowercase=True, min_df=1)
tf_idf = tf_idf_vectorizer.fit_transform(docs)
tf_idf_terms = tf_idf_vectorizer.get_feature_names()  # these terms are the same as the terms created from the 
                                                      # frequency count document-term matrix, so we do not need to
                                                      # recreate the query vector.
tf_idf_terms == terms
# calculate the score for each document against the query. Docs with more occurences of the query words 
# will score higher

tf_idf_scores = tf_idf.dot(q)
sum(tf_idf_scores >0)   # how many abstracts include at least one of the query words
# sort scores in descending order

tf_idf_scores_sorted = np.sort(tf_idf_scores)[::-1]
tf_idf_scores_sorted[:10]
tfidf_idx, tfidf_top_abstracts = return_top_abstracts(docs, tf_idf_scores, -1)  # CHANGE NUMBER OF TOP DOCS RETURNED
tfidf_top_abstracts
tfidf_top_abstracts.iloc[0]
df['ABSTRACT'][235101]<jupyter_output><empty_output><jupyter_text>Latent Semantic Indexing (LSI) ApproachLSI Uses the TF-IDF matrix.  LSI is a tecnique that utilizes a truncated Singular Value Decomposition of the document-term matrix.  Basically, LSI still returns relevant documents to the query; however some of the documents returned may not include the exact search terms!  LSI is finding the latent or hidden relationships in the terms.<jupyter_code># Find the Truncated SVD of the TF-IDF matrix

lsa = TruncatedSVD(n_components=500, random_state=1)  # CHANGE THE NUMBER OF COMPONENTS - NOTE: MORE COMPONENTS 
                                                      # GIVES YOU A MORE ACCURATE APPROXIMATION OF THE DOC-TERM 
                                                      # MATRIX, BUT IS ALSO MORE EXPENSIVE AND MAY NOT LEAD TO THE 
                                                      # BEST INFO RETRIEVAL RESULTS.
USigma = lsa.fit_transform(tf_idf)
Vtrans = lsa.components_
# transform query to be in same space as documents

q = q.reshape(1,-1)
qhat = lsa.transform(q)
print(qhat.shape)
print(USigma.shape)
print(Vtrans.shape)
lsa_scores = pairwise_distances(qhat, USigma, metric='cosine', n_jobs=7)  # CHANGE N_JOBS TO BE NUMBER OF CORES - 1
lsa_scores.shape
lsa_scores
lsa_scores[0]
sum(lsa_scores[0] > 0)  # how many abstracts scored above 0
# sort scores in descending order

lsa_scores_sorted = np.sort(lsa_scores[0])[::-1]
lsa_scores_sorted[:10]
lsa_idx, lsa_top_abstracts = return_top_abstracts(docs, lsa_scores[0], 50)  # CHANGE NUMBER OF TOP DOCS RETURNED
lsa_top_abstracts
lsa_top_abstracts.iloc[41]
df['ABSTRACT'][344482]
# I don't think LSI is giving us AI results - I spot checked 5-10 of these.<jupyter_output><empty_output><jupyter_text>Create pandemics corpusWe use the results of our three information retrieval techniques to create a new, smaller corpus that only contains abstracts relevant to the query.<jupyter_code>docs_ix = np.concatenate([f_idx, tfidf_idx]) #lsa_idx])
docs_idx = np.unique(docs_ix)
docs_idx.shape<jupyter_output><empty_output><jupyter_text>**create case-study corpus**<jupyter_code>ai_corpus = df.loc[docs_idx, :]
ai_corpus.shape
# save or read corpus

#ai_corpus.to_pickle("./ai_corpus.pkl")
#ai_corpus = pd.read_pickle("ai_corpus.pkl")
lim_docs = ai_corpus["final_frqwds_removed"]
# input needed for LDA, NMF (all from Scikit-Learn) is one string per document (not a list of strings)

text = []

for token_list in lim_docs:
    text.append(" ".join(token_list))
len(lim_docs)<jupyter_output><empty_output><jupyter_text>Topic Modeling with relevant AI abstracts<jupyter_code># function slightly modified from https://nlpforhackers.io/topic-modeling/

def print_topics(model, vectorizer, top_n=10):
    for idx, topic in enumerate(model.components_):  # loop through each row of H.  idx = row index.  topic = actual row
        print("\nTopic %d:" % (idx))
        #print([(vectorizer.get_feature_names()[i], topic[i])  # printing out words corresponding to indices found in next line
                        #for i in topic.argsort()[:-top_n - 1:-1]])  # finding indices of top words in topic
            
        print_list = [(vectorizer.get_feature_names()[i], topic[i])  
                        for i in topic.argsort()[:-top_n - 1:-1]]
        for item in print_list:
            print(item)
        
# Create a TF-IDF document-term matrix for the AI corpus 

# TRY DIFFERENT PARAMETERS IN THE TF-IDF DOC-TERM MATRIX SET-UP
nmf_vectorizer = TfidfVectorizer(max_df=1.0, min_df=3, lowercase=True) #, max_features=int(len(lim_docs)/2))

# by default TfidfVectorizer has l2 normalization for rows: 
# from Scikit Learn documentation: Each output row will have unit norm, either: * ‘l2’: Sum of squares of vector 
# elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied.

nmf_tf_idf = nmf_vectorizer.fit_transform(text)
nmf_tf_idf.shape
AI_terms = nmf_vectorizer.get_feature_names()
AI_terms[2830:2850]
# topic modeling with NMF

nmf_model = NMF(n_components=30, random_state=1)  # TRY DIFFERENT NUMBERS OF TOPICS
W = nmf_model.fit_transform(nmf_tf_idf)
H = nmf_model.components_
print_topics(nmf_model, nmf_vectorizer, 10)<jupyter_output><empty_output><jupyter_text>properties of AI corpus<jupyter_code>ai_corpus.head()
# project terms - worth looking into??

ai_corpus['PROJECT_TERMS'].iloc[500] # does not have artificial intelligence as key term
ai_corpus['ABSTRACT'].iloc[500]
# plot number of abstracts by department

y = ai_corpus["DEPARTMENT"].value_counts()
y.plot(kind = 'bar')
plt.ylabel('Number of Abstracts')
plt.title('Abstract Count by Department');

print(y)
# plot number of abstracts by project start year

# extract year from project start date

def getYear(a):   
    a = str(a)
    if a.find("/"):
        splitdate = a.split("/")
        if len(splitdate) == 3:
            a = splitdate[2]
        else:
            a = splitdate[0]
    year = str(a)
    return year

ai_corpus['START_YEAR'] = ai_corpus['PROJECT_START_DATE'].apply(getYear)
y = ai_corpus["START_YEAR"].value_counts().sort_index()
y.plot(kind = 'bar')
plt.ylabel('Number of Abstracts')
plt.title('Abstract Count by Project Start Year');

print(y)
# plot dollars by year

dollars_by_year = ai_corpus.groupby(by = ['START_YEAR']).sum()
dollars_by_year["FY_TOTAL_COST"] = dollars_by_year["FY_TOTAL_COST"]/1000000
dollars_by_year.plot.bar(y = "FY_TOTAL_COST")
plt.ylabel('Dollars (millions)')
plt.title('Dollars Spent by Project Start Year');

# plot dollars by year & department

dollars_by_deptNyear = ai_corpus.groupby(by = ['START_YEAR','DEPARTMENT']).sum()
dollars_by_deptNyear["FY_TOTAL_COST"] = dollars_by_deptNyear["FY_TOTAL_COST"]/1000000

dollars_by_deptNyear
dollars_by_deptNyear.plot.bar(y = "FY_TOTAL_COST", figsize=(15,5))
plt.ylabel('Dollars (millions)')
plt.title('Dollars Spent by Project Start Year and Department');
dollars_by_deptNyear = dollars_by_deptNyear.reset_index()
fig, ax = plt.subplots(figsize=(10,5))
sns.barplot(ax=ax, x="START_YEAR", y="FY_TOTAL_COST", hue="DEPARTMENT", data=dollars_by_deptNyear)
# next step - look at documents containing machine learning applications (such as topic 21)

topic_docs = W[:, 21]
sum(topic_docs > 0) 
max_score = max(topic_docs)
max_score
topic_docs[0:20]
print(lim_docs.iloc[2])
idx = np.where(topic_docs == max_score)
idx
idx = np.where(topic_docs > 0.3)
idx
topic_docs[10:30]
print(lim_docs.iloc[745]) # psychosis topic with AI component
ai_corpus["ABSTRACT"].iloc[759]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09100797679477883, 'fraction_numerical': 0.010949963741841914, 'mean_word_length': 4.0814296241709656, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '53968', 'license': 'MIT', 'n_tokens_mistral': 4703, 'n_tokens_neox': 4437, 'n_words': 1305, 'path': 'src/Summer-2020-approach/AI-corpus-creation-CO.ipynb', 'repo_name': 'DSPG-Young-Scholars-Program/dspg21RnD', 'size': '13790'}