<jupyter_start><jupyter_text>Capstone Project Submission- Student name: <NAME>- Student pace: self-paced- Scheduled project review date/time: July 27, 2020- Instructor name: <NAME>- Blog post URL: https://bronwencc.github.io/pandas_dataframe_quick-start_guide_python  AbstractThe problem was to see whether the text of abstracts authored by Nobel Prize winners that were highly cited was different from the text of abstracts from highly-cited publications authored by scientists who were listed as an author on one of the Nobel Prize winners' highly-cited publications.  The publications were from Google Scholar and the information about the Nobel Prize winners was from the Nobel Prize website.  Publications may be books as well as journal articles, and abstract text was assumed to be in English, although there is at least one suspected instance of French.Then, the publications that had abstract text (a non-null `bib_abstract`) were put into one dataframe and the text was stripped of punctuation, tokenized and lemmatized with `nltk`.  After that, I found frequency distributions and subsequently TF-IDF (term frequency - inverse document frequency) values for each word (or name or number) in all the abstracts for each abstract.  These values were reduced to two dimensions with `sklearn`'s `TSNE` and plotted with varying color codes: author name, field of study (as related to the field in which the connected Nobel Prize was won), and whether the author of the publication was a Nobel Prize winner.The TF-IDF values were put through Sequential models to get predictions for field of study (4 possibilities): 19% accuracy and for coauthor (Nobel Prize winner or not): 58% accuracy.  A t-test found the average TF-IDF values to not be significantly different between abstracts from a non-Prize winner's author page and those from a Prize-winner's author page.  Import Statements<jupyter_code>import pandas as pd
import numpy as np
from keras.layers import Input, Dense, LSTM, Embedding
from keras.layers import Dropout, Activation, Bidirectional, GlobalMaxPool1D
from keras.models import Sequential
from keras import initializers, regularizers, constraints, optimizers, layers
from keras.preprocessing import sequence
from sklearn.manifold import TSNE
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, TfidfTransformer, CountVectorizer
from scipy import stats
from statsmodels.stats.multicomp import pairwise_tukeyhsd
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use("seaborn")
from nltk.tokenize import word_tokenize
from nltk import FreqDist
from nltk.corpus import stopwords
import string
#use WordNet database to reduce words to roots (lemmatize)
from nltk.stem.wordnet import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
import nltk
nltk.download("punkt") #download Punkt sentence tokenizer
nltk.download('wordnet') #download wordnet for WordNetLemmatizer
np.random.seed(321)<jupyter_output><empty_output><jupyter_text>ObtainThe bulk of obtaining is in the obtain.ipynb notebook, where it retrieved and parsed results through my modified `scholarly` package.This section reads in the .CSV files that were created by putting together information from `newscholarly` search results for filled publication information.  It also creates new dataframe with additional informative columns consisting of those records that have abstract text (where `bib_abstract` is not null).<jupyter_code>#all filled publications that were first five results for Nobel Prize winners
publidf = pd.read_csv("csvdata\publications.csv",index_col=[0])
publidf.reset_index(drop=True, inplace=True)
publidf.info()
#all first five filled publications for at most three authors that were first 
#listed on first five results for Nobel Prize winners
coapublidf = pd.read_csv("csvdata\copublications.csv",index_col=[0])
coapublidf.reset_index(drop=True,inplace=True)
coapublidf.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 75 entries, 0 to 74
Data columns (total 23 columns):
bib_eprint         38 non-null object
bib_cites          75 non-null int64
citations_link     75 non-null object
url_scholarbib     0 non-null float64
url_add_sclib      0 non-null float64
bib_abstract       44 non-null object
bib_author_list    0 non-null float64
bib_venue          0 non-null float64
bib_year           75 non-null int64
bib_gsrank         0 non-null float64
bib_title          75 non-null object
bib_url            44 non-null object
bib_author         75 non-null object
bib_listauthors    75 non-null object
bib_journal        64 non-null object
bib_volume         65 non-null float64
bib_number         60 non-null object
bib_publisher      61 non-null object
bib_pages          69 non-null object
source             75 non-null object
id_citations       75 non-null object
cites_per_year     75 non-null object
fileID             75 non-null object
dtypes: float64(6), int6[...]<jupyter_text>Create author identifier from fileID (an index number plus the last four characters of an author's name) for both dataframes listing filled publications.<jupyter_code>publidf["authID"]=[ident[:-1] for ident in publidf["fileID"]]
coapublidf["authID"]=[ident[:-1] for ident in coapublidf["fileID"]]
#how many publications that have an abstract are there for "co-authors"
coapublidf[coapublidf.bib_abstract.isna()==False].authID.value_counts()
#add coauthor binary column
coapublidf["coauthor"]=1
publidf["coauthor"]=0<jupyter_output><empty_output><jupyter_text>ScrubRemove author's publications known to be an incorrect search result and combine dataframes into one with all publications that have abstracts.<jupyter_code>#drop rows at indices where authID is "1Bado"
coapublidf.drop(index=coapublidf[coapublidf.authID=="1Bado"].index,inplace=True)
coapublidf.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 70 entries, 0 to 74
Data columns (total 25 columns):
bib_eprint         34 non-null object
bib_cites          70 non-null int64
citations_link     70 non-null object
url_scholarbib     0 non-null float64
url_add_sclib      0 non-null float64
bib_abstract       40 non-null object
bib_author_list    0 non-null float64
bib_venue          0 non-null float64
bib_year           70 non-null int64
bib_gsrank         0 non-null float64
bib_title          70 non-null object
bib_url            40 non-null object
bib_author         70 non-null object
bib_listauthors    70 non-null object
bib_journal        59 non-null object
bib_volume         60 non-null float64
bib_number         55 non-null object
bib_publisher      57 non-null object
bib_pages          64 non-null object
source             70 non-null object
id_citations       70 non-null object
cites_per_year     70 non-null object
fileID             70 non-null object
authID             70 no[...]<jupyter_text>With 44 out of 70 or 75 for each dataframe having a non-null abstract, put together dataframe of those with abstract and whether from coauthor or prize-winner list.  This is excepting 1Bado from the `coapublidf`, which I know to be information for a different author than intended.Matching names was one of the more difficult aspects of getting the desired search results.<jupyter_code>#concatenate where abstract is not null
abstractdf = pd.concat([coapublidf[coapublidf.bib_abstract.isna()==False],publidf[publidf.bib_abstract.isna()==False]])
abstractdf.info()
#reset index and not add as a column the one being replaced (drop=True)
abstractdf.reset_index(drop=True,inplace=True)
abstractdf.tail()
#save this dataframe
abstractdf.to_csv("csvdata\pubs_with_abstracts.csv")
abstractdf = pd.read_csv("csvdata\pubs_with_abstracts.csv", index_col=[0])
abstractdf.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 84 entries, 0 to 83
Data columns (total 25 columns):
bib_eprint         60 non-null object
bib_cites          84 non-null int64
citations_link     84 non-null object
url_scholarbib     0 non-null float64
url_add_sclib      0 non-null float64
bib_abstract       84 non-null object
bib_author_list    0 non-null float64
bib_venue          0 non-null float64
bib_year           84 non-null int64
bib_gsrank         0 non-null float64
bib_title          84 non-null object
bib_url            84 non-null object
bib_author         84 non-null object
bib_listauthors    84 non-null object
bib_journal        70 non-null object
bib_volume         74 non-null float64
bib_number         70 non-null object
bib_publisher      75 non-null object
bib_pages          76 non-null object
source             84 non-null object
id_citations       84 non-null object
cites_per_year     84 non-null object
fileID             84 non-null object
authID             84 no[...]<jupyter_text>Add field from Nobel Prize winner information to authorinfo.csv.<jupyter_code>authdf = pd.read_csv("csvdata/authorinfo.csv",index_col=[0])
authdf.reset_index(drop=True, inplace=True)
authdf.info()
#checking all fileID's are unique to each record
sum(authdf.fileID.value_counts())
#get sciences Nobel Prize winner information from 2010-2019
sciwindf = pd.read_csv("files/science10years.csv",index_col=[0])
sciwindf.info()
sciwindf["thor"]=[name[-4:] for name in sciwindf.name1] #last four characters of authorname
#be sure last four characters are unique
sum(sciwindf.thor.value_counts())
#create new empty column for winning area in authdf
authdf["area"]=np.nan
#add field if sciwindf.thor matches last 4 characters from fileID from authdf
for idx, nameid in zip(authdf.index,authdf.fileID):
    last4 = nameid[-4:]
    winarea = sciwindf[sciwindf.thor==last4].field
    area = None
    try:
        for i in winarea:
            area = i
    except:
        continue
        
    authdf.iloc[idx,-1]=area #last column is "area"
#it should be where area is missing, is also not Nobel Prize winners
authdf.head()
#save new file of author information with winning areas
authdf.to_csv("files/authors.csv")
#create column of last four characters of an author name from fileID to compare with abstractdf
authdf["name4"] = [fileid[-4:] for fileid in authdf["fileID"]]<jupyter_output><empty_output><jupyter_text>Use information from `authdf` to fill in `abstractdf` with Nobel Prize winning field for that author or coauthor.<jupyter_code>#initialize column to be null
abstractdf["area"]=None
#if an author in bib_listauthors matches an authdf.name4 and area is not NaN
for idx, authors in zip(abstractdf.index,abstractdf.bib_listauthors):
    authlist = authors.strip("[]").split(", ")#expecting string with appearance of a list: ['<NAME>', 'I M Here']
    area = None
    hasfield=False
    for auth in authlist:
        authname = auth.strip("'")#remove trailing or leading apostrophes
        abslast4 = authname[-4:] #find whether last four characters match a name in authdf and add that field
        winarea = authdf[authdf.name4==abslast4].area #find the winning subject area for that match
        try:
            for i in winarea: #winarea is a Series with an index, so iterating gives the desired value
                area = i
            hasfield = True
        except:
            continue
        if hasfield:#the first author listed in bib_listauthors that matches an author in authdf and has a useful result
            abstractdf.iloc[idx,-1]=area #last column is "area", that record will be classified in that area
            break #and leave the loop since it found an area for this list of authors for this record
        else:
            continue
#create list with one dataframe for each authID and concatenate back together
#after filling with nearby values in area
concatlist = []
for authorID in abstractdf.authID.value_counts().index: #set of authID values
    tempdf = abstractdf[abstractdf.authID==authorID].copy() #separate out each dataframe by authID
    tempdf.area.fillna(method="ffill",inplace=True) #fill forwards
    tempdf.area.fillna(method="bfill",inplace=True) #fill backwards
    concatlist.append(tempdf)
newabsdf = pd.concat(concatlist)
newabsdf[newabsdf.coauthor==0].tail()
#some values for area are missing from Nobel Prize winners (where authdf.coauthor is 0)
#add area if last four characters from newabsdf.authID matches fileID from authdf
for idx, nameid in zip(newabsdf.index,newabsdf.authID):
    if newabsdf.iloc[idx,-2]==0: #if the coauthor column is 0, the record is for a Nobel winner
        last4 = nameid[-4:]
        winarea = authdf[authdf.name4==last4].area #get area value from author info dataframe
        absarea = None
        hasarea = False
        try:
            for i in winarea:
                absarea = i
        except:
            continue #go on to next record
    
        if hasarea and newabsdf.iloc[idx,-1]==None:
            newabsdf.iloc[idx,-1]=absarea #"area" was last column added
            # the value at the corresponding index for this authID is set to the area from authdf
        continue
newabsdf.area.value_counts()<jupyter_output><empty_output><jupyter_text>This method of using the last four characters did not work completely well and left some records in `area` empty, so I manually searched for who were coauthors of <NAME> in my rough draft notebook and then looked up in `sciwindf` in which field they won.<jupyter_code>#example lookup for gham having alij as a coauthor
sciwindf[sciwindf.thor=="gham"]
#since gham won in chemistry, alij's publications will also be classified as such
prefindx = newabsdf[newabsdf.authID=="1alij"].index
newabsdf.loc[prefindx,"area"]="chemistry"<jupyter_output><empty_output><jupyter_text>I saved the resulting `newabsdf`, after manually adding the rest of the values for `area` as "abstractsinfo.csv", which can be read in below as `infodf`.<jupyter_code>infodf = pd.read_csv("csvdata/abstractsinfo.csv", index_col=[0])
infodf.info()
#sorting by index column, then dropping it
infodf.set_index(infodf["index"], inplace=True)
infodf.sort_index(inplace=True)
infodf.drop(columns=["index"],inplace=True)
infodf.tail()
sum(infodf.area.isna())
infodf.area.value_counts()<jupyter_output><empty_output><jupyter_text>Possible duplicates<jupyter_code>infodf[infodf.citations_link=="/scholar?cites=10229049581137351676"]<jupyter_output><empty_output><jupyter_text>Before truly beginning analysis, some publications may have been added more than once so I will remove those with enough similar information with `drop duplicates`; keeping one of each publication.<jupyter_code>infodf.drop_duplicates().info()
#there are no records that match completely, so I will specify where the abstract, eprint and citations_link are the same
finaldf = infodf.drop_duplicates(subset=["bib_abstract","citations_link","bib_eprint"])
finaldf.info()
#reset index and save as abstracts76
finaldf.reset_index(drop=True,inplace=True)
finaldf.to_csv("csvdata/abstracts76.csv")
finaldf.tail()<jupyter_output><empty_output><jupyter_text>ExploreWith dataframe of records that have abstracts, analyze abstract text for each with `nltk` and get TF-IDF values for each.<jupyter_code>def cleantokens(textstring):
    '''
    A function to remove punctuation, various other symbol characters, newlines, and English stopwords from textstring.
    It returns words or sequences of numbers (and/or letters) from textstring as tokens using nltk.word_tokenize.
    
    Args
        textstring (str) : A string of text, assumed to be in English, with words separated by spaces.
    
    Returns
        cleantokens (list) : lowercase words (including acronyms) from textstring not in stopwords.words("english")
    '''
    stopwordspunct = stopwords.words("english")
    stopwordspunct += list(string.punctuation)
    stopwordspunct += ["…","\+","''","‘","’","“","”","—","\*","à"] #other punctuation that was unaccounted for
    for symbol in "\+,.?!;-\\:\∼\~'\n\'…‘’“”—\*":#replace punctuation with an empty string, including ~ and ∼
        textstring = textstring.replace(symbol, '').lower() #put into lowercase
    tokenized = word_tokenize(textstring)
    #alternately:
    cleanedtokens = [word.lower() for word in tokenized if word not in stopwordspunct]
    return cleanedtokens<jupyter_output><empty_output><jupyter_text>The next code blocks get a list of lists of all words in all abstracts.<jupyter_code>abstractext = [abstract for abstract in finaldf.bib_abstract]#each entry in bib_abstract is one string
tokenslists = []
abstractlist = []
for each in abstractext: #a list of Strings
    tokens = cleantokens(each) #get tokens
    lemmlist = []
    for token in tokens:
        lemtoken = lemmatizer.lemmatize(token)
        lemmlist.append(lemtoken) #lemmatize each token, add as item to list
        abstractlist.append(lemtoken) #add to one big list with no distinction between abstracts
    tokenslists.append(lemmlist) #each list of lemmatized tokens is an item in the list
#freqdist for all words in all abstracts
absfreq = FreqDist(abstractlist)
absfreq.most_common(10)
len(absfreq)<jupyter_output><empty_output><jupyter_text>The most common is "cell" although there are 2500 unique words, numbers (and likely, names as well) in total for all abstract text.Now, I will get the TF-IDF values for each word for each abstract text.<jupyter_code>#get frequency dictionaries for each list of words
freqlist = []
for tokenslist in tokenslists:
    freqdistr = FreqDist(tokenslist)
    freqlist.append(freqdistr)<jupyter_output><empty_output><jupyter_text>The below are based on the TF-IDF lab comparing song lyrics https://github.com/learn-co-curriculum/ds-word-vectorization-lab/ that get term frequency - inverse term frequency.<jupyter_code>#get term frequency:
def proportional_freq(freqdict):
    '''
    If given a frequency dictionary, would return a new dictionary with the same keys and the frequencies relative to the
    total sum of the frequencies (total number of times all keys were counted): a key's given value / sum of all values.
    
    Args
        freqdict (dict or similar) : has words as keys and their frequencies in the text as the values
        
    Returns
        propor (dict) : same keys as freqdict, values are freqdict's value divided by the total sum of freqdict's values
    '''
    totalcount = sum(freqdict.values()) #the total number of words, summing up all the frequency counts
    
    propor=dict()
    
    for item, freq in freqdict.items(): #iterate over each item-frequency pair
        propor[item] = freq/totalcount
    
    return propor
#find inverse document frequency given a list of frequency dictionaries
def calc_inverse_freq(freqdlist):
    '''
    Takes in a list of frequency dictionaries and finds the inverse document frequency for every key. Returns one dict
    containing all of the set of keys across all freqdlist's dictionaries with the inverse document frequency as the values
    
    Args
        freqdlist (list) : list of frequency dictionaries that are part of the same corpus
    
    Returns
        inversedict (dict) : the keys are from the dictionaries in freqdlist's keys and the values are log-transformed
                            inverse document frequency (the number of dictionaries / how many dicts the key appears in)
    '''
    listlen = len(freqdlist)
    
    totfreqd = dict()
    for freqd in freqdlist: #create frequency dictionary with every key across all dictionaries in freqdlist
        for key in freqd.keys():
            totfreqd[key]=totfreqd.get(key,0)+1 #if key does not exist already, creates it with value of 1
            #totfreqd contains how many times a word appears at least once in a freqd from freqdlist (the number of 
            #documents a word appears in)
    
    inversedict=dict()
    for word, freq in totfreqd.items():#for every token-frequency pair in totfreqd
        #inversedict contains log of (the total number of dictionaries (documents) / how many documents word appears in)
        inversedict[word] = np.log(listlen/float(freq)) #the log of this quotient (to divide well, needs float type)
        # (so the difference between 10 and 20 is bigger than between 100 and 120)
    return inversedict
#for term frequency, I used FreqDist object from nltk
def termf_invdf(listofdicts):
    '''
    Finds term frequency-inverse document frequency for every key in each given frequency dictionary in listofdicts and
    returns the values in tfidf_dictslist list of dictionaries, each having every key from across listofdicts.
    
    Args
        listofdicts (list) : list of frequency dictionaries
    
    Returns
        tfidf_dictslist (list) : list of dictionaries, each with keys of all of the keys across listofdicts' dictionaries
                                and the values are term frequency * inverse document frequency derived from same
    '''
    idfdict = calc_inverse_freq(listofdicts)
    newdict = {i:0 for i in list(idfdict.keys())} #all the words from all dictionaries in the list
    
    tfidf_dictslist=[]

    for eachdict in listofdicts:
        tfidf_dict = newdict.copy() #dictionary with keys as all the tokens in the corpus and all have values of 0
        profreq = proportional_freq(eachdict) #gets term frequency
        for word in profreq.keys():
            tfidf_dict[word] = profreq[word]*idfdict[word] #multiply proportional frequency by inverse frequency
        tfidf_dictslist.append(tfidf_dict) #each dictionary lists every word in the corpus
        #if a word is not in eachdict, it has a value of 0 in its corresponding tfidf_dict
        
    return tfidf_dictslist<jupyter_output><empty_output><jupyter_text>Use frequency distributions FreqDist objects (similar to dictionaries) to get list of dictionaries of TF-IDF values for each abstract.<jupyter_code>tfidflist = termf_invdf(freqlist)
#put all TF-IDF values into dataframe format
tfidframe = pd.DataFrame(tfidflist)
#tfidflist is a list of dictionaries where each dictionary is every word's tfidf value for an abstract
tfidflist[41]["cell"]<jupyter_output><empty_output><jupyter_text>As a dataframe, each record is for one asbtract.<jupyter_code>tfidframe.head()
#interesting note: there are two different strings for micrometers
"µm"=="μm"
tfidframe.info()
#save TF-IDF values to .CSV file
tfidframe.to_csv("csvdata/tfidfvalues.csv")
tfidframe.describe()
#quick calculation for average tokens per abstract
totalwords=0
for i in tokenslists: #list of lists of tokens
    totalwords+=len(i) #how many tokens in each abstract = length of list
totalwords/finaldf.shape[0] #with number of rows in finaldf<jupyter_output><empty_output><jupyter_text>Most values are zero, likely indicating that most tokens are not in most abstracts, as there are 70 tokens on average in each abstract.  Of the tokens shown by `.describe()` at a glance, β is the most unique; with a maximum TF-IDF value of 0.2 (closest to 1).<jupyter_code>statsframe = tfidframe.describe()
statsframe.info()
#finding the highest values for which tokens
statsframeT = statsframe.T
statsframeT[statsframeT["max"]>.37]<jupyter_output><empty_output><jupyter_text>There are ten tokens with the TF-IDF values in the dataset that are above 0.37.  The highest, or most unique, is "decision" at 0.58 although six have the same `max` value of 0.481193.  Plot TF-IDF valuesUse t-SNE (t-Stochastic Neighbors Embedding) from sci-kit learn to graph the values.<jupyter_code>#list of lists of values from the list of tf-idf dictionaries
tfidfvalues = []
for tfidfdict in tfidflist:
    tfidfvalues.append(list(tfidfdict.values())) #convert values of each to list
def graphdims(transformed, dims=3):
    '''
    Transpose a list of lists or similar iterable from one N-length list of dims-length lists -> one list of dims 
    number of N-length lists.  It's like switching a matrix coordinate from (i, j) to be (j, i) except using a list of lists
    
    Args
        transformed (list or arraylike) : an arraylike of an arraylike of dimension dims x N (dims lists each of size N)
        dims (int) : defaults to 3. The desired dimension of the returned list of lists
    
    Returns
        coordlist (list) : a list of lists of dimension N x dims (N lists each of size dims)
    '''
    coordlist = [[] for i in range(0,dims)] #list of lists
    for coord in transformed:
        for j in range(0,dims):
            coordlist[j].append(coord[j])
    return coordlist
#reduce to two dimensions to graph on x-y axes
dim2TSNE = TSNE(n_components=2) #using sklearn
dim2data = dim2TSNE.fit_transform(tfidfvalues)
dim2abstr = graphdims(dim2data,2) #kind of transpose list of lists
#check graphdims changed shape as expected
print(dim2data.shape, len(dim2abstr), len(dim2abstr[0]))
#add coordinates as columns to finaldf
finaldf["xcoord"] = dim2abstr[0]
finaldf["ycoord"] = dim2abstr[1]
finaldf.describe()
#save with coordinates
finaldf.to_csv("csvdata/abs76coords.csv")
finaldf = pd.read_csv("csvdata/abs76coords.csv", index_col=[0])<jupyter_output><empty_output><jupyter_text>Match individual colors to each `authID` for graph.<jupyter_code>authidlist = finaldf.authID
authidset = set(authidlist)
len(authidset)
#create list of 26 color names to be for each authID
colors = ["darkgrey","dimgray","brown","red","tomato","sienna",
          "chocolate","darkorange","tan","gold","darkkhaki","y",
         "yellowgreen","green","c","royalblue","slateblue","navy",
         "blue","mediumpurple","violet","fuchsia","deeppink",
         "lawngreen","cyan","darkgoldenrod"]
len(colors)==len(authidset)
#create dictionary for color names
colordict = dict()
for authid,color in zip(authidset,colors):
    colordict[authid]=color
#graph with the authID's for the corresponding abstracts
fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(111)
for xcoord, ycoord, authid in zip(dim2abstr[0], dim2abstr[1], authidlist):
    colr = colordict[authid]
    ax.scatter(xcoord, ycoord, c=colr, label=authid)

pts = 18 #font pt size

plt.rc('axes', titlesize=pts, labelsize=pts)     # font size of the axes titles and labels
plt.rc('xtick', labelsize=pts-2)    # font size of the tick labels
plt.rc('ytick', labelsize=pts-2)    # font size of the tick labels
plt.rc('figure', titlesize=30) #title font size, slightly larger than the other text

plt.title('Abstract TF-IDF Color-coded by Author')
#plt.savefig("images/dots-by-author.png")

#ax.legend(ncol=8) #with the legend, it was hard to see which was a datapoint and which might be in the key
plt.show()<jupyter_output><empty_output><jupyter_text>Match four colors for Nobel Prize field to graph and look for clusters.<jupyter_code>fourcolors = ["darkgreen","blue","goldenrod","magenta"]
fieldset = ["physics","chemistry","economics","medicine"]
#sort by area
areasort = finaldf.sort_values(by=["area"])
phys = areasort[areasort["area"]=="physics"]
chem = areasort[areasort["area"]=="chemistry"]
econ = areasort[areasort["area"]=="economics"]
medi = areasort[areasort["area"]=="medicine"]
areadflist = [phys,chem,econ,medi]
fig = plt.figure(figsize=(16,8))
ax = fig.add_subplot(111)
for areadf, colr, field in zip(areadflist,fourcolors,fieldset):
    ax.scatter(areadf.xcoord,areadf.ycoord,color=colr,label=field)
ax.legend()

pts = 18 #font pt size

plt.rc('axes', titlesize=pts, labelsize=pts)     # font size of the axes titles and labels
plt.rc('xtick', labelsize=pts-2)    # font size of the tick labels
plt.rc('ytick', labelsize=pts-2)    # font size of the tick labels
plt.rc('figure', titlesize=30) #title font size, slightly larger than the other text

plt.title('Abstract TF-IDF Color-coded by Field')
#plt.savefig("images/dots-by-field.png")
plt.show()<jupyter_output><empty_output><jupyter_text>The most noticeable group is about 10 Physics publications near center, at about coordinate pair (50, 100).<jupyter_code>#see how coauthor column compares
winners = finaldf[finaldf["coauthor"]==0]
coauths = finaldf[finaldf["coauthor"]==1]

coautcatlist=[winners,coauths]
labellist = ["Winners","Other Authors"]

twocolors = ["blue","goldenrod"]

fig = plt.figure(figsize=(16,8))
ax = fig.add_subplot(111)
for df, colr, each in zip(coautcatlist,twocolors,labellist):
    ax.scatter(df.xcoord,df.ycoord,color=colr,label=each)
ax.legend()

pts = 18 #font pt size

plt.rc('axes', titlesize=pts, labelsize=pts)     # font size of the axes titles and labels
plt.rc('xtick', labelsize=pts-2)    # font size of the tick labels
plt.rc('ytick', labelsize=pts-2)    # font size of the tick labels
plt.rc('figure', titlesize=30) #title font size, slightly larger than the other text

plt.title('Abstract TF-IDF Color-coded by <NAME>')
#plt.savefig("images/dots-by-winner.png")
plt.show()<jupyter_output><empty_output><jupyter_text>There appears to be a few clusters, one in particular is made up of non-winners' publications text near the left center of the plot around coordinates (-100, 100).  In other graphs, the dots are different colors in this cluster, indicating that they are not by the same author, nor are they in the same field.  ModelUsing keras to determine a model of 7 clusters using TF-IDF values as X, predict for each y: `area` and `coauthor`.<jupyter_code>#get dummies to create two targets: coauthor and field
ycoa = pd.get_dummies(finaldf["coauthor"]).values
yarea = pd.get_dummies(finaldf["area"]).values
tfidframe.shape
X = [row for row in tfidframe.values] #list of rows (numpy arrays)
X = np.array(X) #Sequential prefers numpy arrays to lists
type(X[0][0])<jupyter_output><empty_output><jupyter_text>Model for `coauthor` as Target<jupyter_code>#train test split
X_train, X_test, yc_train, yc_test = train_test_split(X, ycoa, test_size=0.33, random_state=321)
modelc = Sequential()
embedsize = 32 #there are 84 records
numwords = tfidframe.shape[1] #and 2500 tokens
modelc.add(Embedding(numwords, embedsize))
modelc.add(LSTM(10, return_sequences=True))
modelc.add(GlobalMaxPool1D())
modelc.add(Dropout(0.5))
modelc.add(Dense(20, activation='relu'))
modelc.add(Dropout(0.5))
modelc.add(Dense(2, activation='softmax')) #coauthor has two possibilities
modelc.compile(loss="categorical_crossentropy",optimizer="SGD",metrics=["accuracy"])
modelc.fit(X_train, yc_train,epochs=10,batch_size=15)<jupyter_output>Epoch 1/10
4/4 [==============================] - 9s 2s/step - loss: 0.6983 - accuracy: 0.3800
Epoch 2/10
4/4 [==============================] - 9s 2s/step - loss: 0.6947 - accuracy: 0.4800
Epoch 3/10
4/4 [==============================] - 9s 2s/step - loss: 0.6926 - accuracy: 0.5400
Epoch 4/10
4/4 [==============================] - 9s 2s/step - loss: 0.6981 - accuracy: 0.4000
Epoch 5/10
4/4 [==============================] - 9s 2s/step - loss: 0.6917 - accuracy: 0.4400
Epoch 6/10
4/4 [==============================] - 9s 2s/step - loss: 0.6981 - accuracy: 0.3800
Epoch 7/10
4/4 [==============================] - 9s 2s/step - loss: 0.6926 - accuracy: 0.5800
Epoch 8/10
4/4 [==============================] - 9s 2s/step - loss: 0.6930 - accuracy: 0.5400
Epoch 9/10
4/4 [==============================] - 9s 2s/step - loss: 0.6925 - accuracy: 0.5000
Epoch 10/10
4/4 [==============================] - 9s 2s/step - loss: 0.6944 - accuracy: 0.5200<jupyter_text>The loss seems large, at .7, but the accuracy improved over the epochs from .38 to .52.<jupyter_code>yc_preds = modelc.predict(X_test) #I can compare with .evaluate more conveniently
traincresults = modelc.evaluate(X_train, yc_train)
testcresults = modelc.evaluate(X_test, yc_test)<jupyter_output>2/2 [==============================] - 0s 215ms/step - loss: 0.6932 - accuracy: 0.5000
1/1 [==============================] - 0s 0s/step - loss: 0.6919 - accuracy: 0.5769<jupyter_text>The accuracy on the test set is about 57%.  The loss is lower than 1, but still rather high and improved minimally from the first epoch.   Model for `area` as the target<jupyter_code>#train test split
X_train, X_test, ya_train, ya_test = train_test_split(X, yarea, test_size=0.33, random_state=321)
modela = Sequential()
embedsize = 32 #there are 84 records
numwords = tfidframe.shape[1] #and 2500 tokens
modela.add(Embedding(numwords, embedsize))
modela.add(LSTM(10, return_sequences=True))
modela.add(GlobalMaxPool1D())
modela.add(Dropout(0.5))
modela.add(Dense(20, activation='relu'))
modela.add(Dropout(0.5))
modela.add(Dense(4, activation='softmax')) #area has 4 possibilities
modela.compile(loss="categorical_crossentropy",optimizer="SGD",metrics=["accuracy"])
modela.fit(X_train, ya_train,epochs=10,batch_size=15)<jupyter_output>Epoch 1/10
4/4 [==============================] - 9s 2s/step - loss: 1.3882 - accuracy: 0.2400
Epoch 2/10
4/4 [==============================] - 9s 2s/step - loss: 1.3825 - accuracy: 0.3200
Epoch 3/10
4/4 [==============================] - 9s 2s/step - loss: 1.3885 - accuracy: 0.2000
Epoch 4/10
4/4 [==============================] - 9s 2s/step - loss: 1.3874 - accuracy: 0.3400
Epoch 5/10
4/4 [==============================] - 9s 2s/step - loss: 1.3826 - accuracy: 0.3200
Epoch 6/10
4/4 [==============================] - 9s 2s/step - loss: 1.3829 - accuracy: 0.3400
Epoch 7/10
4/4 [==============================] - 9s 2s/step - loss: 1.3876 - accuracy: 0.2600
Epoch 8/10
4/4 [==============================] - 9s 2s/step - loss: 1.3915 - accuracy: 0.1800
Epoch 9/10
4/4 [==============================] - 9s 2s/step - loss: 1.3838 - accuracy: 0.3200
Epoch 10/10
4/4 [==============================] - 9s 2s/step - loss: 1.3837 - accuracy: 0.2600<jupyter_text>The accuracy improved a little over the epochs, but perhaps it needs more of them to converge.<jupyter_code>ya_preds = modela.predict(X_test)
trainaresults = modela.evaluate(X_train, ya_train)
testaresults = modela.evaluate(X_test, ya_test)<jupyter_output>2/2 [==============================] - 0s 213ms/step - loss: 1.3824 - accuracy: 0.3000
1/1 [==============================] - 0s 0s/step - loss: 1.3945 - accuracy: 0.1923<jupyter_text>The accuracy is 19% for the test data and 30% for the training data, both with loss values near 1.4.  T-test for significance for `coauthor` feature I'd like to see whether there is a significant difference between TF-IDF values for abstract text by a Nobel winner (0) or a coauthor (1).  As a hypothesis test:$H_0$ : The mean difference between the TF-IDF values for non-coauthors and coauthors is zero. Rephrased, $\mu_{nobel} = \mu_{coauthor}$$H_1$ : The mean difference between the TF-IDF values for non-coauthors and coauthors is nonzero. Rephrased, $\mu_{nobel} \ne \mu_{coauthor}$To prove the alternative hypothesis, I can use a two-tailed less-than test.With the $p$ and $t$ values from a two-tailed test (returned by `ttest_ind`), I can reject the null hypothesis when $p  0$.<jupyter_code>finaldf["tfidf_avg"] = [np.mean(i) for i in tfidframe.values] #average each row
#create lists of average TF-IDF values for whether coauthor/Nobel winner
coauth1tfidf = [tival for tival in finaldf[finaldf["coauthor"]==1].tfidf_avg]
coauth0tfidf = [tival for tival in finaldf[finaldf["coauthor"]==0].tfidf_avg]
ttestresults = stats.ttest_ind(coauth1tfidf,coauth0tfidf)
ttestresults<jupyter_output><empty_output><jupyter_text>That appears to not be significant; a publication authored by a Nobel prize-winner and that which was authored by a collaborator do not have abstracts that are significantly different.  T-test for significance for `area` feature I'd like to see whether there is a significant difference between TF-IDF values for abstract text by one field or another.  To prove the alternative hypothesis, these would be two-tailed, paired tests comparing the four possibilities for `area` (each pair would be field A compared to field B).$H_0$ : The mean difference between the TF-IDF values for field A and field B is zero. Rephrased, $\mu_{A} = \mu_{B}$$H_1$ : The mean difference between the TF-IDF values for field A and field B is nonzero. Rephrased, $\mu_{A} \ne \mu_{B}$With the $p$ and $t$ values from a two-tailed test (returned by `ttest_ind`), I can reject the null hypothesis when $p  0$.<jupyter_code>phystfidf = [tival for tival in finaldf[finaldf["area"]=="physics"].tfidf_avg]
chemtfidf = [tival for tival in finaldf[finaldf["area"]=="chemistry"].tfidf_avg]
econtfidf = [tival for tival in finaldf[finaldf["area"]=="economics"].tfidf_avg]
meditfidf = [tival for tival in finaldf[finaldf["area"]=="medicine"].tfidf_avg]
#pairwise Tukey test:
#information needs to be in numpy arrays
area_arrays = np.concatenate([np.array(phystfidf),np.array(chemtfidf),np.array(econtfidf),np.array(meditfidf)])
#and labels are important for it to handle the data properly
areanames = ["physics"]*len(phystfidf) + ["chemistry"]*len(chemtfidf) + ["economics"]*len(econtfidf) + ["medicine"]*len(meditfidf)
#get tukeyhsd test results with 0.05 for significant p-value
print(pairwise_tukeyhsd(area_arrays,areanames,0.05))<jupyter_output>Multiple Comparison of Means - Tukey HSD,FWER=0.05
==================================================
  group1    group2  meandiff  lower  upper  reject
--------------------------------------------------
chemistry economics   0.0      -0.0  0.0001 False 
chemistry  medicine   0.0      -0.0  0.0001 False 
chemistry  physics  -0.0001  -0.0001  -0.0   True 
economics  medicine   0.0    -0.0001 0.0001 False 
economics  physics  -0.0001  -0.0002  -0.0   True 
 medicine  physics  -0.0001  -0.0002  -0.0   True 
--------------------------------------------------<jupyter_text>By field, three combinations' null hypotheses can be rejected: chemistry-economics, chemistry-medicine, and economics-medicine.  Conversely, it means pairs with physics are not significantly different.  However, all values for `meandiff`, `lower` and `upper` are very close to 0.  InterpretUltimately, the results rely on too little data. The model has a relatively high loss values and accuracy metrics that barely improve over the last few epochs of the Sequential Model from `keras`.  The t-test comparing TF-IDF values for co-authors and those for Nobel Prize winners did not find significance.  Pairwise tests comparing four different fields (`area`) found significance for half of the pairs (those combinations of chemistry, medicine and economics).Comparing the TF-IDF for abstract text gives closeness based on the words' importances to the abstract.  With many abstracts containing specialized jargon from different disciplines, it was surprising that when graphing the values, the abstracts were not more clustered by `area`.  Comparison to `sklearn`'s `TfidfTransformer` with `CountVectorizer`Using the tokenized abstracts and a `set` of the words from all the abstracts, get TF-IDF values from `sklearn` methods.<jupyter_code>#take tokenslists and put each list into one String, each word separated by spaces
corpus = []
vocab = [] #vocab will be set of all words from all tokens
for tokenlist in tokenslists:
    tokenstring=""
    lasttoken = tokenlist[-1]
    for token in tokenlist[:-1]:
        vocab.append(token)
        tokenstring+=token
        tokenstring+=" " #last token does not need space after it
    tokenstring+=lasttoken #each separated by spaces
    vocab.append(lasttoken)
    corpus.append(tokenstring)
len(corpus)
vocabul = set(vocab)
len(vocabul)
tokenslists[10][-1]
corpus[10]
#without a pipeline
vectorizer = CountVectorizer()
vectorized = vectorizer.fit_transform(corpus)
vectorized[10]
transformer = TfidfTransformer()
tfidf = transformer.fit_transform(vectorized)
tfidf
vectarr = vectorized.toarray()
tfidfarr = tfidf.toarray()
vectdf = pd.DataFrame(vectarr, columns=vectorizer.get_feature_names())
vectdf.head()
tfidfdf = pd.DataFrame(tfidfarr, columns=vectorizer.get_feature_names())
tfidfdf.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11025906212950584, 'fraction_numerical': 0.025401710264107157, 'mean_word_length': 4.429941103958362, 'pattern_counts': {'":': 1, '<': 121, '<?xml version=': 0, '>': 123, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '396858', 'license': 'RSA-MD', 'n_tokens_mistral': 13005, 'n_tokens_neox': 12066, 'n_words': 4241, 'path': 'analysis.ipynb', 'repo_name': 'bronwencc/Capstone', 'size': '39690'}