<jupyter_start><jupyter_text>Create an edgelist of identified acknowledged scholar from citation approachWe wiil see that an acknowledged individual name is registered in MAG and on the list of references in the acknowledged papers.<jupyter_code>import ast
import pandas as pd
import vaex
import os.path
from ast import literal_eval
from pathlib import Path

FILES = ['compbiology', 'biology', 'medicine', 'genetics', 'ntds', 'pathogenes', 'plosone', 'srep']

# Path to data
PATH_ROW_ACKNOW = '../../data/TreatedEdgeList21'
PATH_MAG = '../../data-MAG/'

# Path to data to save
PATH_SAVE = '../../data-computed-MAG'
Path(PATH_SAVE).mkdir(parents=True, exist_ok=True) # create save dir if not exists
def see(var):
    """funtion to check the contents of variable (dict or set)"""
    print(type(var))
    if type(var) == dict:
        for i, (k, v) in enumerate(var.items()):
            print(k, ":", v)
            if i == 4:
                break
    if type(var) == set:
        print(list(var)[:5])
        
        
def read_vaex(file_path):
    """
    It may take times to read files for the first time.
    From second time, the loading time will be shorter.
    """
    if os.path.exists(file_path):
        print('file exists')
        return vaex.open(file_path+'.hdf5')
    return vaex.from_csv(file_path, convert=True, chunk_size=10_000_000)<jupyter_output><empty_output><jupyter_text>Read Acknow data (PLOS and SciRep)<jupyter_code>%%time
# concat all dfs
dfs = {}
dfs_all = pd.DataFrame()
for file in FILES:
#     print(f"read file: {file}")
    dfs[file] = pd.read_csv(f'{PATH_ROW_ACKNOW}/{file}.csv')[['paperId', 'author', 'acknow']]
    dfs_all = pd.concat([dfs_all, dfs[file]])
dfs_all.head()<jupyter_output>/Users/keigokusumegi/.anyenv/envs/pyenv/versions/miniforge3-4.10/envs/network-analysis/lib/python3.9/site-packages/IPython/core/magics/execution.py:1321: DtypeWarning: Columns (1) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code, glob, local_ns)<jupyter_text>Read MAG data (Paper & PaperAuthorAffil) and put pid from MAG  read MAG Paper with the doi of collected data<jupyter_code># read paper id data
df_magpaper_id_doi_plos = pd.read_table(f'{PATH_MAG}/papers_plos.txt', sep=' ', names=['pid', 'doi'])
df_magpaper_id_doi_srep = pd.read_table(f'{PATH_MAG}/papers_srep.txt', sep=' ', names=['pid', 'doi'])
df_magpaper_id_doi = pd.concat([df_magpaper_id_doi_plos, df_magpaper_id_doi_srep])

# change doi to lower case
df_magpaper_id_doi["doi"] = df_magpaper_id_doi["doi"].str.lower()

# clear error row: Eliminate cases where multiple paper ids are assigned to a single doi.
s_vs = df_magpaper_id_doi['doi'].value_counts()
mag_available_dois = set(s_vs[s_vs.values == 1].index)
df_magpaper_id_doi2 = df_magpaper_id_doi[df_magpaper_id_doi['doi'].isin(mag_available_dois)]

df_magpaper_id_doi2.head()<jupyter_output><empty_output><jupyter_text>Rate of DOI mathcing with MAG<jupyter_code>row_dois = set(dfs_all.paperId.tolist())
mag_dois = set(df_magpaper_id_doi2.doi.tolist())

common_dois = row_dois & mag_dois
print('Intersected dois = ', len(common_dois)/len(row_dois))<jupyter_output>Intersected dois =  0.8262019771987638<jupyter_text>Collect author information of MAG via doi<jupyter_code>df_magpaper_commondois = df_magpaper_id_doi2[df_magpaper_id_doi2['doi'].isin(common_dois)]
common_authors_pid = df_magpaper_commondois.pid.tolist()<jupyter_output><empty_output><jupyter_text>Extract necessary data with PaperReferences and PaperAuthorAffiliations of MAG<jupyter_code># read PaperAuthorAffiliations.csv
vaex_mag_pauthorAffil = read_vaex(f'{PATH_MAG}/PaperAuthorAffiliations/PaperAuthorAffiliations.txt')

# read Preferences.csv
vaex_mag_prefs = read_vaex(f'{PATH_MAG}/PaperReferences/PaperReferences.csv')

# extract necessary row with ```common_authors_pid```
paper_references_extracted = vaex_mag_prefs[vaex_mag_prefs.pid.isin(common_authors_pid)]
rfpids = paper_references_extracted[' rfpid'].tolist()

# extract necessary row with ```rfpids```
vaex_mag_pauthorAffil_extracted = vaex_mag_pauthorAffil[vaex_mag_pauthorAffil.PaperId.isin(rfpids)]<jupyter_output>file exists
file exists<jupyter_text>Convert extracted data of `PaperReferences` & `PaperAuthorAffi.` into DataFrameUsing `PaperReferences = {paperId: referencedPaperId}` and `PaperAuthorAffi = {paperId: authorIds}`,create `{paperId: [authorIds]}`<jupyter_code># dict of paper:　referenced paper id
df_paper_references_extracted = paper_references_extracted.to_pandas_df()
# dict of paper: author id
df_mag_pauthorAffil_extracted = vaex_mag_pauthorAffil_extracted.to_pandas_df()
%%time
# groupby to a dict of {paperid: [authorIds]}
df_mag_pauthorAffil_extracted_grouped = df_mag_pauthorAffil_extracted.groupby('PaperId',)[' AuthorId'].apply(list)
df_mag_pauthorAffil_extracted_grouped = df_mag_pauthorAffil_extracted_grouped.reset_index()<jupyter_output>CPU times: user 31 s, sys: 801 ms, total: 31.8 s
Wall time: 32.1 s<jupyter_text>merge with `df_paper_references_extracted`<jupyter_code>%%time
df_merged = pd.merge(df_paper_references_extracted, df_mag_pauthorAffil_extracted_grouped,  
          how='inner',left_on =' rfpid',right_on='PaperId')
df_pid_authors = df_merged[['pid', ' AuthorId']].rename({' AuthorId': 'ReferencedAuthorIds'}, axis=1)
%%time
df_merged = pd.merge(df_paper_references_extracted, df_mag_pauthorAffil_extracted_grouped,  
          how='inner',left_on =' rfpid',right_on='PaperId')
df_pid_authors = df_merged[['pid', ' AuthorId']].rename({' AuthorId': 'ReferencedAuthorIds'}, axis=1)
df_pid_authors.head()<jupyter_output>CPU times: user 2.1 s, sys: 253 ms, total: 2.35 s
Wall time: 2.35 s<jupyter_text>save authors informations as csv<jupyter_code># # save as authors informations
# df_pid_authors.to_csv(f'{PATH_SAVE}/pid_authors.csv', index=False)

df_pid_authors = pd.read_csv(f'{PATH_SAVE}/pid_authors.csv', converters={'ReferencedAuthorIds': literal_eval})
%%time
# format as DataFrame
# df_pid_authors['ReferencedAuthorIds'] = df_pid_authors['ReferencedAuthorIds'].apply(literal_eval)
df_pid_author = df_pid_authors.explode('ReferencedAuthorIds')
l_ReferencedAuthorIds = df_pid_author['ReferencedAuthorIds'].tolist()
df_pid_author.head()<jupyter_output>CPU times: user 5.96 s, sys: 1.14 s, total: 7.1 s
Wall time: 7.3 s<jupyter_text>Check acknoweldged names are included in ReferencedAuthorIdsWith `Authors.csv`，convert `{paperId: authorIds}`into `{paperId: authorNames}`  Read Authors.csv<jupyter_code># read Authors.csv
vaex_mag_authors = read_vaex(f'{PATH_MAG}/Authors/Authors.csv')
# extract necessary rows and change to pandas
vaex_mag_authors_extracted = vaex_mag_authors[vaex_mag_authors.AuthorId.isin(l_ReferencedAuthorIds)]
# vaex_mag_authors_extracted.head()
df_mag_authors_extracted = vaex_mag_authors_extracted[['AuthorId', 'DisplayName']].to_pandas_df()<jupyter_output><empty_output><jupyter_text>Convert `{paperId: authorIds}` into `{paperId: authorNames}`<jupyter_code>%%time
df_merged_auth_refauth = pd.merge(df_pid_author, df_mag_authors_extracted,  
          how='inner',left_on ='ReferencedAuthorIds',right_on='AuthorId')
df_merged_auth_refauth.head()
%%time
# groupby to paperid: [authorIds]
df_pid_refAuthIds = df_merged_auth_refauth[['pid', 'DisplayName']].groupby('pid')['DisplayName'].apply(set)

df_pid_refAuthIds = df_pid_refAuthIds.reset_index()<jupyter_output>CPU times: user 21.9 s, sys: 3.64 s, total: 25.5 s
Wall time: 28.2 s<jupyter_text>save df_pid_refAuthIds informations as csv<jupyter_code># # save as authors informations
# df_pid_refAuthIds.to_csv(f'{PATH_SAVE}/pid_refrencedAuthorIds.csv', index=False)

# df_pid_refAuthIds = pd.read_csv(f'{PATH_SAVE}/pid_refrencedAuthorIds.csv')<jupyter_output><empty_output><jupyter_text>check acknowledged individual paper are cited in the paper<jupyter_code>%%time
df_pid_refAuthIds = pd.read_csv(f'{PATH_SAVE}/pid_refrencedAuthorIds.csv', converters={'DisplayName': literal_eval})
df_pid_refAuthIds.head()
df_magpaper_id_doi_w_regAuthIds = pd.merge(df_magpaper_id_doi, df_pid_refAuthIds,  
          how='inner', on='pid')
df_magpaper_id_doi_w_regAuthIds.head()
%%time
dfs_all_w_pid = pd.merge(dfs_all, df_magpaper_id_doi_w_regAuthIds,  
          how='inner',left_on ='paperId',right_on='doi')
# change acknow's name
dfs_all_w_pid['acknow'] = dfs_all_w_pid['acknow'].apply(lambda x: " ".join(x.split('_')))
dfs_all_w_pid[['pid', 'acknow', 'DisplayName']].head()
# 入ってるかどうかをバイナリで判断
dfs_all_w_pid['cited_as_same_name'] = dfs_all_w_pid[['acknow', 'DisplayName']].apply(lambda row: True if (row['acknow'] in row['DisplayName']) else False, axis=1)
# acknowの名前がはいってるrowだけを抽出
a = dfs_all_w_pid[dfs_all_w_pid['cited_as_same_name']][['acknow', 'pid', 'DisplayName']]
df_referenced_acknow = a.drop_duplicates(subset=['acknow', 'pid'])
df_referenced_acknow.head()<jupyter_output><empty_output><jupyter_text>Identify the acknowledged scholarID  make sure a name with different person are listed in the referencesIf the case exists, we don't consider it. That is, we only consider the matched acknowledged scholar ID which appear in the references list once.<jupyter_code>df_merged_auth_refauth_tmp = df_merged_auth_refauth[df_merged_auth_refauth.pid.isin(set(df_referenced_acknow['pid'].tolist()))]
df_merged_auth_refauth_tmp.head()
# Identify
df_ref_author_ids = df_merged_auth_refauth_tmp.groupby(['pid', 'DisplayName'])['AuthorId']
df_ref_author_ids_grouped = df_ref_author_ids.nunique().reset_index()

# Those who have unique ID
df_merged_auth_refauth_tmp2 = df_ref_author_ids_grouped[df_ref_author_ids_grouped['AuthorId'] == 1]
df_merged_auth_refauth_tmp2.head()
df_merged_auth_refauth_tmp3 = pd.merge(df_merged_auth_refauth_tmp, df_merged_auth_refauth_tmp2,  
          how='inner', on =['pid', 'DisplayName'])[['pid', 'ReferencedAuthorIds', 'DisplayName']]
df_merged_auth_refauth_tmp3.head()<jupyter_output><empty_output><jupyter_text>Finally identify the ID<jupyter_code>df_referenced_acknow.head()
%%time
df_result1 = pd.merge(df_referenced_acknow, df_merged_auth_refauth_tmp3,  
          how='inner',left_on =['pid', 'acknow'],right_on=['pid', 'DisplayName'])
df_result1.head()<jupyter_output><empty_output><jupyter_text>Save<jupyter_code>df_result1[['pid', 'acknow', 'ReferencedAuthorIds']].to_csv('../../data-computed-MAG/references_result.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12469768791719067, 'fraction_numerical': 0.01006094611589436, 'mean_word_length': 7.515650741350906, 'pattern_counts': {'":': 1, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '617325', 'license': 'MIT', 'n_tokens_mistral': 4003, 'n_tokens_neox': 3798, 'n_words': 663, 'path': 'scripts/main/5-MAG_ReferenceIndentify.ipynb', 'repo_name': 'Keygoksmg/acknow-identification', 'size': '10337'}