<jupyter_start><jupyter_text>Download databases from WoS and Scopus<jupyter_code>df_WOS = pd.read_csv("./intermed/df_WOS.csv")
print( "Loaded %d X %d  WOS dataframe" % (len(df_WOS), len(df_WOS.columns) ))
df_Scopus = pd.read_csv("./intermed/df_Scopus.csv")
print( "Loaded %d X %d  Scopus dataframe" % (len(df_Scopus), len(df_Scopus.columns) ))<jupyter_output>Loaded 3800 X 10  WOS dataframe
Loaded 4181 X 9  Scopus dataframe<jupyter_text>Concatenate the databases<jupyter_code>df = pd.concat([df_Scopus, df_WOS], sort = False)<jupyter_output><empty_output><jupyter_text>Remove duplicates based on the DOIWrite DOI in lower cases<jupyter_code>df['low_doi'] = df['doi'].str.lower()
df_wo_doi = df[df.doi.isnull()]
print('There are %d papers without doi' % len(df_wo_doi))
df_with_doi = df[df.doi.notnull()]
df_without_dupli_doi = df_with_doi.groupby(['low_doi']).first().reset_index()
print('There are %d different papers with doi' % len(df_without_dupli_doi))
df = pd.concat([df_wo_doi, df_without_dupli_doi], sort = False)
print('There are %d papers after removing duplicates based on doi' % len(df))<jupyter_output>There are 4944 papers after removing duplicates based on doi<jupyter_text>Remove duplicates based on the titleWe transform the title by * using lower characters* deleting words in [ ] if it isn't the case for all the title* (i) removing punctuation and replacing it be nothing* (ii) replacing punctuation by one white space & then replacing double white spaces by one white space<jupyter_code>string.punctuation =  string.punctuation + '’—☆–−()()©“”‘'
string.punctuation<jupyter_output><empty_output><jupyter_text>First, we remove all punctuation and group duplicates by title without punctuationExample of duplicates identified : * The effect of Estonian electricity production scenarios on CO(2) and SO(2) emissions in 2000-2030* The effect of Estonian electricity production scenarios on CO2 and SO2 emissions in 2000-2030<jupyter_code>df['title_red'] = pd.Series([re.sub("[\[\]].*[\[\]]", "", title) for title in df.title.values],index = df.index)
df['title_red'] = np.where(df['title_red'].str.len()<=1,df['title'],df['title_red'])
table_remove_punct = str.maketrans('','', string.punctuation)
titles = df['title_red'].str.lower().tolist()
titles_wo_punct = [title.translate(table_remove_punct) for title in titles]
df['prepro_title_wo_punct'] = pd.Series(titles_wo_punct,index = df.index)<jupyter_output><empty_output><jupyter_text>We order the dataframe by 'prepro_title_wo_punct' and doi in order to keep papers with doi when there are duplicates (ex : published paper and conference paper)<jupyter_code>df.sort_values(by = ['prepro_title_wo_punct','doi'], ascending = [True,True], inplace = True)
df = df.groupby(['prepro_title_wo_punct']).first().reset_index()
print('There are %d papers after removing duplicates based on low title without punctuation' % len(df))<jupyter_output>There are 4769 papers after removing duplicates based on low title without punctuation<jupyter_text>Then we remove punctuation and replace it by on white space and then replace double white by one white spaceExample of duplicates identified : * Future Japan Power Generation Sector by Introducing Hydrogen Plant with 80% CO2 Emission Reduction Target : A Preliminary Analysis* Future Japan power generation sector by introducing hydrogen plant with 80% CO2 emission reduction target: A preliminary analysis<jupyter_code>titles = df['title_red'].str.lower().tolist()
table_remove_punct_white_space = str.maketrans(string.punctuation, ' '*len(string.punctuation))
titles_wo_punct_white_space = [title.translate(table_remove_punct_white_space) for title in titles]
titles_wo_punct_one_white_space = [' '.join(title.split()) for title in titles_wo_punct_white_space]
df['prepro_title'] = pd.Series(titles_wo_punct_one_white_space,index = df.index)
df.sort_values(by = ['prepro_title','doi'], ascending = [True,True], inplace = True)
df = df.groupby(['prepro_title']).first().reset_index()
print('There are %d papers after removing duplicates based on low title with punctuation replaced by white space' % len(df))<jupyter_output>There are 4755 papers after removing duplicates based on low title with punctuation replaced by white space<jupyter_text>Keep papers with abstract<jupyter_code>df = df.loc[df.abstract.notnull()]
print('There are %d papers with abstract' % len(df))<jupyter_output>There are 4743 papers with abstract<jupyter_text>Remove duplicates based on the abstract<jupyter_code>df.sort_values(by = ['abstract','doi'], ascending = [True,True], inplace = True)
df = df.groupby(['abstract']).first().reset_index()
print('There are %d papers after removing duplicates based on abstract' % (len(df)))<jupyter_output>There are 4737 papers after removing duplicates based on abstract<jupyter_text>Remove duplicates based on the beginning of the abstract<jupyter_code>abstracts = df['abstract'].str.lower().tolist()
abstracts_wo_punct = [abstract.translate(table_remove_punct_white_space) for abstract in abstracts]
abstracts_wo_punct_one_white_space = [' '.join(abstract.split()) for abstract in abstracts_wo_punct]
df['prepro_abs'] = pd.Series(abstracts_wo_punct_one_white_space, index = df.index)
df['beg_abs'] = df['prepro_abs'].str[:250]
df.sort_values(by = ['beg_abs','doi'], ascending = [True,True], inplace = True)
df = df.groupby(['beg_abs']).first().reset_index()

print('There are %d papers after removing duplicates based on first 250 characters of abstract' % (len(df)))<jupyter_output>There are 4723 papers after removing duplicates based on first 250 characters of abstract<jupyter_text>Look more closely at papers having the same first 200 characters<jupyter_code>df['beg_abs_200'] = df['prepro_abs'].str[:200]
df_deb_first = df.loc[df.beg_abs_200.duplicated(keep='first')]
df_deb_first[['title','doi','publication_year','docu_type','doc_type','scopus_number','WOS_number']]
df_deb_last = df.loc[df.beg_abs_200.duplicated(keep='last')]
df_deb_last[['title','doi','publication_year','docu_type','doc_type','scopus_number','WOS_number']]<jupyter_output><empty_output><jupyter_text>* 2194 and 2193 have the same abstracts but relate respectively to China and India so we keep both. * We merge 3445 and 3444* 3468 is the conference paper for 3467 so we keep the published version* 2294 is the preversion of 2293, so we keep the last version<jupyter_code>df[3445,] = df.loc[3445,].combine_first(df.loc[3444,])
df = df.drop([3444,2294,3468])<jupyter_output><empty_output><jupyter_text>Sort the dataframe<jupyter_code>df.sort_values(by = ['publication_year','title'], ascending = [False,True], inplace = True)
df['d_type'] = df.docu_type.combine_first(df.doc_type)
col = ['title','authors','source','doi','d_type','abstract','publication_year','horizon_year','author_keywords','scopus_number','WOS_number','prepro_title']
df = df.reindex(columns=col)
print( "%d X %d dataframe" % (len(df), len(df.columns) ))<jupyter_output>4720 X 12 dataframe<jupyter_text>Store the dataframe<jupyter_code>df.to_csv("./output/df_for_analysis.csv", index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12473572938689217, 'fraction_numerical': 0.020859760394644114, 'mean_word_length': 7.814906832298137, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '130179', 'license': 'MIT', 'n_tokens_mistral': 2516, 'n_tokens_neox': 2305, 'n_words': 600, 'path': '1_Constitution_database/3_Fusion_Scopus_WOS.ipynb', 'repo_name': 'ClaireLepault/Mapping-country-mitig-pathways', 'size': '7095'}