<jupyter_start><jupyter_text>Download the database with one row for each publication<jupyter_code>df = pd.read_csv("../2_Treatment_database/output/database_one_row_each_paper.csv")
print( "Loaded %d X %d dataframe with unique papers" % (len(df), len(df.columns) ))<jupyter_output>Loaded 4691 X 16 dataframe with unique papers<jupyter_text>Preprocess the abstracts Transform abstracts in lower cases<jupyter_code>abstracts = df['abstract'].str.lower().to_list()<jupyter_output><empty_output><jupyter_text>Remove punctuation and replace it by on white space and then replace double white by one white space<jupyter_code>string.punctuation =  string.punctuation + '’—☆–−()()©“”‘'
table_remove_punct_white_space = str.maketrans(string.punctuation, ' '*len(string.punctuation))
abstracts = [abstract.translate(table_remove_punct_white_space) for abstract in abstracts]
abstracts = [' '.join(abstract.split()) for abstract in abstracts]<jupyter_output><empty_output><jupyter_text>Define stopwords * nltk stopwords* countries and demonyms<jupyter_code>nltk_stopwords = list(set(stopwords.words('english')))
countriesPath = "../0_Reference_files/UNSD_database.csv"
countries_df = pd.read_csv(countriesPath)
countries_list = countries_df['Country0'].tolist()
demonym1_list = countries_df['Demonym1'][countries_df['Demonym1'].notnull()].tolist()
demonym2_list = countries_df['Demonym2'][countries_df['Demonym2'].notnull()].tolist()
names_list = countries_list + demonym1_list + demonym2_list
names_list = [str(name).lower().translate(table_remove_punct_white_space) for name in names_list]
names_list = [' '.join(name.split()) for name in names_list]<jupyter_output><empty_output><jupyter_text>Remove names that are not single word<jupyter_code>not_single = [name for name in names_list
                     if (len(name)>3 & len(name.split())>1)]
l = [''] *len(not_single)
trans = {name:delet for name,delet in zip(not_single,l)}
for a, b in trans.items():
    for i in range(len(abstracts)):
        abstracts[i] = abstracts[i].replace(a, b)
custom_stopwords = nltk_stopwords + names_list<jupyter_output><empty_output><jupyter_text>Remove stopwords and group words by stem :<jupyter_code>abstracts = [" ".join([stem(word) for word in abstract.split(" ") 
                       if ((word not in custom_stopwords) & (len(word)>2))]) for abstract in abstracts]<jupyter_output><empty_output><jupyter_text>Remove stem stopwords * synonyms of mitigation* that appeared useless after topic modeling experiences:<jupyter_code>stem_stopwords = ['mitig','carbon','decarbonis','decarbon','co2','ghg','greenhous','gas','emiss','reduc','reduct']
stem_stopwords += ['million','ton','billion','proceed','paper','data','present','result','studi','describ','also','base','explor','analys','analyz','elsevi']
abstracts = [" ".join([stem for stem in abstract.split(" ") if stem not in stem_stopwords]) for abstract in abstracts]<jupyter_output><empty_output><jupyter_text>Remove digits<jupyter_code>abstracts =[" ".join([stem for stem in abstract.split(" ") if not stem.isdigit()]) for abstract in abstracts]<jupyter_output><empty_output><jupyter_text>Save processed abstracts<jupyter_code>abstract_prepro = pd.DataFrame(abstracts, columns = ['abstracts_prepro'])
abstract_prepro.to_csv("./interm/processed_abstracts.csv",index=False)<jupyter_output><empty_output><jupyter_text>Term weighting with TF-IDF<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=0.01,smooth_idf= False)
                                   
tfidf = tfidf_vectorizer.fit_transform(abstracts)
print('Created %d X %d TF-IDF-normalized document-term matrix' % (tfidf.shape[0], tfidf.shape[1]))
# extract the resulting vocabulary
tfidf_feature_names = tfidf_vectorizer.get_feature_names()
print("Vocabulary has %d distinct terms" % len(tfidf_feature_names))<jupyter_output>Vocabulary has 1300 distinct terms<jupyter_text>Save TF-IDF matrix and vocabulary features names under pkl format<jupyter_code>joblib.dump((tfidf, tfidf_feature_names), "./interm/tfidf_matrix-features_names.pkl")<jupyter_output><empty_output><jupyter_text>Look at weightest words<jupyter_code>def rank_terms(tfidf, terms):
    # get the sums over each column
    sums = tfidf.sum(axis=0)
    # map weights to the terms
    weights = {}
    for col, term in enumerate(terms):
        weights[term] = sums[0,col]
    # rank the terms by their weight over all documents
    return sorted(weights.items(), key = operator.itemgetter(1), reverse=True)
ranking = rank_terms(tfidf,tfidf_feature_names)
for i, pair in enumerate(ranking[0:20]):
    print( "%02d. %s (%.2f)" % ( i+1, pair[0], pair[1] ) )<jupyter_output>01. energi (398.26)
02. scenario (216.68)
03. climat (196.51)
04. chang (177.69)
05. electr (174.56)
06. model (174.07)
07. use (172.87)
08. polici (168.62)
09. power (168.03)
10. sector (161.10)
11. develop (153.57)
12. technolog (146.68)
13. increas (145.72)
14. cost (143.19)
15. system (133.95)
16. target (127.75)
17. industri (124.57)
18. renew (123.35)
19. futur (115.95)
20. generat (115.86)
{'fraction_non_alphanumeric': 0.13834999020184205, 'fraction_numerical': 0.03625318440133255, 'mean_word_length': 6.429403202328967, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '19098', 'license': 'MIT', 'n_tokens_mistral': 1898, 'n_tokens_neox': 1762, 'n_words': 374, 'path': '3_Topic_modeling/1.Preprocessing_TFIDF.ipynb', 'repo_name': 'ClaireLepault/Mapping-country-mitig-pathways', 'size': '5103'}