<jupyter_start><jupyter_text>Natural Language Processing on COVID-19 medical literature papers<NAME>   University of Toronto   Faculty of Information <EMAIL>   1. Data Cleanning<jupyter_code>import numpy as np
import pandas as pd
import re
import matplotlib.pyplot as plt
from wordcloud import WordCloud, STOPWORDS

import seaborn as sns

import gensim
from gensim.utils import simple_preprocess
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
import gensim.corpora as corpora
from gensim.models import CoherenceModel
import pyLDAvis
import pyLDAvis.gensim_models


from pprint import pprint
import warnings
warnings.filterwarnings('ignore')
df = pd.read_csv('metadata.csv')
df.info(verbose=True)
df.head()
# load the meta data from the CSV file and extract out fields that will be useful to our application
df = df[['title','abstract','authors','publish_time']]
df = df.drop_duplicates(subset='abstract', keep="first")
df=df.dropna()
df["abstract"] = df["abstract"].str.lower()
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Since the inception of COVID-19 outbreak was first reported on 31 December 2019, papers prior to the date are assumed to be irrelevant to COVID-19, I filtered out those rows and construct a dataframe on `COVID-19`.<jupyter_code>len(df[df['publish_time']>='2020-01-01'])
df_covid = df[df['publish_time']>='2020-01-01']
df_covid.head()
# add a column for publish year
df_covid['publish_year']= pd.DatetimeIndex(df_covid['publish_time']).year
df_covid.head()
df_covid['publish_monyr'] = pd.to_datetime(df_covid['publish_time']).dt.to_period('M')
df_covid.head()<jupyter_output><empty_output><jupyter_text>2. Data Visualization & Exploratory Data Analysis **Distribution of published papers over time**<jupyter_code>plt.figure(figsize=(10,6))
monyr_pub = df_covid.groupby('publish_monyr')['publish_monyr'].count()
monyr_pub.plot.barh(rot=0, color='mediumblue', alpha=0.6)
plt.xlabel('Count')
plt.ylabel('Publish month')
plt.title('Number of published COVID-19 papers by month')
plt.show()
annual_pub = df_covid.groupby('publish_year')['publish_year'].count()
annual_pub.plot.bar(rot=0, color='mediumblue', alpha=0.6)
plt.xlabel('Publish year')
plt.ylabel('Count')
plt.title('Number of published COVID-19 papers by year')
plt.show()<jupyter_output><empty_output><jupyter_text>Since the Novel Coronavirus (COVID-19) outbreak on Decemeber 31st 2019, researchers are sparing no effort to investigate about this highly-contagious pandemic in order to find the ultimate cure for mankind. So far there has been over 150,000 related papers issued, and January 2021 witnesses a jump in the number of published COVID-19 papers at its "one-year anniversary". **Check how many COVID-19 papers are related to vaccine💉**<jupyter_code>vaccine_df1 = df_covid[df_covid['abstract'].str.contains("vaccine", regex=False)]
vaccine_df1.shape
monyr_pub1 = vaccine_df1.groupby('publish_monyr')['publish_monyr'].count()
monyr_pub1.plot.line(figsize=(10,6),label='publish_month', color='mediumblue', alpha=0.6, lw=2)
plt.xlabel('Publish month')
plt.ylabel('Count')
plt.title('Published vaccine-related papers by month')
plt.xlim('2020-01','2021-04')
plt.show()<jupyter_output><empty_output><jupyter_text>The timeline for the vaccine paper release somehow echos with the [vaccine development timeline](https://www.biopharmadive.com/news/coronavirus-vaccine-pipeline-types/579122/). It is observed that the first rise in the number of vaccine related paper happens in March, when the major 5 pharmaceutical companies (Pfizer, BioNTech, Moderna, AstraZeneca, and Novavax) announced their vaccine developments plan and released experiment reports. The number of publications addressing COVID-19 vaccine peaked in January 2021, prior to their large-scale vaccine production.Notice that among the published hundreds of thousands of papers, papers related to vaccine account for only 6%.  **Visualize the top frequent words in the COVID-19 paper abstracts**<jupyter_code># create a word cloud to visualize the most frequent key words in the paper abstracts on COVID-19
plt.figure(figsize=(10,6))
text = " ".join(ab for ab in df_covid['abstract'])
stopwords = set(STOPWORDS)
stopwords.update(["covid", "cov", "result", "conclusion", "used", "use", "study" ,'method','patient','pandemic'])
wordcloud = WordCloud(stopwords=stopwords, background_color="white").generate(text)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>**Visualize the top frequent words in the COVID-19 paper titles**<jupyter_code># create a word cloud to visualize the top frequent keywords in the paper titles on COVID-19
plt.figure(figsize=(10,6))
text = " ".join(ab for ab in df_covid['title'])
stopwords = set(STOPWORDS)
stopwords.update(["covid", "cov", "result", "conclusion", "used", "use", "study" ,'method','patient','pandemic'])
wordcloud = WordCloud(stopwords=stopwords, background_color="white").generate(text)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>After removing the specific common words for COVID-19 and research papers (i.e. covid, pandemic, study, etc.), we can see that the most mentioned words are SARS, patients, infection, impact, treatment, effect and so on, which are problem-solving-oriented, aiming to study the subject matter so as to cure mankind of the Novel coronavirus pandemic.     Besides the big pitcure shown by word clouds on the top keywords for paper abstract and title, the next question is what topic dimension are those papers focusing on, what areas have been studied thoroughly, and which topic needs the most attention? The questions above motivate me to explore the COVID-19 literature papers through **topic mining**.  Consequently, I would be able to gain insights on the current medical research landscape to better inform how our healthcare system, governments, industries can fight against COVID-19.  3. Modeling: Topic modelling using Latent Dirichlet Allocation (LDA)**Topic mining on COVID-19 scientific papers**source code: https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/15visualizethetopicskeywords Gensim documentation: https://radimrehurek.com/gensim/auto_examples/tutorials/run_lda.html Topic Modeling is a technique to extract the hidden topics from large volumes of text through conducting Dimensionality Reduction, Unsupervised Learning, and Tagging on the given texts. This project uses the **Latent Dirichlet Allocation model** to mine the topics in the COVID-19 medical research papers published after 2020.  Create Bigram and Trigram Models<jupyter_code>def sent_to_words(sentences):
    for sentence in sentences:
        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))  # deacc=True removes punctuations

data = df_covid.abstract.values.tolist()
data_words = list(sent_to_words(data))
# Build the bigram and trigram models
bigram = gensim.models.Phrases(data_words, min_count=5, threshold=100) # higher threshold fewer phrases.
trigram = gensim.models.Phrases(bigram[data_words], threshold=100)  

# Faster way to get a sentence clubbed as a trigram/bigram
bigram_mod = gensim.models.phrases.Phraser(bigram)
trigram_mod = gensim.models.phrases.Phraser(trigram)

# See trigram example
print(trigram_mod[bigram_mod[data_words[0]]])
stop_words = stopwords.words('english')
stop_words.extend(['covid', 'cov', 'result', 'conclusion', 'used','use','study','method','patients','pandemic',\
                   'disease','coronavirus','case','data','model','also','research','paper'])


def remove_stopwords(texts):
    return [[word for word in simple_preprocess(str(doc)) if word not in stop_words] for doc in texts]

def make_bigrams(texts):
    return [bigram_mod[doc] for doc in texts]

# Lemmatize the documents
def lemm(texts):
    return [[WordNetLemmatizer().lemmatize(word, pos='v') for word in doc] for doc in texts]
# remove stop words
data_words = remove_stopwords(data_words)

# form bigrams
data_words = make_bigrams(data_words)

# Lemmatization
data_words = lemm(data_words)
print(data_words[:1][0][:30])
# Create Dictionary
id2word = corpora.Dictionary(data_words)
# Filter out words that occur less than 20 documents, or more than 50% of the documents
id2word.filter_extremes(no_below=20, no_above=0.5)

# Create Corpus
texts = data_words
# Term Document Frequency
corpus = [id2word.doc2bow(text) for text in texts]
# View
print(corpus[:1][0][:30])<jupyter_output>[(0, 1), (1, 1), (2, 1), (3, 1), (4, 3), (5, 1), (6, 2), (7, 1), (8, 2), (9, 1), (10, 3), (11, 1), (12, 1), (13, 1), (14, 2), (15, 1), (16, 1), (17, 1), (18, 1), (19, 2), (20, 2), (21, 2), (22, 1), (23, 2), (24, 5), (25, 7), (26, 1), (27, 1), (28, 1), (29, 1)]<jupyter_text>**Build the LDA model**set number of topics to be 10, set number of keywords to view by 20<jupyter_code># number of topics
num_topics = 10

# Build LDA model
lda_model = gensim.models.LdaMulticore(corpus=corpus,
                                      id2word=id2word,
                                      num_topics=num_topics,
                                      random_state=116)<jupyter_output><empty_output><jupyter_text>Compute Coherence Score<jupyter_code># Compute Coherence Score
coherence_model_lda = CoherenceModel(model=lda_model, texts=data_words, dictionary=id2word, coherence='c_v')
coherence_lda = coherence_model_lda.get_coherence()
print('\nCoherence Score: ', coherence_lda)<jupyter_output>Coherence Score:  0.39209841114899013<jupyter_text>**Find the optimal number of topics for LDA**<jupyter_code>def compute_coherence_values(dictionary, corpus, texts, limit, start=2, step=2):
    """
    Compute c_v coherence for various number of topics

    Parameters:
    ----------
    dictionary : Gensim dictionary
    corpus : Gensim corpus
    texts : List of input texts
    limit : Max num of topics

    Returns:
    -------
    model_list : List of LDA topic models
    coherence_values : Coherence values corresponding to the LDA model with respective number of topics
    """
    coherence_values = []
    model_list = []
    for num_topics in range(start, limit, step):
        model = gensim.models.LdaMulticore(corpus=corpus,id2word=id2word,num_topics=num_topics,random_state=116)
        model_list.append(model)
        coherencemodel = CoherenceModel(model=model, texts=texts, dictionary=dictionary, coherence='c_v')
        coherence_values.append(coherencemodel.get_coherence())

    return model_list, coherence_values
model_list, coherence_values = compute_coherence_values(dictionary=id2word, corpus=corpus, texts=data_words, start=2, limit=15, step=2)
# show graph
limit=15; start=2; step=2;
x = range(start, limit, step)
plt.plot(x, coherence_values)
plt.xlabel("Number of Topics")
plt.ylabel("Coherence score")
plt.legend(("coherence_values"), loc='best')
plt.show()<jupyter_output><empty_output><jupyter_text>Therefore, the best LDA model has 8 topics. **Build the best model**<jupyter_code># number of topics
num_topics = 8

# Build the best LDA model
best_model = gensim.models.LdaMulticore(corpus=corpus,
                                      id2word=id2word,
                                      num_topics=num_topics,
                                      random_state=116)
pprint(best_model.show_topics(num_topics=8,num_words=20))<jupyter_output>[(0,
  '0.012*"health" + 0.008*"test" + 0.006*"risk" + 0.006*"report" + '
  '0.006*"sars" + 0.005*"ci" + 0.005*"among" + 0.005*"participants" + '
  '0.005*"increase" + 0.005*"anxiety" + 0.005*"symptoms" + 0.004*"level" + '
  '0.004*"find" + 0.004*"mental" + 0.004*"social" + 0.004*"age" + '
  '0.004*"associate" + 0.004*"time" + 0.004*"methods" + 0.004*"include"'),
 (1,
  '0.009*"cells" + 0.008*"de" + 0.007*"cell" + 0.005*"sars" + '
  '0.005*"infection" + 0.005*"expression" + 0.004*"protein" + 0.004*"virus" + '
  '0.004*"show" + 0.003*"may" + 0.003*"viral" + 0.003*"human" + 0.003*"immune" '
  '+ 0.003*"include" + 0.003*"analysis" + 0.003*"find" + 0.003*"increase" + '
  '0.003*"la" + 0.003*"target" + 0.003*"response"'),
 (2,
  '0.015*"sars" + 0.010*"health" + 0.007*"drug" + 0.005*"virus" + '
  '0.004*"human" + 0.004*"include" + 0.004*"viral" + 0.004*"review" + '
  '0.004*"base" + 0.004*"transmission" + 0.003*"potential" + 0.003*"find" + '
  '0.003*"report" + 0.003*"spread" + 0.003*"increa[...]<jupyter_text>Visualize the topics in LDA model Notice that the topics are ranked by number of occurrences in the abstracts instead of topic number output by the LDA model.<jupyter_code>pyLDAvis.enable_notebook()
vis_best = pyLDAvis.gensim_models.prepare(best_model, corpus, id2word)
vis_best
pyLDAvis.save_html(vis_best, './ldatopics_vis.html')<jupyter_output><empty_output><jupyter_text>Find the dominant topic for each paper abstract<jupyter_code>def format_topics_document(ldamodel=lda_model, corpus=corpus, texts=data):
    # Init output
    sent_topics_df = pd.DataFrame()
    # Get main topic in each document
    for i, row in enumerate(ldamodel[corpus]):
        row = sorted(row, key=lambda x: (x[1]), reverse=True)
        # Get the Dominant topic, Perc Contribution and Keywords for each document
        for j, (topic_num, prop_topic) in enumerate(row):
            if j == 0:  # => dominant topic
                wp = ldamodel.show_topic(topic_num)
                topic_keywords = ", ".join([word for word, prop in wp])
                sent_topics_df = sent_topics_df.append(pd.Series([int(topic_num), round(prop_topic,4), topic_keywords]), ignore_index=True)
            else:
                break
    sent_topics_df.columns = ['Dominant_Topic', 'Perc_Contribution', 'Topic_Keywords']

    # Add original text to the end of the output
    contents = pd.Series(texts)
    sent_topics_df = pd.concat([sent_topics_df, contents], axis=1)
    return(sent_topics_df)
df_topic_best_keywords = format_topics_document(ldamodel=best_model, corpus=corpus, texts=data)
df_topic_best_keywords.columns = ['Dominant_Topic', 'Topic_Perc_Contrib', 'Keywords', 'Text']
# convert topic number to integer
df_topic_best_keywords['Dominant_Topic'] = df_topic_best_keywords['Dominant_Topic'].astype(int)
df_topic_best_keywords.head(10)<jupyter_output><empty_output><jupyter_text>**Summarize 8 topics into meaningful labels based on their salient key words**Topic 0 : anxiety, mental, depression -> `mental health`Topic 1: cell, expression, protein, genes -> `virus genetics`Topic 2: drug, effect -> `drug`Topic 3: care, patient, clinical, treatment -> `medical care`Topic 4: sars, infection, respiratory, syndrome -> `diagnostics`Topic 5: base, test -> `test`Topic 6: age, severe, years, mortality -> `patient description`Topic 7: countries, epidemic, population, public -> `population studies`<jupyter_code># map the 8 numeric topic numbers to its corresponding text label
topic_map_best = {0: 'mental health',
            1: 'virus genetics',
            2: 'drug',
            3: 'medical care',
            4: 'diagnostics',
            5: 'test',
            6: 'patient description',
            7: 'population studies'}
df_topic_best_keywords['Dominant_Topic_Label'] = df_topic_best_keywords['Dominant_Topic'].map(topic_map_best)
topic_label_countb = df_topic_best_keywords.groupby('Dominant_Topic_Label')['Dominant_Topic_Label'].count()
topic_label_countb.plot.barh(rot=0, color='mediumblue', alpha=0.6)
plt.xlabel('Count')
plt.ylabel('Topic label')
plt.title('Topic distribution for COVID-19 papers')
plt.show()<jupyter_output><empty_output><jupyter_text>There are eight topics (`mental health`, `virus genetics`, `drug`, `medical care`, `diagnotics`, `test`, `patient description`, and `population studies`) identified from the COVID-19 biomedical literaure dataset by the LDA model. Amongst them, the field of **drug** (i.e. vaccine development and therapeutics) has been studied the least. Perhaps it is because that it takes significant amount of time for drug development such as vaccine and therapeutics, from experimental design, model building to clinical tests. In addition, the drug development needs to be based on theoretical and clinical evidence from other research papers, which justifies for its low publish number.   Sub-topic mining on vaccine related papersNext, let's further explore the sub-topics in the vaccine related papers. What aspects have they addressed in detail? Are they similar in nature?<jupyter_code>vac_data = vaccine_df1.abstract.values.tolist()
vac_data_words = list(sent_to_words(vac_data))

# build bigram and trigram models
vac_bigram = gensim.models.Phrases(vac_data_words, min_count=5, threshold=60)
vac_trigram = gensim.models.Phrases(vac_bigram[vac_data_words], threshold=60)

vac_bigram_mod = gensim.models.phrases.Phraser(vac_bigram)
vac_trigram_mod = gensim.models.phrases.Phraser(vac_trigram)
stop_words = stopwords.words('english')
stop_words.extend(['covid','cov','vaccine','pandemic','study','disease','vaccination','vaccines','use','coronavirus','also','research','may'])

# remove stop words
vac_data_words = remove_stopwords(vac_data_words)

# form bigrams
vac_data_words = make_bigrams(vac_data_words)

# Lemmatization
vac_data_words = lemm(vac_data_words)
# create dictionary for vaccine data
id2word_vac = corpora.Dictionary(vac_data_words)

# create corpus
vac_texts = vac_data_words

# term document frequency
vac_corpus = [id2word_vac.doc2bow(vac_text) for vac_text in vac_texts]
# build the LDA model for vaccine data
n_topics = 5

lda_model_vac = gensim.models.LdaMulticore(corpus=vac_corpus,
                                          id2word=id2word_vac,
                                          num_topics=n_topics,
                                          random_state=116)
pprint(lda_model_vac.show_topics(num_topics=5, num_words=20))<jupyter_output>[(0,
  '0.016*"sars" + 0.006*"infection" + 0.005*"virus" + 0.004*"protein" + '
  '0.004*"immune" + 0.004*"development" + 0.004*"result" + 0.004*"include" + '
  '0.004*"patients" + 0.004*"respiratory" + 0.004*"dose" + 0.004*"human" + '
  '0.004*"effective" + 0.004*"severe" + 0.003*"responses" + 0.003*"show" + '
  '0.003*"target" + 0.003*"report" + 0.003*"viral" + 0.003*"clinical"'),
 (1,
  '0.011*"sars" + 0.006*"virus" + 0.005*"cell" + 0.005*"protein" + '
  '0.005*"health" + 0.005*"patients" + 0.004*"result" + 0.004*"infection" + '
  '0.004*"base" + 0.004*"drug" + 0.004*"viral" + 0.004*"test" + '
  '0.004*"development" + 0.003*"identify" + 0.003*"clinical" + 0.003*"cells" + '
  '0.003*"develop" + 0.003*"case" + 0.003*"novel" + 0.003*"respiratory"'),
 (2,
  '0.018*"sars" + 0.007*"virus" + 0.006*"health" + 0.006*"infection" + '
  '0.004*"result" + 0.004*"viral" + 0.004*"drug" + 0.004*"influenza" + '
  '0.004*"control" + 0.004*"clinical" + 0.004*"base" + 0.004*"test" + '
  '0.003*"severe" [...]
{'fraction_non_alphanumeric': 0.13014389635214782, 'fraction_numerical': 0.04178835023628737, 'mean_word_length': 5.067654639175258, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '416092', 'license': 'MIT', 'n_tokens_mistral': 6896, 'n_tokens_neox': 6090, 'n_words': 1615, 'path': 'topic_model_covid_papers.ipynb', 'repo_name': 'StephaininZ/topic-model-medical-papers-covid19', 'size': '18863'}