<jupyter_start><jupyter_text>Pre-processing Pipeline COVID-19 Open Research Dataset Challenge (CORD-19) Task DetailsWhat is known about transmission, incubation, and environmental stability? What do we know about natural history, transmission, and diagnostics for the virus? What have we learned about infection prevention and control?The first question we need to ask is what we mean by transmission, incubation, and environmental stability -- or, rather, what should a computer understand when we ask this? We can go about encoding this information in several ways: 1) keywords for analysis in some kind of TF-IDF format, probably including a list of synonyms that we would need to develop by hand, 2) high-dimensional vectors vis a vis word2vec or GloVe, or 3) using heavy, but state-of-the-art transformer models for vectorization. Keywords probably aren't going to give us the robust results we're looking for, because typical pre-processing methods remove all sorts of punctuation and numbers, but these are really important in biomedical texts! We could skip the pre-processing except for removing stop words, but we'd still need to address the fact that keywords have synonyms, and we'd need to hand-write these. But there may be an easier way to get better results without all the hassle. I propose method 2: spaCy is a popular NLP package that's blazingly fast and has (mostly) everything we need to process the text. It'll break sentences apart, lemmatize, and even provide vectors for us. Spacy vectors are somewhat simplistic because the vector of several tokens is just the average of the vectors of each token individually -- so we may not get state of the art results. But we'll get them fast, and we'll know if we need to change something up!<jupyter_code>#!pip install spac scispacy spacy_langdetect https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.2.3/en_core_sci_lg-0.2.3.tar.gz
import spacy 
import scispacy
import pandas as pd 
import os
import numpy as np
#import scispacy
import json
from tqdm.notebook import tqdm
from scipy.spatial import distance
import ipywidgets as widgets
from scispacy.abbreviation import AbbreviationDetector
from spacy_langdetect import LanguageDetector
# UMLS linking will find concepts in the text, and link them to UMLS. 
from scispacy.umls_linking import UmlsEntityLinker
import time<jupyter_output><empty_output><jupyter_text>Time for NLP!Let's load our language model. Based on the type of text we'll be dealing with, we want something that's been pretrained on biomedical texts, as the vocabulary and statistical distribution of words is much different from, say, the news or Wikipedia articles. Luckily, there's already pre-trained models for spacy, so let's load the largest one we can!<jupyter_code>#nlp = spacy.load("en_core_sci_lg")
nlp = spacy.load("en_core_sci_lg", disable=["tagger"])
# If you're on kaggle, load the model with the following, if you run into an error:
#nlp = spacy.load("/opt/conda/lib/python3.6/site-packages/en_core_sci_lg/en_core_sci_lg-0.2.3/", disable=["tagger"])

# We also need to detect language, or else we'll be parsing non-english text 
# as if it were English. 
nlp.add_pipe(LanguageDetector(), name='language_detector', last=True)

# Add the abbreviation pipe to the spacy pipeline. Only need to run this once.
abbreviation_pipe = AbbreviationDetector(nlp)
nlp.add_pipe(abbreviation_pipe)

# Our linker will look up named entities/concepts in the UMLS graph and normalize
# the data for us. 
linker = UmlsEntityLinker(resolve_abbreviations=True)
nlp.add_pipe(linker)<jupyter_output>/home/juanluis/covid19/env/lib/python3.6/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator TfidfTransformer from version 0.20.3 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/home/juanluis/covid19/env/lib/python3.6/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator TfidfVectorizer from version 0.20.3 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)<jupyter_text>Adding a vector for COVID-19One last thing. COVID-19 is a new word, and doesn't exist in the vocabulary for our spaCy model. We'll need to add it manually; let's try setting it to equal the average vector of words that should represent what COVID-19 refers to, and see if that works. I'm not an expert so I just took definitions from Wikipedia and the etiology section of https://onlinelibrary.wiley.com/doi/full/10.1002/jmv.25740. There's a much better way of doing this (fine-tuning the model on our corpus) but I have no idea how to do this in spaCy...<jupyter_code>from spacy.vocab import Vocab
new_vector = nlp(
               """Single‐stranded RNA virus, belongs to subgenus 
                   Sarbecovirus of the genus Betacoronavirus.5 Particles 
                   contain spike and envelope, virions are spherical, oval, or pleomorphic 
                   with diameters of approximately 60 to 140 nm.
                   Also known as severe acute respiratory syndrome coronavirus 2, 
                   previously known by the provisional name 2019 novel coronavirus 
                   (2019-nCoV), is a positive-sense single-stranded RNA virus. It is 
                   contagious in humans and is the cause of the ongoing pandemic of 
                   coronavirus disease 2019 that has been designated a 
                   Public Health Emergency of International Concern""").vector

vector_data = {"COVID-19": new_vector,
               "2019-nCoV": new_vector,
               "SARS-CoV-2": new_vector}

vocab = Vocab()
for word, vector in vector_data.items():
    nlp.vocab.set_vector(word, vector)<jupyter_output><empty_output><jupyter_text>Sanity CheckAlright, let's check if this work.<jupyter_code>print(
    nlp("COVID-19").similarity(nlp("novel coronavirus")), "\n",
    nlp("SARS-CoV-2").similarity(nlp("severe acute respiratory syndrome")), "\n",
    nlp("COVID-19").similarity(nlp("sickness caused by a new virus")))<jupyter_output>0.5324297344523997 
 0.34796126970622626 
 0.7016356811120861<jupyter_text>I guess we'll find out if that's good enough for our purposes! Let's save it so other people can use it!<jupyter_code>#nlp.to_disk('/home/acorn/Documents/covid-19-en_lg')<jupyter_output><empty_output><jupyter_text>Some of the texts is particularly long, so we need to increase the max_length attribute of nlp to more then 1.25mil. The alternative would be cutting the length of the article or dropping it entirely (I believe there's some sort of anomaly with this particular article), but we'll keep it for now.<jupyter_code>nlp.max_length=2000000<jupyter_output><empty_output><jupyter_text>Next, we want to replace all abbreviations with their long forms. This is important for semantic indexing because the model has probably seen words like "Multiple sclerosis" but may have seen the abbreviation "MS" in different contexts. That means their vector representations are different, and we don't want that! So here we'll add the abbreviation expansion module to our scispaCy pipeline.<jupyter_code>doc = nlp("Attention deficit disorcer (ADD) is treated using various medications. However, ADD is not...")

print("Abbreviation", "\t", "Definition")
for abrv in doc._.abbreviations[0:10]:
	print(f"{abrv} \t ({abrv.start_char}, {abrv.end_char}) {abrv._.long_form}")<jupyter_output>Abbreviation 	 Definition
ADD 	 (80, 83) Attention deficit disorcer
ADD 	 (28, 31) Attention deficit disorcer<jupyter_text>Notice we get some weird results towards the end if you print **all** of them (lots of a's being converted to at's, but we can ignore that for now. If we need to remove stop words later, we can.   Making the Vector DataFramesAppending to a dataframe increases time to copy data linearly because df.append copies the entire object. The following will take an article's text, break it into sentences, and vectorize each sentence (using scispacy's pre-trained word2vec model). Finally, the list of dicts is loaded as a DataFrame and saved.So here's the real meat of our pre-processing. This is really heavy because it processes line-by-line and then generates a lot of metadata (entities, vectors). We can break it into pieces later depending on the task we want to use this information for, but querying lines is a lot more useful that querying whole documents when you want to know about something specific like seroconversion, spike proteins, or something else. Once you identify lines of interest, you can generate more data about the actual document, since each line will be indexed with document, start and end character, entities, vectors, and language.  Lemmatized TextJust in case we need it, let's do some text cleaning and include that in a different column. Lemmatization normalizes data so that when you're creating word clouds or simplified TF-IDF, the number of dimesions you're dealing with are significantly reduced. It's also nice to remove words that don't contribute much meaning, but do note that removing stop-words will make neural models less accurate depending on the task you're using them for.<jupyter_code>def df_cleaner(df):
    df.fillna("Empty", inplace=True) # If we leave floats (NaN), spaCy will break.
    for i in df.index:
        for j in range(len(df.columns)):
            if " q q" in df.iloc[i,j]:
                df.iloc[i,j] = df.iloc[i,j].replace(" q q","") # Some articles are filled with " q q q q q q q q q"

#Unnabreviate text. This should be done BEFORE lemmatiztion and vectorization. 
def unnabreviate_my_text(doc):
    if len(doc._.abbreviations) > 0 and doc._.language["language"] == "en":
        doc._.abbreviations.sort()
        join_list = []
        start = 0
        for abbrev in doc._.abbreviations:
            join_list.append(str(doc.text[start:abbrev.start_char]))
            if len(abbrev._.long_form) > 5: #Increase length so "a" and "an" don't get un-abbreviated
                join_list.append(str(abbrev._.long_form))
            else:
                join_list.append(str(doc.text[abbrev.start_char:abbrev.end_char]))
            start = abbrev.end_char
        # Reassign fixed body text to article in df.
        new_text = "".join(join_list)
        # We have new text. Re-nlp the doc for futher processing!
        doc = nlp(new_text)
        return(doc)
    
def pipeline(df, column, dataType, filename):
    create = pd.DataFrame(columns={"_id","language","section","sentence","startChar","endChar","entities","lemma","w2vVector"})
    create.to_csv(filename + "_text_processed" + ".csv", index=False)
    
    docs = nlp.pipe(df[column].astype(str))
    i = -1
    for doc in tqdm(docs):
        languages = []
        start_chars = []
        end_chars = []
        entities = []
        sentences = []
        vectors = []
        _ids = []
        columns = []
        lemmas = []
        i = i + 1
        
        if doc._.language["language"] == "en" and len(doc.text) > 5:
            for sent in doc.sents:
                languages.append(doc._.language["language"])
                sentences.append(sent.text)
                vectors.append(sent.vector)
                start_chars.append(sent.start_char)
                end_chars.append(sent.end_char)
                doc_ents = []
                for ent in sent.ents: 
                    if len(ent._.umls_ents) > 0:
                        poss = linker.umls.cui_to_entity[ent._.umls_ents[0][0]].canonical_name
                        doc_ents.append(poss)
                entities.append(doc_ents)
                _ids.append(df.iloc[i,0])
                if dataType == "tables":
                    columns.append(df.iloc[i]["figure"])
                elif dataType == "text":
                    columns.append(column)
                lemmatized_doc = [token.lemma_ for token in sentence]
                lemmas.append(lemmatized_doc)
        else: 
            start_chars.append(0)
            end_chars.append(len(doc.text))
            entities.append(["Non-English"])
            sentences.append(doc.text)
            vectors.append(np.zeros(200))
            _ids.append(df.iloc[i,0])
            languages.append(doc._.language["language"])
            if dataType == "tables":
                columns.append(df.iloc[i]["figure"])
            elif dataType == "text":
                columns.append(column)
            lemmas.append(["Non-English"])
            
        rows = pd.DataFrame(data={"_id": _ids, "language": languages, "section": columns, "sentence": sentences, 
            "startChar": start_chars, "endChar": end_chars, "entities": entities, "lemma": lemmas, "w2vVector":vectors})
        rows.to_csv(filename, mode='a', header=False, index=False)
        del rows
files = [f for f in os.listdir("./unnabreviated_parts/") if f.startswith("unna") and not f.endswith("csv")]
for f in tqdm(files):
    f = "./unnabreviated_parts/" + f
    df = pd.read_csv(f)
    pipeline(df=df, column="text", dataType="text", filename="tables_unnabrev_lemma")
    os.remove(f)
df_list = []
df = pd.concat([i for i in [pd.read_csv(f) for f in files]])
timestamp = time.strftime("%Y%m%d")
df.to_csv(f"covid_TitleAbstract_processed-{timestamp}.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0871552757793765, 'fraction_numerical': 0.014763189448441247, 'mean_word_length': 3.445369753497668, 'pattern_counts': {'":': 17, '<': 30, '<?xml version=': 0, '>': 34, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '514948', 'license': 'MIT', 'n_tokens_mistral': 4129, 'n_tokens_neox': 3782, 'n_words': 1447, 'path': 'notebooks/COVID-19 Preprocessing.ipynb', 'repo_name': 'shahparthiv/common', 'size': '13344'}