<jupyter_start><jupyter_text>OverviewAs we saw in the last notebook, spaCy doesn't work great for clinical text out of the box. We're interested in extracting different types of information from clinical text than news or Wikipedia articles. Clinical text is also very different from general domain language. - **It is very messy**, with semi-structured formatting from EHR- Clinical documents include **many abbreviations**, some of which are ambiguous- There are **specific tasks** needed in clinical NLP, such as **detecting negation or uncertainty** for concepts in the textOne of the most powerful components of spaCy is that is **very customizable**. In addition to working with the default models provided in the core library, you can create your own [custom components](https://spacy.io/usage/processing-pipelinescustom-components) or add your own [extension attributes](https://spacy.io/usage/processing-pipelinescustom-components-attributes). Developers and researchers can then publish their spaCy extensions to the open-source community. Some examples of these openly available libraries are:- [scispacy](https://allenai.github.io/scispacy/): Includes models trained on biomedical literature- [medCAT](https://github.com/CogStack/MedCAT): Models trained for medical concept extractionIn the next two notebooks, we'll use [medspacy](https://github.com/medspacy/medspacy), a newly released package for performing clinical NLP tasks in spaCy.  medspacy[Medspacy](https://github.com/medspacy/medspacy) is an open-source package maintained by NLP developers at the University of Utah and the US Department of Veterans Affairs. The goal of medSpaCy is to provide flexible, easy-to-use spaCy components for common clinical NLP tasks, such as:- Concept extraction- Negation detection- Document section splittingOne of the early uses of medSpaCy includes a [biosurveillance system for identifying positive cases of COVID-19](https://openreview.net/forum?id=ZQ_HvBxcdCv).**MedSpaCy is still in beta**, and you are one of the first users! How to complete this notebook YouTube LinksThroughout this notebook, we'll provide links to Dr. <NAME>man's talk [**"Developing Clinical Natural Language Processing System"**](https://youtu.be/UEm7H8cfz80), where she talks about the challenges in clinical NLP and how we can overcome them. We'll then put those ideas into context processing examples of clinical text with medSpaCy. TODO ExercisesSome cells will be marked with **TODO**. Whenever you see this, you'll be asked to complete the following code. Some of the code will be partially completed with `___` as "fill-in-the-blank" placeholders. For example, if you see:--- TODOChange the code below to print out **"Hello, world!"**```pythonprint(___)```---You should chage the code to:```pythonprint("Hello, world!")```  I. Getting started with medSpaCyLet's get started with medSpaCy. Just like with spaCy, we'll load a model containing a processing pipeline. Unlike the typical spaCy models, this pipeline will include some additional components for specific clinical tasks.<jupyter_code>from IPython.display import YouTubeVideo
import spacy
import medspacy
nlp = medspacy.load()
nlp.pipe_names<jupyter_output><empty_output><jupyter_text>For data, we'll use these short example texts:<jupyter_code>texts = [
    "Patient presents for management of Type II Diabetes Mellitus",
    "No evidence of pneumonia",
    "Past medical history significant for afib, CHF, and CKD Stage 3, now CKD stage five.",
    "Mother with breast cancer",
    "continue metformin for type 2 dm",
    "Her grandmother was recently diagnosed with Alzheimer's"
]<jupyter_output><empty_output><jupyter_text>DiscussionWhat information would be useful to extract from these texts? What processing steps do you need to take?  II. Concept extractionThe first step we'll take is to define the **target concepts** we're interested in. In the previous notebook, spaCy extracted concepts like **"PERSON"** and **"ORG"**. In this notebook, we'll extract the following labels:- **"PROBLEM"**- **"TREATMENT"**- **"TEST"**We'll start by building a **rule-based system**. In rule-based NLP, we define patterns to match concepts in text. SpaCy offers many [rule-based methods](https://spacy.io/usage/rule-based-matching). MedSpaCy uses a pipeline component called `TargetMatcher` and rules defined by a class called `TargetRule`. Extracted concepts will be stored as `Span` objects in `doc.ents`.We can access the target matcher through the `get_pipe()` method:<jupyter_code>target_matcher = nlp.get_pipe("target_matcher")
target_matcher
# Import class for defining rules
from medspacy.ner import TargetRule<jupyter_output><empty_output><jupyter_text>Target rules require two positional arguments:- `literal`: A span of text to match in the text (case insensitive)- `category`: The label to assign to extracted conceptsLet's define rules to extract any relevant clinical concepts in the texts above.<jupyter_code>target_rules = [
    TargetRule("pneumonia", "PROBLEM"),
    TargetRule("afib", "PROBLEM"),
    TargetRule("CHF", "PROBLEM"),
    TargetRule("Breast Cancer", "PROBLEM"),
    TargetRule("Alzheimer's", "PROBLEM"),
    TargetRule("metformin", "TREATMENT"),
]<jupyter_output><empty_output><jupyter_text>We then add these rules to our target matcher:<jupyter_code>target_matcher.add(target_rules)<jupyter_output><empty_output><jupyter_text>The simplest form of string-matching in spaCy is just exact strings to match, as shown above. However, we can also add more complex patterns to match concepts with varying form. For example, the same or similar concepts can be mentioned in multiple formats:- **"Type II Diabetes Mellitus"** and **"type 2 dm"**- **"CKD Stage 3"** and **"CKD Stage Five"**We can write more complex rules using token attributes to match multiple string formats at once. A pattern is a list of dictionaries representing conditions to match on each token. See SpaCy's documentation on [rule-based matching](https://spacy.io/usage/rule-based-matching) for more information on how these patterns work.<jupyter_code>target_rules2 = [
    TargetRule("CKD", "PROBLEM", pattern=[
        {"LOWER": "ckd"}, # Token 1
        {"LOWER": "stage"}, # Token 2
        {"LIKE_NUM": True} # Token 3
        ]),
    
    TargetRule("Type II Diabetes Mellitus", "PROBLEM", 
              pattern=[
                  {"LOWER": "type"},
                  {"LOWER": {"IN": ["2", "ii", "two"]}},
                  {"LOWER": {"IN": ["dm", "diabetes"]}},
                  {"LOWER": "mellitus", "OP": "?"}
              ]),
]
target_matcher.add(target_rules2)<jupyter_output><empty_output><jupyter_text>Now we can process the texts by calling `nlp.pipe(texts)`:<jupyter_code>docs = list(nlp.pipe(texts))<jupyter_output><empty_output><jupyter_text>Let's visualize the entities extracted in these docs.<jupyter_code>from spacy import displacy
displacy.render(docs, style="ent")<jupyter_output><empty_output><jupyter_text>TODO Add rules to extract any problems, treatments, or tests in the text below. Add it to the target matcher and process the text.<jupyter_code>text = """Patient presents to emergency department with complaint of SOB, cough and chills.
Ordered a respiratory panel to test for SARS-COV-2. 

Patient has an allergy to penicillin.
"""
new_target_rules = [
    TargetRule(____, "PROBLEM", pattern=[]),
    TargetRule(____, ____), pattern=[],
    # ...
]
target_matcher.add(new_target_rules)
doc = nlp(text)
displacy.render(doc, style="ent")<jupyter_output><empty_output><jupyter_text>What makes named entity recognition difficult in clinical texts?<jupyter_code>YouTubeVideo("UEm7H8cfz80", start=1638, end=1735, rel=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09851132686084142, 'fraction_numerical': 0.003883495145631068, 'mean_word_length': 5.185748598879103, 'pattern_counts': {'":': 10, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '153245', 'license': 'MIT', 'n_tokens_mistral': 2254, 'n_tokens_neox': 2101, 'n_words': 859, 'path': 'notebooks/nlp-02-medspacy-concept-extraction.ipynb', 'repo_name': 'chapmanbe/isys90069_w2020_explore', 'size': '7729'}