<jupyter_start><jupyter_text>A note on adjusting the rulesIn order to use your new rules, you have to restart the kernel and re-import the libraries every time you add rules to the knowledge_base files.<jupyter_code>import medspacy
import spacy
import cov_bsv
#from cov_bsv import visualize_doc
from medspacy.visualization import visualize_ent
from medspacy.ner import TargetMatcher, TargetRule
from spacy import displacy

import warnings
warnings.filterwarnings("ignore")
with open("O:\\VINCI_COVIDNLP\\va_external_c19_tests\\examples\\outside_test.txt", 'r') as reader:
    text = reader.read()

#print(type(text))
#print(text)
nlp = cov_bsv.load(disable=['document_classifier', "postprocessor"])
#disbled = nlp.disable_pipes("ner")
nlp.pipeline<jupyter_output><empty_output><jupyter_text>Exploring Target Matcher and Rules From the util.py file...- concept_tagger: Assigns a semantic tag in a custom attribute "token._.concept_tag"to each Token in a Doc, which helps with concept extraction and normalization.Concept tag rules are defined in cov_bsv.knowledge_base.concept_tag_rules.- target_matcher: Extracts spans to doc.ents using extended rule-based matching.Target rules are defined in cov_bsv.knowledge_base.target_rules.  DatesUsing the ner model, which comes after the 'concept_tagger', 'target_matcher', and an "entity ruler" with manual date patterns.<jupyter_code>print(spacy.explain("DATE"))<jupyter_output>Absolute or relative dates or periods<jupyter_text>Using entity ruler<jupyter_code>from spacy.pipeline import EntityRuler<jupyter_output><empty_output><jupyter_text>From the spacy docsThe entity ruler is designed to add to the ner. If the entity ruler is added before the ner component, the entity recognizer will respect the existing entity spans and adjust its predictions around it, significantly improving accuracy. If added after the ner component, the entity ruler will only add spans to the doc.ents if they don't overlap with existing entities predicted by the model. TL;DR- Entity ruler first: manual labels take precidence- Ner first: model labels take precidence<jupyter_code>ruler = EntityRuler(nlp)
patterns = [
    {'label': "ORG", "pattern": "VA"},
    {'label': "ORG", "pattern": [{"LOWER":"non"}, {"ORTH":"-", "OP": "?"}, {"LOWER":"va"}, {"LOWER":"facility"}]},
    {'label': "ORG", "pattern": "CVS"},
    {'label': "PRODUCT", "pattern": [{"LOWER":"cprs"}]},
    {'label': "PRODUCT", "pattern": [{"LOWER":"epic", "OP": "?"}, {"LOWER":"ehr"}]},
    {"label": "DATE", "pattern": [{"LOWER": {"REGEX": "\d{1,4}(\\|-|/)\d{1,2}(\\|-|/)\d{1,4}$"}}]},
    {"label": "DATE", "pattern":[{"LOWER": "on"},{"LOWER": {"REGEX": "\d{1,2}(\\|-|/)\d{1,2}$"}}]},
    {"label": "DATE","pattern":[{"LOWER": {"IN": ["january","february","march","april","may","june","july","august","september"
                                         ,"october","november","december","jan","feb","mar","apr","may","jun","jul"
                                         ,"aug","sept","oct","nov","dec"]}}
                       ,{"IS_DIGIT": True, "LENGTH": 2}
                       ,{"ORTH": ",", "OP": "?"}
                       ,{"IS_DIGIT": True, "LENGTH": 4, "OP": "?"}]}, #when there is a space between the day and year
    {"label": "DATE","pattern":[{"LOWER": {"IN": ["january","february","march","april","may","june","july","august","september"
                                         ,"october","november","december","jan","feb","mar","apr","may","jun","jul"
                                         ,"aug","sept","oct","nov","dec"]}}
                       ,{"LOWER": {"REGEX": "\d{1,2},\d{2,4}"}}]}, #when there is no space after the comma

    {"label": "DATE", "pattern":[{"IS_DIGIT": True, "LENGTH": 2}
                      ,{"ORTH": {"IN": ["-","/"]}}
                      ,{"IS_DIGIT": True, "LENGTH": 2}
                      ,{"ORTH": {"IN": ["-","/"]}}
                      ,{"IS_DIGIT": True, "LENGTH": 4}]},
    {"label": "DATE", "pattern":[{"IS_DIGIT": True, "LENGTH": 2}
                      ,{"ORTH": {"IN": ["-","/"]}}
                      ,{"IS_DIGIT": True, "LENGTH": 2}
                      ,{"ORTH": {"IN": ["-","/"]}}
                      ,{"IS_DIGIT": True, "LENGTH": 2}]},
    {"label": "DATE", "pattern":[{"IS_DIGIT": True, "LENGTH": 4}
                      ,{"ORTH": {"IN": ["-","/"]}}
                      ,{"IS_DIGIT": True, "LENGTH": 2}
                      ,{"ORTH": {"IN": ["-","/"]}}
                      ,{"IS_DIGIT": True, "LENGTH": 2}]},
    {"label":"DATETIME", "pattern":[{"LOWER": {"REGEX": "\d{1,2}(\\|-|/)\d{1,2}@\d{2,4}$"}}]}
]
ruler.add_patterns(patterns)
nlp.add_pipe(ruler, before="ner")
nlp.pipeline
dates = """435-908-7890
5/5/2020
March 23, 2020
Apr 26,2020
Jul 22 
3/23/20
10-18-20
Covid negative 4/18@1357
diagnosed with covid on 4/14 at non VA facility
Covid negative on 4/18 at non-va hospital"""
doc = nlp(dates)
cov_bsv.visualize_doc(doc)
dates2 = """
    Positive COVID-19 Lab test done at a location other than this VA. Date: March 3, 2021 
    Covid test from last week
    tested covid (+) this morning
    Negative COVID-19 Non-PCR Antigen Lab test done at a location other than this VA. Date: March 4, 2021 
    he tested positive for covid at a community location 1 week ago
    he reports had COVID on  2/8/2021, tested positive at an urgent care.
    Testing Facility: LabCorp Test Date: Apr 14,2020 Result is by: Patient self-report COVID-19 lab test was positive
"""
doc = nlp(dates2)
cov_bsv.visualize_doc(doc)
for ent in doc.ents:
    print(ent, ent.label_, sep = " | ")
    print()<jupyter_output>COVID-19 Lab test | COVID-19

a location other than this VA | EXTERNAL_LOCATION

March 3, 2021 
     | DATE

Covid | COVID-19

last week | DATE

covid (+) | COVID-19

this morning | TIME

COVID-19 | COVID-19

a location other than this VA | EXTERNAL_LOCATION

March 4, 2021 
     | DATE

covid | COVID-19

community location | EXTERNAL_LOCATION

1 week ago | DATE

COVID | COVID-19

2/8/2021 | DATE

an urgent care | EXTERNAL_LOCATION

Apr 14,2020 | DATE

COVID-19 lab test | COVID-19<jupyter_text>Old - using target rules<jupyter_code>#nlp2 = cov_bsv.load(enable=["tagger", "parser"], load_rules=False) 

nlp2 = cov_bsv.load(disable=['concept_tagger','target_matcher','sectionizer','ner','document_classifier'], load_rules=False)
target_matcher = TargetMatcher(nlp2)
#nlp2.add_pipe(target_matcher)
type(target_matcher)
target_rules = [
   TargetRule(literal="month day, year", category="DATE",
              pattern=[{"LOWER": {"IN": ["january","february","march","april","may","june","july","august","september","october","november","december","jan","feb","mar","apr","may","jun","jul","aug","sept","oct","nov","dec"]}},{"IS_DIGIT": True, "LENGTH": 2},{"ORTH": ",", "OP": "?"},{"IS_DIGIT": True, "LENGTH": 4, "OP": "?"}]),
    TargetRule(literal="day month,year", category="DATE",
              pattern=[{"LOWER": {"IN": ["january","february","march","april","may","june","july","august","september","october","november","december","jan","feb","mar","apr","may","jun","jul","aug","sept","oct","nov","dec"]}},{"LOWER": {"REGEX": "\d{1,2},\d{2,4}"}}]),
    TargetRule(literal="dd/mm/yyyy or yyyy/mm/dd", category="DATE",
              pattern=[{"LOWER": {"REGEX": "\d{1,4}(\\|-|/)\d{1,2}(\\|-|/)\d{1,4}$"}}]),
    TargetRule(literal="on dd/yyyy", category="DATE",
              pattern=[{"LOWER": "on"},{"LOWER": {"REGEX": "\d{1,2}(\\|-|/)\d{1,4}$"}}]),
    TargetRule(literal="dd-mm-yyyy", category="DATE",
              pattern=[{"IS_DIGIT": True, "LENGTH": 2},{"ORTH": {"IN": ["-","/"]}},{"IS_DIGIT": True, "LENGTH": 2},{"ORTH": {"IN": ["-","/"]}},{"IS_DIGIT": True, "LENGTH": 4}]),
    TargetRule(literal="dd-mm-yy", category="DATE",
              pattern=[{"IS_DIGIT": True, "LENGTH": 2},{"ORTH": {"IN": ["-","/"]}},{"IS_DIGIT": True, "LENGTH": 2},{"ORTH": {"IN": ["-","/"]}},{"IS_DIGIT": True, "LENGTH": 2}]),
    TargetRule(literal="yyyy-dd-mm", category="DATE",
              pattern=[{"IS_DIGIT": True, "LENGTH": 4},{"ORTH": {"IN": ["-","/"]}},{"IS_DIGIT": True, "LENGTH": 2},{"ORTH": {"IN": ["-","/"]}},{"IS_DIGIT": True, "LENGTH": 2}]),
    TargetRule(literal="dd/mm@tttt", category="DATETIME",
              pattern=[{"LOWER": {"REGEX": "\d{1,2}(\\|-|/)\d{1,2}@\d{2,4}$"}}])
]

target_matcher.add(target_rules)
dates = """435-908-7890
5/5/2020
March 23, 2020
Apr 26,2020
Jul 22 
3/23/20
11\15\21
10-18-20
4/14: some event
Covid negative 4/18@1357"""
#text = """Patient tested negative for novel coronavirus on 02 Apr 20, and positive for covid-19 on May 5, 2020."""
doc = nlp(dates)
cov_bsv.visualize_doc(doc)
for ent in doc.ents:
    print(ent.text, ent.label_, sep=" -> ")<jupyter_output>5/5/2020 -> DATE
March 23, 2020 -> DATE
Jul 22 -> DATE
3/23/20 -> DATE
10-18-20 -> DATE
Covid -> COVID-19
4/18@1357 -> DATE<jupyter_text>Example of the difference between the default spaCy and the medspaCy tokenizers<jupyter_code>from medspacy.custom_tokenizer import create_medspacy_tokenizer

nlp = spacy.blank("en")

medspacy_tokenizer = create_medspacy_tokenizer(nlp)
default_tokenizer = nlp.tokenizer

example = r'Jul 22,2020' #r'10-18-20 435-908-7890' #r'December(23rd)'  #r'Pt c/o n;v;d h/o chf+cp'

print("spacy tokenizer:")
print(list(default_tokenizer(example)))
print('\n')
print("medspacy tokenizer:")
print(list(medspacy_tokenizer(example)))
#for ent in doc.ents:
    #print(ent.text, ent.label_, sep=" -> ")<jupyter_output>spacy tokenizer:
[Jul, 22,2020]


medspacy tokenizer:
[Jul, 22, ,, 2020]
{'fraction_non_alphanumeric': 0.20914069081718617, 'fraction_numerical': 0.03517270429654591, 'mean_word_length': 3.9853018372703413, 'pattern_counts': {'":': 130, '<': 21, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '386131', 'license': 'MIT', 'n_tokens_mistral': 3593, 'n_tokens_neox': 3257, 'n_words': 792, 'path': 'Dates.ipynb', 'repo_name': 'anniebowles/VA_COVID-19_NLP_BSV', 'size': '9496'}