<jupyter_start><jupyter_text>Adding to the SectionizerBy default, `clinical_sectionizer` comes with a number of built-in patterns. However, this is a non-exhaustive list and your data will almost certainly contain a number of sections which aren't captured by the default patterns. In this notebook, we'll see how to add custom section patterns to our clinical sectionizer to recognize section headers which are not contained in the default knowledge base.<jupyter_code>%load_ext autoreload
%autoreload 2
import sys
sys.path.insert(0, "..")
import spacy
from clinical_sectionizer import Sectionizer
from medspacy.visualization import visualize_ent 

nlp = spacy.load("en_info_3700_i2b2_2012")
sectionizer = Sectionizer(nlp)
nlp.add_pipe(sectionizer)
nlp.pipe_names<jupyter_output><empty_output><jupyter_text>Available default sectionsYou can see which section titles are available in the `sectionizer` by the `sectionizer.section_titles` property:<jupyter_code>sectionizer.section_titles<jupyter_output><empty_output><jupyter_text>You can also view the patterns in `sectionizer.patterns`. This will be explained more below.<jupyter_code>sectionizer.patterns[:5]<jupyter_output><empty_output><jupyter_text>In this example, we'll use a smaller section of the note before:<jupyter_code>text = """
Admission Date:  [**2573-5-30**]              Discharge Date:   [**2573-7-1**]
 
Date of Birth:  [**2498-8-19**]             Sex:   F
 
Service: SURGERY
 
Allergies: 
Hydrochlorothiazide
 
Attending:[**First Name3 (LF) 1893**] 
Chief Complaint:
Abdominal pain


Pertinent Results:
[**2573-5-30**] 09:10PM BLOOD WBC-19.2*# RBC-4.81 Hgb-15.5 Hct-44.0 
MCV-92 MCH-32.3* MCHC-35.2* RDW-13.3 Plt Ct-230
[**2573-5-30**] 09:10PM BLOOD Neuts-87* Bands-10* Lymphs-3* Monos-0 
"""
doc = nlp(text)
visualize_ent(doc)
doc._.section_titles<jupyter_output><empty_output><jupyter_text>The sectionizer correctly recognizes **"Allergies"** and **"Chief Complaint"** as section headers. However, some other titles which might be useful to extract are:- **"Admission Date"**: Many MIMIC notes start this way and you could consider this first section to be **visit_information**- **"Pertinent Results**: This is a section of **"lab"** results  Add patternsTo recognize these sections, we can add **patterns** to the sectionizer. These patterns resemble spaCy's [rule-based matching API](https://spacy.io/usage/rule-based-matching). Each pattern is a dictionary with two keys:- `section_title`: The normalized name of the section which will be available in `ent._.section_title`- `pattern`: Either a string (for exact match, case insensitive) or a list of dictionaries (for matching on additional token attributes) which define the text to match<jupyter_code>new_patterns = [
    {"section_title": "visit_information", "pattern": [{"LOWER": {"REGEX": "admi(t|ssion)"}}, {"LOWER": "date"}, {"LOWER": ":"}]},
    {"section_title": "labs_and_studies", "pattern": "Pertinent Results:"}
]<jupyter_output><empty_output><jupyter_text>We add this list of patterns through the `sectionizer.add` method:<jupyter_code>sectionizer.add(new_patterns)<jupyter_output><empty_output><jupyter_text>Now if we reprocess and visualize our doc, we can see that the new headers have been extracted:<jupyter_code>doc = nlp(text)
visualize_ent(doc)
doc._.section_titles<jupyter_output><empty_output><jupyter_text>Loading a blank sectionizerYou can load the `sectionizer` without the default patterns and only custom patterns:<jupyter_code>blank_sectionizer = Sectionizer(nlp, patterns=None)
blank_sectionizer._patterns
blank_sectionizer._section_titles<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10743801652892562, 'fraction_numerical': 0.02534435261707989, 'mean_word_length': 6.320564516129032, 'pattern_counts': {'":': 9, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '384908', 'license': 'MIT', 'n_tokens_mistral': 1230, 'n_tokens_neox': 1140, 'n_words': 354, 'path': 'notebooks/01-adding_sections.ipynb', 'repo_name': 'medspacy/sectionizer', 'size': '3630'}