<jupyter_start><jupyter_text>___ ___  Sentence SegmentationIn **spaCy Basics** we saw briefly how Doc objects are divided into sentences. In this section we'll learn how sentence segmentation works, and how to set our own segmentation rules.<jupyter_code># Perform standard imports
import spacy
nlp = spacy.load('en_core_web_sm')
# From Spacy Basics:
doc = nlp(u'This is the first sentence. This is another sentence. This is the last sentence.')

for sent in doc.sents:
    print(sent)<jupyter_output>This is the first sentence.
This is another sentence.
This is the last sentence.<jupyter_text>`Doc.sents` is a generatorIt is important to note that `doc.sents` is a *generator*. That is, a Doc is not segmented until `doc.sents` is called. This means that, where you could print the second Doc token with `print(doc[1])`, you can't call the "second Doc sentence" with `print(doc.sents[1])`:<jupyter_code>print(doc[1])
print(doc.sents[1])<jupyter_output><empty_output><jupyter_text>However, you *can* build a sentence collection by running `doc.sents` and saving the result to a list:<jupyter_code>doc_sents = [sent for sent in doc.sents]
doc_sents<jupyter_output><empty_output><jupyter_text>**NOTE**: `doc_sents = list(doc.sents)` also works. We show a list comprehension as it allows you to pass in conditionals.<jupyter_code># Now you can access individual sentences:
print(doc_sents[1])<jupyter_output>This is another sentence.<jupyter_text>`sents` are SpansAt first glance it looks like each `sent` contains text from the original Doc object. In fact they're just Spans with start and end token pointers.<jupyter_code>type(doc_sents[1])
print(doc_sents[1].start, doc_sents[1].end)<jupyter_output>6 11<jupyter_text>Adding RulesspaCy's built-in `sentencizer` relies on the dependency parse and end-of-sentence punctuation to determine segmentation rules. We can add rules of our own, but they have to be added *before* the creation of the Doc object, as that is where the parsing of segment start tokens happens:<jupyter_code># Parsing the segmentation start tokens happens during the nlp pipeline
doc2 = nlp(u'This is a sentence. This is a sentence. This is a sentence.')

for token in doc2:
    print(token.is_sent_start, ' '+token.text)<jupyter_output>None  This
None  is
None  a
None  sentence
None  .
True  This
None  is
None  a
None  sentence
None  .
True  This
None  is
None  a
None  sentence
None  .<jupyter_text>Notice we haven't run `doc2.sents`, and yet `token.is_sent_start` was set to True on two tokens in the Doc. Let's add a semicolon to our existing segmentation rules. That is, whenever the sentencizer encounters a semicolon, the next token should start a new segment.<jupyter_code># SPACY'S DEFAULT BEHAVIOR
doc3 = nlp(u'"Management is doing things right; leadership is doing the right things." -<NAME>')

for sent in doc3.sents:
    print(sent)
# ADD A NEW RULE TO THE PIPELINE
def set_custom_boundaries(doc):
    for token in doc[:-1]:
        if token.text == ';':
            doc[token.i+1].is_sent_start = True
    return doc

nlp.add_pipe(set_custom_boundaries, before='parser')

nlp.pipe_names<jupyter_output><empty_output><jupyter_text>The new rule has to run before the document is parsed. Here we can either pass the argument `before='parser'` or `first=True`.<jupyter_code># Re-run the Doc object creation:
doc4 = nlp(u'"Management is doing things right; leadership is doing the right things." -<NAME>')

for sent in doc4.sents:
    print(sent)
# And yet the new rule doesn't apply to the older Doc object:
for sent in doc3.sents:
    print(sent)<jupyter_output>"Management is doing things right; leadership is doing the right things."
-<NAME><jupyter_text>Why not change the token directly?Why not simply set the `.is_sent_start` value to True on existing tokens?<jupyter_code># Find the token we want to change:
doc3[7]
# Try to change the .is_sent_start attribute:
doc3[7].is_sent_start = True<jupyter_output><empty_output><jupyter_text>spaCy refuses to change the tag after the document is parsed to prevent inconsistencies in the data.  Changing the RulesIn some cases we want to *replace* spaCy's default sentencizer with our own set of rules. In this section we'll see how the default sentencizer breaks on periods. We'll then replace this behavior with a sentencizer that breaks on linebreaks.<jupyter_code>nlp = spacy.load('en_core_web_sm')  # reset to the original

mystring = u"This is a sentence. This is another.\n\nThis is a \nthird sentence."

# SPACY DEFAULT BEHAVIOR:
doc = nlp(mystring)

for sent in doc.sents:
    print([token.text for token in sent])
# CHANGING THE RULES
from spacy.pipeline import SentenceSegmenter

def split_on_newlines(doc):
    start = 0
    seen_newline = False
    for word in doc:
        if seen_newline:
            yield doc[start:word.i]
            start = word.i
            seen_newline = False
        elif word.text.startswith('\n'): # handles multiple occurrences
            seen_newline = True
    yield doc[start:]      # handles the last group of tokens


sbd = SentenceSegmenter(nlp.vocab, strategy=split_on_newlines)
nlp.add_pipe(sbd)<jupyter_output><empty_output><jupyter_text>While the function `split_on_newlines` can be named anything we want, it's important to use the name `sbd` for the SentenceSegmenter.<jupyter_code>doc = nlp(mystring)
for sent in doc.sents:
    print([token.text for token in sent])<jupyter_output>['This', 'is', 'a', 'sentence', '.', 'This', 'is', 'another', '.', '\n\n']
['This', 'is', 'a', '\n']
['third', 'sentence', '.']
{'fraction_non_alphanumeric': 0.09194159004867496, 'fraction_numerical': 0.004687218316206959, 'mean_word_length': 4.731404958677686, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '66956', 'license': 'MIT', 'n_tokens_mistral': 1819, 'n_tokens_neox': 1757, 'n_words': 665, 'path': 'NLP_COURSE/02-Parts-of-Speech-Tagging/.ipynb_checkpoints/04-Sentence-Segmentation-checkpoint.ipynb', 'repo_name': 'ivy-Ruxin-Tong/Spark-and-Python-for-Big-Data-with-PySpark', 'size': '5568'}