<jupyter_start><jupyter_text>Pattern matching in spaCyLooking at the token matcher and phrase matcher as well as some tricks to speed up the matcher.This notebook is based on https://spacy.io/usage/rule-based-matching, which also goes into the dependency matcher and the entity ruler.<jupyter_code>import spacy
from spacy.matcher import Matcher
from spacy.matcher import PhraseMatcher
nlp = spacy.load("en_core_web_sm")<jupyter_output><empty_output><jupyter_text>Matching on tokensHere we define patterns by using a dicitonary for each token. The following patterns matches 'iPhone X':```json[{"TEXT": "iPhone"}, {"TEXT": "X"}]),```Instead of accessing the text you can access many of the features on a token, the following matches "2018 FIFA World Cup":```json[{"IS_DIGIT": True}, {"LOWER": "fifa"}, {"LOWER": "world"}, {"LOWER": "cup"}]```You can access parts of speech and lemmas:```json[{"LEMMA": "love", "POS": "VERB"}, {"POS": "NOUN"}]```And use some Kleene operators (possible values are "!", "?", "*" and "+", where "!" is negation, as in, no match):```json[{"LEMMA": "buy"}, {"POS": "DET", "OP": "?"}, {"POS": "NOUN"}]```<jupyter_code>class SimpleMatcher(object):

    """A simple matcher that you give a list of named pattern."""

    def __init__(self, patterns):
        self.matcher = Matcher(nlp.vocab)
        for pattern_name, pattern in patterns:
            self.matcher.add(pattern_name, [pattern])

    def run(self, sentence):
        doc = nlp(sentence)
        spacy.displacy.render(doc, options={"word_spacing": 30, "distance": 100})
        matches = self.matcher(doc)
        for match_id, start, end in matches:
            pattern_name = self.matcher.vocab.strings[match_id]
            print(f"{pattern_name:12} {start:2} {end:2}  [{doc[start:end]}]")
            
patterns = [

    ("ROOT", [{"DEP": "ROOT"}]),
    ("IPHONE", [{"TEXT": "iPhone"}, 
                {"TEXT": "X"}]),
    ("WORLD_CUP", [{"IS_DIGIT": True},
                   {"LOWER": "fifa"},
                   {"LOWER": "world"},
                   {"LOWER": "cup"},
                   {"IS_PUNCT": True}]),
    ("LOVE_THING", [{"LEMMA": "love", "POS": "VERB"},
                    {"POS": "NOUN"}]),
    ("BUY_STUFF", [{"LEMMA": "buy"},
                   {"POS": "DET", "OP": "?"},
                   {"POS": "NOUN"}])
]

matcher = SimpleMatcher(patterns)
matcher.run("Upcoming iPhone X release date leaked")
matcher.run("2018 FIFA World Cup: France won")
matcher.run("I loved dogs but now I love cats more.")
matcher.run("I bought a smartphone. Now I am buying apps.")<jupyter_output><empty_output><jupyter_text>Matching on phraseshttps://spacy.io/usage/rule-based-matchingphrasematcherIf you need to match large terminology lists, you can also use the PhraseMatcher and create Doc objects instead of token patterns, which is much more efficient overall. The Doc patterns can contain single or multiple tokens.<jupyter_code># With attr="LOWER" you do a full lowercase match, without it we match on the text by default
# Note that you cannot lower case the "D.C" part because it changes the tokenization, so using
# lower is of limited value.
pmatcher = PhraseMatcher(nlp.vocab, attr="LOWER")
terms = ["barack obama", "<NAME>", "washington, D.C."]

# Only run nlp.make_doc to speed things up
patterns = [nlp.make_doc(text) for text in terms]
pmatcher.add("TerminologyList", patterns)

# same here, use make_doc if you do not need anything beyond tokenization 
doc = nlp.make_doc(
    "German Chancellor <NAME> and US President <NAME> "
    "converse in the Oval Office inside the White House in Washington, D.C.")

matches = pmatcher(doc)
for match_id, start, end in matches:
    span = doc[start:end]
    print(start, end, span.text)<jupyter_output>2 4 <NAME>
7 9 Barack Obama
19 22 Washington, D.C.<jupyter_text>Some speed bench marksUsing `make_doc()` gives a speed boost. For an additional boost, you can also use the `nlp.tokenizer.pipe()` method, which will process the texts as a stream. This boost is not as spectacular as bypassing the entire piple line. In one run using the code below the elapsed time (wall time) went from 1.24s to 3.58 ms to 2.70 ms, actual values will vary each time you run it.<jupyter_code>LOTS_OF_TERMS = ["Barack Obama", "<NAME>", "Washington, D.C."] * 100
%%time
patterns = [nlp(term) for term in LOTS_OF_TERMS]
%%time
patterns = [nlp.make_doc(term) for term in LOTS_OF_TERMS]
%%time
patterns = list(nlp.tokenizer.pipe(LOTS_OF_TERMS))<jupyter_output>CPU times: user 2.85 ms, sys: 63 µs, total: 2.91 ms
Wall time: 2.94 ms
{'fraction_non_alphanumeric': 0.1407244785949506, 'fraction_numerical': 0.010537870472008782, 'mean_word_length': 3.9737991266375547, 'pattern_counts': {'":': 31, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '503992', 'license': 'Apache-2.0', 'n_tokens_mistral': 1535, 'n_tokens_neox': 1436, 'n_words': 508, 'path': 'code/week4-spacy/spacy-matching.ipynb', 'repo_name': 'marcverhagen/CS138A', 'size': '4589'}