<jupyter_start><jupyter_text>We start the previous notebook with this code. Let's give it another spin.<jupyter_code>import spacy
import pandas as pd
from spacy import displacy
nlp = spacy.load("en_core_web_sm")
def has_go_token(doc):
    for t in doc:
        if t.lower_ in ['go', 'golang', 'python', 'ruby', 'objective-c']:
            if t.pos_ != 'VERB':
                return True
    return False
doc = nlp("i am an iOS dev and I like to code in objective-c")
[t for t in doc]<jupyter_output><empty_output><jupyter_text>Our current approach won't work for languages with more than one token. So instead I'll need to use matchers instead. There's a bunch of them applied below.<jupyter_code>from spacy.matcher import Matcher
obj_c_pattern1 = [{'LOWER': 'objective'},
                  {'IS_PUNCT': True, 'OP': '?'},
                  {'LOWER': 'c'}]

obj_c_pattern2 = [{'LOWER': 'objectivec'}]

golang_pattern1 = [{'LOWER': 'golang'}] 
golang_pattern2 = [{'LOWER': 'go', 
                    'POS': {'NOT_IN': ['VERB']}}]

python_pattern = [{'LOWER': 'python'}]
ruby_pattern   = [{'LOWER': 'ruby'}]
js_pattern     = [{'LOWER': {'IN': ['js', 'javascript']}}]
matcher = Matcher(nlp.vocab, validate=True)
matcher.add("OBJ_C_LANG", None, obj_c_pattern1, obj_c_pattern2)
matcher.add("PYTHON_LANG", None, python_pattern)
matcher.add("GO_LANG", None, golang_pattern1, golang_pattern2)
matcher.add("JS_LANG", None, js_pattern)
matcher.add("RUBY_LANG", None, ruby_pattern)
doc = nlp("I am an iOS dev who codes in both python, go/golang as well as objective-c")
for match_id, start, end in matcher(doc):
    print(doc[start: end])
doc = nlp("I've done some js and ruby and go programming")
for match_id, start, end in matcher(doc):
    print(doc[start: end])<jupyter_output>js
ruby<jupyter_text>BenchmarkingOur current approach works, but it would be good to confirm this with data. I'll do a soft benchmark; I'll check for the occurence of a string, like "objective", and I'll see which instances my matcher does not pick up. If there's stuff that I am missing I should get a pretty clear picture of it.<jupyter_code>import pandas as pd

df = (pd.read_csv("Questions.csv", nrows=1_000_000, 
                  encoding="ISO-8859-1", usecols=['Title', 'Id']))
titles = (_ for _ in df['Title'] if "python" in _.lower())
for i in range(200):
    doc = nlp(next(titles))
    if len(matcher(doc)) == 0:
        print(doc)<jupyter_output>mod_python/MySQL error on INSERT with a lot of data: "OperationalError: (2006, 'MySQL server has gone away')"
Running subversion under apache and mod_python
What's the best way to embed IronPython inside my C# App?
How to set the PYTHONPATH in Emacs?
wxPython wxDC object from win32gui.GetDC
Need skeleton code to call Excel VBA from PythonWin
Questions for python->scheme conversion
wxPython and sharing objects between windows
Django on IronPython
IronPython Webframework
A SuggestBox for wxPython?
Intercepting Method Access on the Host Program of IronPython
Is there anything like IPython / IRB for Perl?
{'fraction_non_alphanumeric': 0.1165623971024037, 'fraction_numerical': 0.009878169245966415, 'mean_word_length': 4.386524822695035, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '329555', 'license': 'MIT', 'n_tokens_mistral': 1053, 'n_tokens_neox': 984, 'n_words': 357, 'path': '02-match-more-languages.ipynb', 'repo_name': 'alanderex/spacy-youtube-material', 'size': '3037'}