<jupyter_start><jupyter_text>NER v.2 - Training for Custom Entities on SpaCyThe previous notebook handled the basics of using SpaCy's built-in entity recognition model to identify entities in any random string of words. This post will handle the creation of custom entities for any kind of unique task you want to use it for. At [Quiddity Solutions](https://www.quidditysolutions.com/), we use Statistical Language Modeling to identify specific, useful medical terminology and figures as part of our Treatment Matrix. This kind of useful information isn't readily available in a neat package off-the-shelf online. Therefore, we need to design our own custom model. Because a lot of NLP is very experimental and not yet mainstream, we began some of our experiments using [SpaCy](https://spacy.io/), a neat package designed by [Explosion](https://explosion.ai/), a German software company specializing in AI-related packages. Some of the initial research on what a custom, self-improving statistical language model to identify named entities would eventually look like was inspired by SpaCy's approach. You can see what we're currently working on at the Mercury [project website](https://ncaldwell17.github.io/Mercury/). Before I begin, make sure that this is the right place for you to start. What does your task data look like? What do you want your model to achieve? Do you have a finite number of examples you want it to look like? Is there a clear, structured pattern in the data that can be used to identify the entities? If that sounds like your data, you might be better suited to use __rule-based matching__ instead of training a model from scratch. You can find more information on that [here](https://spacy.io/usage/rule-based-matching). If your data is complex, you only have some limited examples, you want the system to be able to generalize across a variety of tasks, and instead of a clear, structured pattern, there are only clues in the data that you can use to train - you definitely want to train your own model. That's what this post is all about. I've found the best approach is a hybrid model, one that can handle both rules and context clues to achieve good accuracy. SpaCy's recently come out with a new option that lets you do this, which you can read about [here](https://spacy.io/usage/rule-based-matchingentityruler). In future posts, when I cover some of the precision pitfalls we encountered in designing Quiddity Solutions' flagship product, we'll start to get into the design and implementation of such a hybrid model.  I want to build a custom model, how do I start?Say you have a task like Quiddity's - you want to build your custom model but don't know where to begin. Let's start by installing SpaCy - they're the preëminent off-the-shelf Python-based NER package out there. One unique thing about Spacy is that you'll be using a pretrained model for some of the initial work, which means the installation is a bit different from what you'll be used to when working with Python modules. In order to get it just right, you need to make sure that the model SpaCy has for your language is specifically downloaded and installed to the right directory. I'll detail the exact terminal command in the cell below, but its essential that you do this in the way described because it can cause all kinds of headaches otherwise. If you want more information on the installation process, you can go [here](https://spacy.io/usage/models). You needed to do this to achieve the results of the previous post, but I like reiterating things when I write code tutorials - I find it's hard to keep track of every little thing as we go, and it's helpful for those who skipped the first one. __NOTE:__ This tutorial assumes that you have already installed SpaCy on your computer - if you haven't, just input the following terminal command if you have pip enabled.`pip install spacy`<jupyter_code># ! indicates a terminal command for Jupyter notebooks
# en_core_web_sm is the name of SpaCy's English model
!python -m spacy download en_core_web_sm
# python -m allows you to use the proceeding module as  
#    if you had typed out its entire path.<jupyter_output>Requirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in /Users/noahcg/anaconda3/lib/python3.6/site-packages (2.0.0)
[33mYou are using pip version 18.0, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.[0m

[93m    Linking successful[0m
    /Users/noahcg/anaconda3/lib/python3.6/site-packages/en_core_web_sm -->
    /Users/noahcg/anaconda3/lib/python3.6/site-packages/spacy/data/en_core_web_sm

    You can now load the model via spacy.load('en_core_web_sm')<jupyter_text>Notice that there's a little printout telling you that you can load the model via the `spacy.load('en_core_web_sm')` method and argument. That means that you still have to call it within whatever program you write.<jupyter_code>import spacy
# load the pre-trained model 
nlp = spacy.load("en_core_web_sm")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.057666345226615234, 'fraction_numerical': 0.007714561234329798, 'mean_word_length': 4.582346609257266, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '490116', 'license': 'Apache-2.0', 'n_tokens_mistral': 1420, 'n_tokens_neox': 1332, 'n_words': 743, 'path': 'notes/.ipynb_checkpoints/Training Model SpaCy Commandline-checkpoint.ipynb', 'repo_name': 'ncaldwell17/Mercury', 'size': '5185'}