<jupyter_start><jupyter_text>![license_header_logo](../../../images/license_header_logo.png)> **Copyright (c) 2021 <EMAIL>ifAI Sdn. Bhd.**This program is part of OSRFramework. You can redistribute it and/or modifyit under the terms of the GNU Affero General Public License as published bythe Free Software Foundation, either version 3 of the License, or(at your option) any later version.This program is distributed in the hope that it will be usefulbut WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See theGNU Affero General Public License for more details.You should have received a copy of the GNU Affero General Public Licensealong with this program.  If not, see .  IntroductionThis notebook will introduce you the basic of spaCy and some simple implementations of spaCy. SpaCy is widely used in NLP tasks.  Notebook Content* [What’s SpaCy?](What’s-spaCy?)* [Features](Features)* [Installation](Installation)* [Linguistic Annotations](Linguistic-Annotations)* [Tokenization](Tokenization)* [Part-of-speech Tags and Dependencies](Part-of-speech-Tags-and-Dependencies)    * [Named Entities](Named-Entities)* [Word Vectors and Similarity](Word-Vectors-and-Similarity)* [Vocab, Hashes and Lexemes](Vocab,-Hashes-and-Lexemes)   What’s spaCy?    spaCy is a free, **open-source library** for **advanced Natural Language Processing (NLP)** in Python.If you’re working with a lot of text, you’ll eventually want to know more about it. For example, what’s it about? What do the words mean in context? Who is doing what to whom? What companies and products are mentioned? Which texts are similar to each other?spaCy is designed specifically for **production use** and helps you build applications that process and “understand” large volumes of text. It can be used to build **information extraction** or **natural language understanding systems**, or to **pre-process text for deep learning**.  Features    In this notebook, you’ll come across mentions of spaCy’s features and capabilities. Some of them refer to linguistic concepts, while others are related to more general machine learning functionality.![todo](../../../images/todo.png)  InstallationspaCy’s trained pipelines can be installed as Python packages. This means that they’re a component of your application, just like any other module. They’re versioned and can be defined as a dependency in your requirements.txt. Trained pipelines can be installed from a download URL or a local directory, manually or via pip. Their data can be located anywhere on your file system.Installation link: https://spacy.io/usage/models![installation](../../../images/installation.png)  Linguistic AnnotationsspaCy provides a variety of linguistic annotations to give you **insights into a text’s grammatical structure**. This includes the word types, like the parts of speech, and how the words are related to each other. For example, if you’re analyzing text, it makes a huge difference whether a noun is the subject of a sentence, or the object – or whether “google” is used as a verb, or refers to the website or company in a specific context.<jupyter_code>import spacy

# !python -m spacy download en_core_web_sm

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for token in doc:
    print(token.text, token.pos_, token.dep_)<jupyter_output>Apple PROPN nsubj
is AUX aux
looking VERB ROOT
at ADP prep
buying VERB pcomp
U.K. PROPN compound
startup NOUN dobj
for ADP prep
$ SYM quantmod
1 NUM compound
billion NUM pobj<jupyter_text>Even though a `Doc` is processed – e.g. split into individual words and annotated – it still holds **all information of the original text**, like whitespace characters. You can always get the offset of a token into the original string, or reconstruct the original by joining the tokens and their trailing whitespace. This way, you’ll never lose any information when processing text with spaCy.  TokenizationDuring processing, spaCy first **tokenizes** the text, i.e. segments it into words, punctuation and so on. This is done by applying rules specific to each language. For example, punctuation at the end of a sentence should be split off – whereas “U.K.” should remain one token. Each `Doc` consists of individual tokens, and we can iterate over them:<jupyter_code>import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for token in doc:
    print(token.text)<jupyter_output>Apple
is
looking
at
buying
U.K.
startup
for
$
1
billion<jupyter_text>![tokenization](../../../images/tokenization.png)First, the raw text is split on whitespace characters, similar to `text.split(' ')`. Then, the tokenizer processes the text from left to right. On each substring, it performs two checks:    1. Does the substring match a tokenizer exception rule? For example, “don’t” does not contain whitespace, but should be split into two tokens, “do” and “n’t”, while “U.K.” should always remain one token.        2. Can a prefix, suffix or infix be split off? For example punctuation like commas, periods, hyphens or quotes.    If there’s a match, the rule is applied and the tokenizer continues its loop, starting with the newly split substrings. This way, spaCy can split **complex, nested tokens** like combinations of abbreviations and multiple punctuation marks.![tokenization](../../../images/tokenization_2.png)  Part-of-speech Tags and DependenciesAfter tokenization, spaCy can **parse** and **tag** a given `Doc`. This is where the trained pipeline and its statistical models come in, which enable spaCy to **make predictions** of which tag or label most likely applies in this context. A trained component includes binary data that is produced by showing a system enough examples for it to make predictions that generalize across the language – for example, a word following “the” in English is most likely a noun.Linguistic annotations are available as Token attributes. Like many NLP libraries, spaCy **encodes all strings to hash values** to reduce memory usage and improve efficiency. So to get the readable string representation of an attribute, we need to add an underscore `_` to its name:<jupyter_code>import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,
            token.shape_, token.is_alpha, token.is_stop)<jupyter_output>Apple Apple PROPN NNP nsubj Xxxxx True False
is be AUX VBZ aux xx True True
looking look VERB VBG ROOT xxxx True False
at at ADP IN prep xx True True
buying buy VERB VBG pcomp xxxx True False
U.K. U.K. PROPN NNP compound X.X. False False
startup startup NOUN NN dobj xxxx True False
for for ADP IN prep xxx True True
$ $ SYM $ quantmod $ False False
1 1 NUM CD compound d False False
billion billion NUM CD pobj xxxx True False<jupyter_text>![POS](../../../images/pos.png)  Named EntitiesA named entity is a “real-world object” that’s assigned a name – for example, a person, a country, a product or a book title. spaCy can **recognize various types of named entities in a document, by asking the model for a prediction**. Because models are statistical and strongly depend on the examples they were trained on, this doesn’t always work *perfectly* and might need some tuning later, depending on your use case.Named entities are available as the `ents` property of a `Doc`:<jupyter_code>import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)<jupyter_output>Apple 0 5 ORG
U.K. 27 31 GPE
$1 billion 44 54 MONEY<jupyter_text>![Named Entity](../../../images/named_entity.png)Here’s what our example sentence and its named entities look like:![Named Entity](../../../images/named_entity_2.png)  Word Vectors and SimilaritySimilarity is determined by comparing **word vectors** or **“word embeddings”**, multi-dimensional meaning representations of a word. Word vectors can be generated using an algorithm like **word2vec** and usually look like this:array([2.02280000e-01,  -7.66180009e-02,   3.70319992e-01, 3.28450017e-02,  -4.19569999e-01,   7.20689967e-02, -3.74760002e-01,   5.74599989e-02,  -1.24009997e-02, 5.29489994e-01,  -5.23800015e-01,  -1.97710007e-01, -3.41470003e-01,   5.33169985e-01,  -2.53309999e-02, 1.73800007e-01,   1.67720005e-01,   8.39839995e-01, 5.51070012e-02,   1.05470002e-01,   3.78719985e-01, 2.42750004e-01,   1.47449998e-02,   5.59509993e-01, 1.25210002e-01,  -6.75960004e-01,   3.58420014e-01,  ... and so on ... 3.66849989e-01,   2.52470002e-03,  -6.40089989e-01, -2.97650009e-01,   7.89430022e-01,   3.31680000e-01, -1.19659996e+00,  -4.71559986e-02,   5.31750023e-01,] dtype=float32) > To make them compact and fast, spaCy’s small **pipeline packages** (all packages that end in `sm`) **don’t ship with word vectors**, and only include context-sensitive **tensors**. This means you can still use the `similarity()` methods to compare documents, spans and tokens – but the result won’t be as good, and individual tokens won’t have any vectors assigned. So in order to use real word vectors, you need to download a larger pipeline package:> Small pipeline: python -m spacy download en_core_web_sm> Large pipeline: python -m spacy download en_core_web_lg Pipeline packages that come with built-in word vectors make them available as the `Token.vector attribute`. `Doc.vector` and `Span.vector` will default to an average of their token vectors. You can also check if a token has a vector assigned, and get the **L2 norm**, which can be used to normalize vectors.<jupyter_code>import spacy

# !python -m spacy download en_core_web_lg

# To install en_core_web_md use > python -m spacy download en_core_web_md
nlp = spacy.load("en_core_web_lg")
tokens = nlp("dog cat banana afskfsd")

for token in tokens:
    print(token.text, token.has_vector, token.vector_norm, token.is_oov)<jupyter_output>dog True 7.0336733 False
cat True 6.6808186 False
banana True 6.700014 False
afskfsd False 0.0 True<jupyter_text>The words “dog”, “cat” and “banana” are all pretty common in English, so they’re part of the pipeline’s vocabulary, and come with a vector. The word “afskfsd” on the other hand is a lot less common and out-of-vocabulary – so its vector representation consists of 300 dimensions of `0`, which means it’s practically nonexistent. If your application will benefit from a **large vocabulary** with more vectors, you should consider using one of the larger pipeline packages or loading in a full vector package, for example, `en_core_web_lg`, which includes 685k unique vectors.spaCy is able to compare two objects, and make a prediction of **how similar they are**. Predicting similarity is useful for building recommendation systems or flagging duplicates. For example, you can suggest a user content that’s similar to what they’re currently looking at, or label a support ticket as a duplicate if it’s very similar to an already existing one.Each `Doc`, `Span`, `Token` and `Lexeme` comes with a `.similarity` method that lets you compare it with another object, and determine the similarity. Of course similarity is always subjective – whether two words, spans or documents are similar really depends on how you’re looking at it. spaCy’s similarity implementation usually assumes a pretty general-purpose definition of similarity.<jupyter_code>import spacy

nlp = spacy.load("en_core_web_lg")  # make sure to use larger package!
doc1 = nlp("I like salty fries and hamburgers.")
doc2 = nlp("Fast food tastes very good.")

# Similarity of two documents
print(doc1, "<->", doc2, doc1.similarity(doc2))
# Similarity of tokens and spans
french_fries = doc1[2:4]
burgers = doc1[5]
print(french_fries, "<->", burgers, french_fries.similarity(burgers))<jupyter_output>I like salty fries and hamburgers. <-> Fast food tastes very good. 0.7687607012190486
salty fries <-> hamburgers 0.6949788<jupyter_text>Computing similarity scores can be helpful in many situations, but it’s also important to maintain **realistic expectations** about what information it can provide. Words can be related to each other in many ways, so a single “similarity” score will always be a **mix of different signals**, and vectors trained on different data can produce very different results that may not be useful for your purpose. Here are some important considerations to keep in mind:> + There’s no objective definition of similarity. Whether “I like burgers” and “I like pasta” is similar **depends on your application**. Both talk about food preferences, which makes them very similar – but if you’re analyzing mentions of food, those sentences are pretty dissimilar, because they talk about very different foods.> + The similarity of `Doc` and `Span` objects defaults to the **average** of the token vectors. This means that the vector for “fast food” is the average of the vectors for “fast” and “food”, which isn’t necessarily representative of the phrase “fast food”.> + Vector averaging means that the vector of multiple tokens is **insensitive to the order of the words**. Two documents expressing the same meaning with dissimilar wording will return a lower similarity score than two documents that happen to contain the same words while expressing different meanings.  Vocab, Hashes and LexemesWhenever possible, spaCy tries to store data in a vocabulary, the `Vocab`, that will be **shared by multiple documents**. To save memory, spaCy also encodes all strings to **hash values** – in this case for example, “coffee” has the hash `3197928453018144401`. Entity labels like “ORG” and part-of-speech tags like “VERB” are also encoded. Internally, spaCy only “speaks” in hash values.![Vocab Hashing](../../../images/hashing.png)If you process lots of documents containing the word “coffee” in all kinds of different contexts, storing the exact string “coffee” every time would take up way too much space. So instead, spaCy hashes the string and stores it in the `StringStore`. You can think of the `StringStore` as a **lookup table that works in both directions** – you can look up a string to get its hash, or a hash to get its string:<jupyter_code>import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("I love coffee")
print(doc.vocab.strings["coffee"])  # 3197928453018144401
print(doc.vocab.strings[3197928453018144401])  # 'coffee'<jupyter_output>3197928453018144401
coffee<jupyter_text>Now that all strings are encoded, the entries in the vocabulary **don’t need to include the word text** themselves. Instead, they can look it up in the StringStore via its hash value. Each entry in the vocabulary, also called `Lexeme`, contains the **context-independent** information about a word. For example, no matter if “love” is used as a verb or a noun in some context, its spelling and whether it consists of alphabetic characters won’t ever change. Its hash value will also always be the same.<jupyter_code>import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("I love coffee")
for word in doc:
    lexeme = doc.vocab[word.text]
    print(lexeme.text, lexeme.orth, lexeme.shape_, lexeme.prefix_, lexeme.suffix_,
            lexeme.is_alpha, lexeme.is_digit, lexeme.is_title, lexeme.lang_)<jupyter_output>I 4690420944186131903 X I I True False True en
love 3702023516439754181 xxxx l ove True False False en
coffee 3197928453018144401 xxxx c fee True False False en<jupyter_text>The mapping of words to hashes doesn’t depend on any state. To make sure each value is unique, spaCy uses a hash function to calculate the hash **based on the word string**. This also means that the hash for “coffee” will always be the same, no matter which pipeline you’re using or how you’ve configured spaCy.However, hashes **cannot be reversed** and there’s no way to resolve `3197928453018144401` back to “coffee”. All spaCy can do is look it up in the vocabulary. That’s why you always need to make sure all objects you create have access to the same vocabulary. If they don’t, spaCy might not be able to find the strings it needs.<jupyter_code>import spacy
from spacy.tokens import Doc
from spacy.vocab import Vocab

nlp = spacy.load("en_core_web_sm")
doc = nlp("I love coffee")  # Original Doc
print(doc.vocab.strings["coffee"])  # 3197928453018144401
print(doc.vocab.strings[3197928453018144401])  # 'coffee' 👍

empty_doc = Doc(Vocab())  # New Doc with empty Vocab
# empty_doc.vocab.strings[3197928453018144401] will raise an error :(

empty_doc.vocab.strings.add("coffee")  # Add "coffee" and generate hash
print(empty_doc.vocab.strings[3197928453018144401])  # 'coffee' 👍

new_doc = Doc(doc.vocab)  # Create new doc with first doc's vocab
print(new_doc.vocab.strings[3197928453018144401])  # 'coffee' 👍<jupyter_output>3197928453018144401
coffee
coffee
coffee
{'fraction_non_alphanumeric': 0.0797263828281637, 'fraction_numerical': 0.04493454416794433, 'mean_word_length': 4.788054607508532, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '196270', 'license': 'Apache-2.0', 'n_tokens_mistral': 5460, 'n_tokens_neox': 4820, 'n_words': 2282, 'path': 'nlp-labs/Day_02/NLP Spacy/Spacy Tutorials.ipynb', 'repo_name': 'synthethic04/NLP-Traininglabs', 'size': '16955'}