<jupyter_start><jupyter_text>**Import Spacy** > The default model which is *english-core-web*, for which we load the "en model.<jupyter_code>import spacy
nlp = spacy.load("en")<jupyter_output><empty_output><jupyter_text>**Word Tokenize** > Tokenize words to get the tokens of the text i.e breaking the sentences into words.<jupyter_code>from collections import Counter

text = """Most of the outlay will be at home. No surprise there, either. 
While Samsung has expanded overseas, South Korea 
is still host to most of its factories and research engineers. """

doc = nlp(text)
words = [token.text for token in doc]
print(words)<jupyter_output>['Most', 'of', 'the', 'outlay', 'will', 'be', 'at', 'home', '.', 'No', 'surprise', 'there', ',', 'either', '.', '\n', 'While', 'Samsung', 'has', 'expanded', 'overseas', ',', 'South', 'Korea', '\n', 'is', 'still', 'host', 'to', 'most', 'of', 'its', 'factories', 'and', 'research', 'engineers', '.']<jupyter_text>**Sentence Tokenize** > Tokenize sentences if there are more than 1 sentence i.e breaking the sentences to list of sentence.<jupyter_code>text = """Natural Language Toolkit, or more commonly NLTK, 
is a suite of libraries and programs for symbolic and statistical 
natural language processing (NLP) for English written in the Python 
programming language. It was developed by <NAME> and <NAME> 
in the Department of Computer and Information Science at the University of Pennsylvania."""

text = nlp(text)
list(text.sents)<jupyter_output><empty_output><jupyter_text>**Stop words removal** > Remove irrelevant words using nltk stop words like is,the,a etc from the sentences as they don’t carry any information.<jupyter_code>text = """Most of the outlay will be at home. No surprise there, either. While Samsung has expanded overseas, South Korea is still host to most of its factories and research engineers. """

doc = nlp(text)
# Remove stop words and punctuations.
#
words = [token.text for token in doc if token.is_stop !=True and token.is_punct != True]

print(words)<jupyter_output>['outlay', 'home', 'surprise', 'Samsung', 'expanded', 'overseas', 'South', 'Korea', 'host', 'factories', 'research', 'engineers']<jupyter_text>**Lemmatization** > lemmatize the text so as to get its root form eg: functions,funtionality as function.<jupyter_code>nlp = spacy.load("en")
text = """While Samsung has expanded overseas, South Korea is still host to most of its factories and research engineers. """
doc = nlp(text)
for token in doc:
  print(token, token.lemma_)<jupyter_output>While while
Samsung Samsung
has have
expanded expand
overseas overseas
, ,
South South
Korea Korea
is be
still still
host host
to to
most most
of of
its -PRON-
factories factory
and and
research research
engineers engineer
. .<jupyter_text>**Get word frequency** > Counting the word occurrence using FreqDist library. Word frequency helps us to determine how important the word is in the document by knowing how many times the word is being used.<jupyter_code>nlp = spacy.load("en")
text = """Most of the outlay will be at home. No surprise there, either. While Samsung has expanded overseas, South Korea is still host to most of its factories and research engineers. """

doc = nlp(text)
# remove stopwords and punctuations.
#
words = [token.text for token in doc if token.is_stop != True and token.is_punct != True]
word_freq = Counter(words)
common_words = word_freq.most_common(6)
print(common_words)<jupyter_output>[('outlay', 1), ('home', 1), ('surprise', 1), ('Samsung', 1), ('expanded', 1), ('overseas', 1)]<jupyter_text>**POS tags** > POS tag helps us to know the tags of each word like whether a word is noun, adjective etc.<jupyter_code>nlp = spacy.load("en")
text = """Natural Language Toolkit, or more commonly NLTK."""
text = nlp(text)
for w in text:
  print(w, w.pos_)<jupyter_output>Natural PROPN
Language PROPN
Toolkit PROPN
, PUNCT
or CCONJ
more ADJ
commonly ADV
NLTK NUM
. PUNCT<jupyter_text>**NER-(Named Entity Recognition)** > NER(Named Entity Recognition) is the process of getting the entity names.<jupyter_code>nlp = spacy.load("en")
text = """Most of the outlay will be at home. No surprise there, either. While Samsung has expanded overseas, South Korea is still host to most of its factories and research engineers. 
Meanwhile, Apple based in the USA can differ any much, according to Paul."""

text = nlp(text)
labels = set([w.label_ for w in text.ents])
for label in labels:
  entities = [e.string for e in text.ents if label==e.label_]
  entities = list(set(entities))
  print(label, entities)
# If you want to understand the tags associated with the outputs.
#
spacy.explain("GPE")
from spacy import displacy<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11702127659574468, 'fraction_numerical': 0.001702127659574468, 'mean_word_length': 5.387228260869565, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '214201', 'license': 'MIT', 'n_tokens_mistral': 1483, 'n_tokens_neox': 1438, 'n_words': 589, 'path': 'NLP_with_Spacy.ipynb', 'repo_name': 'Paul-mwaura/Natural-Language-Processing', 'size': '4711'}