<jupyter_start><jupyter_text>Getting Started with NLPSome common applications of NLP:* Text Summarization* Text Tagging (topic tagging)* Named Entity Recognition* Chatbot* Speech Recognition Some terms:* Phonetics/phonology:The study of linguistic sounds and their relations to written words* Morphology:The study of internal structures of words/composition of words* Syntax:The study of the structural relationships among words in a sentence* Semantics:The study of the meaning of words and how these combine to form the meaning of sentences* Pragmatics:Situational use of language sentences* Discourse:A linguistic unit that is larger than a single sentence (context)* Tokenization:A step which splits longer strings of text into smaller pieces, or tokens. Larger chunks of text can be tokenized into sentences, sentences can be tokenized into words.* Normalization:Before further processing, text needs to be normalized. Normalization generally refers to a series of related tasks meant to put all text on a level playing field: converting all text to the same case (upper or lower), removing punctuation, expanding contractions, converting numbers to their word equivalents, and so on. Normalization puts all words on equal footing, and allows processing to proceed uniformly.* Stemming:is the process of eliminating affixes (suffixed, prefixes, infixes, postfix, circumfixes) from a word in order to obtain a word stem. ex: running → run* Lemmatization:lemmatization is able to capture canonical forms based on a word's lemma.For example, stemming the word "better" would fail to return its citation form (another word for lemma); however, lemmatization would result in the following:better → good* Corpus:(literally Latin for body) refers to a collection of texts. Such collections may be formed of a single language of texts, or can span multiple languages.* Stop words:words which are filtered out before further processing of text, since these words contribute little to overall meaning, given that they are generally the most common words in a language. For instance, "the," "and," and "a," while all required words in a particular passage, don't generally contribute greatly to one's understanding of content.* Parts-of-speech (POS) Tagging:POS tagging consists of assigning a category tag to the tokenized parts of a sentence. The most popular POS tagging would be identifying words as nouns, verbs, adjectives, etc.* Bag of Words:representation model used to simplify the contents of a selection of text. The bag of words model omits grammar and word order, but is interested in the number of occurrences of words within the text. The ultimate representation of the text selection is that of a bag of words.  Ex:  "Well, well, well," said John.  "There, there," said James. "There, there."    The resulting bag of words representation as a dictionary:     {      'well': 3,      'said': 2,      'john': 1,      'there': 4,      'james': 1     }          * N-grams:representation model for simplifying text selection contents. As opposed to the orderless representation of bag of words, n-grams modeling is interested in preserving contiguous sequences of N items from the text selection.  An example of trigram (3-gram) model of the second sentence of the above example ("There, there," said James.     "There, there.") appears as a list representation below:   [      "there there said",      "there said james",      "said james there",      "james there there",   ]   * Statistical Language Modeling:the process of building a statistical language model which is meant to provide an estimate of a natural language. For a sequence of input words, the model would assign a probability to the entire sequence, which contributes to the estimated likelihood of various possible sequences. This can be especially useful for NLP applications which generate text.* Syntactic Analysis:Also referred to as parsing, syntactic analysis is the task of analyzing strings as symbols, and ensuring their conformance to a established set of grammatical rules.* Semantic Analysis:Also known as meaning generation, semantic analysis is interested in determining the meaning of text selections (either character or word sequences). After an input selection of text is read and parsed (analyzed syntactically), the text selection can then be interpreted for meaning. Simply put, syntactic analysis is concerned with what words a text selection was made up of, while semantic analysis wants to know what the collection of words actually means. * Sentiment analysis:the process of evaluating and determining the sentiment captured in a selection of text, with sentiment defined as feeling or emotion.   * Entity recognition:is the process used to classify multiple entities found in a text in predefined categories, such as a person,objects, location, organizations, dates, events, etc.* Word vector:refers to the mapping of the words or phrases from vocabulary to a vector of real numbers.   Virtual EnvironmentThe main purpose of Python virtual environments is to create an isolated environment for Python projects. This means that each project can have its own dependencies, regardless of what dependencies every other project has.<jupyter_code># In python 2
!pip install virtualenv
# If you are using Python 3, then you should already have the venv module from the standard library installed.
# To create a new virtual environment inside the directory
# python3 -m venv (env_name)
!python3 -m venv handsonnlp
# On Debian/Ubuntu systems, you need to install the python3-venv package using the following command.
# sudo apt-get install python3-venv
# since it's not installed by default<jupyter_output><empty_output><jupyter_text>Activate / Deactivate your environment<jupyter_code>!source handsonnlp/bin/activate
!deactivate<jupyter_output><empty_output><jupyter_text>List Installed Packages With Pip<jupyter_code>!pip list<jupyter_output><empty_output><jupyter_text>Create requirements.txt<jupyter_code>!pip freeze > requirements.txt<jupyter_output><empty_output><jupyter_text>Natural Language Processing Libraries 1. NLTK  (www.nltk.org/)Most common package you will encounter working with corpora, categorizing text, analyzing linguistic structure, and more.It was developed by <NAME> and <NAME> in the Department of Computer and Information Science at the University of Pennsylvania.<jupyter_code># install NLTK with pip
# activate your env first
#!source handsonnlp/bin/activate
!pip install nltk<jupyter_output><empty_output><jupyter_text>Tokenization in NLTK<jupyter_code>import nltk

# Tokenization
sent_ = "I am almost dead this time"
tokens_ = nltk.word_tokenize(sent_)
# So we need to install punkt
nltk.download('punkt')
# let's run word_tokenize() now
sents = "I am almost dead this time"
tokens = nltk.word_tokenize(sents)
tokens<jupyter_output><empty_output><jupyter_text>Getting a synonym of a word in NLTK<jupyter_code># Getting a synonym of a word.
# you need to download wordnet first
nltk.download('wordnet')
from nltk.corpus import wordnet
words = wordnet.synsets("spectacular")
print(words)
# Access definition of first synonym
words[0].definition()
# Access definition of each synonym
for word in words:
    print(word, ":", word.definition())<jupyter_output>Synset('spectacular.n.01') : a lavishly produced performance
Synset('dramatic.s.02') : sensational in appearance or thrilling in effect
Synset('spectacular.s.02') : characteristic of spectacles or drama
Synset('outstanding.s.02') : having a quality that thrusts itself into attention<jupyter_text>Stemming in NLTK<jupyter_code>from nltk.stem import PorterStemmer
stemmer = PorterStemmer() # Create the stemmer object
stemmer.stem("decreases")
stemmer.stem("running")
stemmer.stem("cats")
stemmer.stem("singing")
stemmer.stem("shelves")
stemmer.stem("better")<jupyter_output><empty_output><jupyter_text>Lemmatization in NLTK<jupyter_code>from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer() # Create the Lemmatizer object
lemmatizer.lemmatize("decreases")
lemmatizer.lemmatize("rocks")
lemmatizer.lemmatize("corpora")
lemmatizer.lemmatize("better")
# What?! better is supposed to lemmatized to good 
# That's right.
# But, one major difference with stemming is that lemmatize takes a part of speech parameter,
# “pos” If not supplied, the default is “noun.”
# so above it considers better to be a Noun not Adjective
# to fix that supply pos parameter like so
lemmatizer.lemmatize("better", pos="a")
import nltk
sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
tokens = nltk.word_tokenize(sentence)
tokens<jupyter_output><empty_output><jupyter_text>POS tagging in NLTK<jupyter_code>tagged = nltk.pos_tag(tokens)
# so we need to download the "averaged_perceptron_tagger" from NLTK
nltk.download('averaged_perceptron_tagger')
tagged = nltk.pos_tag(tokens)
tagged
# list all tags
nltk.help.upenn_tagset()
nltk.download('tagsets')
nltk.help.upenn_tagset()
type(tagged)
tagged[0:6]<jupyter_output><empty_output><jupyter_text>Identify named entities in NLTK<jupyter_code>entities = nltk.chunk.ne_chunk(tagged)
# download maxent_ne_chunker package
nltk.download('maxent_ne_chunker')
entities = nltk.chunk.ne_chunk(tagged)
# download words package
nltk.download('words')
entities = nltk.chunk.ne_chunk(tagged)
entities
print(entities)<jupyter_output>(S
  At/IN
  eight/CD
  o'clock/NN
  on/IN
  Thursday/NNP
  morning/NN
  .../:
  (PERSON Arthur/NNP)
  did/VBD
  n't/RB
  feel/VB
  very/RB
  good/JJ
  ./.)<jupyter_text>Display a parse tree in NLTK<jupyter_code>from nltk.corpus import treebank
t = treebank.parsed_sents('wsj_0001.mrg')[0]
t.draw()
# so we need to download treebank package
nltk.download('treebank')
from nltk.corpus import treebank
t = treebank.parsed_sents('wsj_0001.mrg')[0]
t.draw()<jupyter_output><empty_output><jupyter_text>2. TextBlob (http://textblob.readthedocs.io/en/dev/index.html)It provides a simple API for diving deep into common NLP tasks, such as part-of-speech tagging, nounphrase extraction, sentiment analysis, classification, and much more.TextBlob stands on the giant shoulders of NLTK and pattern, and plays nicely with both.<jupyter_code># install TextBlob
!pip install -U textblob
# or Install it with python3
!python3 -m pip install -U textblob<jupyter_output>/bin/bash: pip: command not found
Collecting textblob
  Using cached textblob-0.15.3-py2.py3-none-any.whl (636 kB)
Collecting nltk>=3.1
  Using cached nltk-3.6.2-py3-none-any.whl (1.5 MB)
Collecting joblib
  Using cached joblib-1.0.1-py3-none-any.whl (303 kB)
Requirement already satisfied, skipping upgrade: click in /usr/lib/python3/dist-packages (from nltk>=3.1->textblob) (7.0)
Collecting tqdm
  Using cached tqdm-4.60.0-py2.py3-none-any.whl (75 kB)
Collecting regex
  Using cached regex-2021.4.4-cp38-cp38-manylinux2014_x86_64.whl (733 kB)
Installing collected packages: joblib, tqdm, regex, nltk, textblob
[33m  WARNING: The script tqdm is installed in '/home/muhammad/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.[0m
[33m  WARNING: The script nltk is installed in '/home/muhammad/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress t[...]<jupyter_text>POS Tagging in TextBlob<jupyter_code>from textblob import TextBlob

text = '''
The titular threat of The Blob has always struck me as the ultimate movie
monster: an insatiably hungry, amoeba-like mass able to penetrate
virtually any safeguard, capable of--as a doomed doctor chillingly
describes it--"assimilating flesh on contact.
Snide comparisons to gelatin be damned, it's a concept with the most
devastating of potential consequences, not unlike the grey goo scenario
proposed by technological theorists fearful of
artificial intelligence run rampant.
'''

blob = TextBlob(text)
blob.tags<jupyter_output><empty_output><jupyter_text>Noun phrase extraction in TextBlob<jupyter_code>blob.noun_phrases
# need to install brown package
import nltk
nltk.download('brown')
blob.noun_phrases
blob.noun_phrases[0:3]
blob.sentences
# get first sentence
blob.sentences[0].string<jupyter_output><empty_output><jupyter_text>Sentiment analysis with TextBlobPolarity defines negativity or positivity in the sentence, whereassubjectivity implies whether the sentence discusses something vaguely orwith complete surety.<jupyter_code>for sentence in blob.sentences:
    print("\n",sentence.sentiment)
    print("\n polarity:",sentence.sentiment.polarity)
    print("\n subjectivity:",sentence.sentiment.subjectivity)<jupyter_output>Sentiment(polarity=0.06000000000000001, subjectivity=0.605)

 polarity: 0.06000000000000001

 subjectivity: 0.605

 Sentiment(polarity=-0.34166666666666673, subjectivity=0.7666666666666666)

 polarity: -0.34166666666666673

 subjectivity: 0.7666666666666666<jupyter_text>Correct spelling errors with TextBlob<jupyter_code>sent = TextBlob("I thinkk tha model needs to be trainned more!")
sent.correct()<jupyter_output><empty_output><jupyter_text>Language translation with TextBlob<jupyter_code>sent_fr = TextBlob(u"Voulez-vous apprendre le français?")
sent_fr.translate(from_lang='fr', to='en')
sent_ar = TextBlob(u"هل تريد تعلم العربية?")
sent_ar.translate(from_lang='ar', to='en')<jupyter_output><empty_output><jupyter_text>Text Classification using TextBlob (https://textblob.readthedocs.io/en/dev/api_reference.htmlmodule-textblob.classifiers)Text can be classified into various classes, such as positive and negative.<jupyter_code>from textblob import TextBlob
from textblob.classifiers import NaiveBayesClassifier
data = [
('I love my country.', 'pos'),
('This is an amazing place!', 'pos'),
('I do not like the smell of this place.', 'neg'),
('I do not like this restaurant', 'neg'),
('I am tired of hearing your nonsense.', 'neg'),
("I always aspire to be like him", 'pos'),
("It's a horrible performance.", "neg"),
("My boss is horrible.", "neg")
]
model = NaiveBayesClassifier(data)
model.classify("It's an awesome place!")
blob = TextBlob("The game is good. But the hangover is horrible.", classifier=model)
for s in blob.sentences:
    print(s)
    print(s.classify())<jupyter_output>The game is good.
neg
But the hangover is horrible.
neg<jupyter_text>3. SpaCy (https://spacy.io/)provides very fast and accurate syntacticanalysis (the fastest of any library released) and also offers named entityrecognition and ready access to word vectors. It is written in Cythonlanguage and contains a wide variety of trained models on languagevocabularies, syntaxes, word-to-vector transformations, and entitiesrecognition.<jupyter_code># install Spacy
!pip install -U spacy
# or Install it with python3
!python3 -m pip install -U spacy
import spacy
nlp = spacy.load("en")
# To fix that error run the command
!python3 -m spacy download en
nlp = spacy.load("en_core_web_sm")
william_wikidef_str = """William was the son of King <NAME> and <NAME> Russia.
On the abdication of his grandfather <NAME> in 1840, he became the Prince of Orange.
On the death of his father in 1849, he succeeded as king of the
Netherlands. William married his cousin <NAME>
in 1839 and they had three sons, William, Maurice, and
Alexander, all of whom predeceased him. """

william = nlp(william_wikidef_str)
william.ents
[ (i, i.label_, i.label) for i in william.ents]
# let's visualize the recognized entities
import spacy
from spacy import displacy
# if you want to run a web server to serve the visualization
# displacy.serve(william, style="ent") 
# or if you are running jupyter notebook use render() instead
displacy.render(william, style="ent")<jupyter_output><empty_output><jupyter_text>Noun Phrase extraction with Spacy<jupyter_code>sent = nlp('The book deals with NLP')
for noun_ in sent.noun_chunks:
    print("\n",type(noun_))
    print(noun_.text)
    print('---')
    print(noun_.root.dep_)
    print('---')
    print("head:",noun_.root.head.text)<jupyter_output><class 'spacy.tokens.span.Span'>
The book
---
nsubj
---
head: deals

 <class 'spacy.tokens.span.Span'>
NLP
---
pobj
---
head: with<jupyter_text>4. Gensim (https://pypi.python.org/pypi/gensim or https://radimrehurek.com/gensim/) It is used primarily for topic modeling and document similarity.Gensim offers LDA (latent dirichlet allocation—a generative statisticalmodel that allows sets of observations to be explained by unobservedgroups that explain why some parts of the data are similar).For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word's presence is attributable to one of the document's topics. LDA is an example of a topic model and belongs to the machine learning toolbox and in wider sense to the artificial intelligence toolbox. Read more! [about LDA topic model](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation)<jupyter_code># install Gensim
!pip install --upgrade gensim
# or
!python3 -m pip install -U gensim
from gensim.models import Word2Vec
min_count = 0
size = 50
window = 2
sentence= "bitcoin is an innovative payment network and a new kind of money."
tokens=sentence.split()
print(tokens)<jupyter_output>['bitcoin', 'is', 'an', 'innovative', 'payment', 'network', 'and', 'a', 'new', 'kind', 'of', 'money.']<jupyter_text>Getting a word vector using Gensim.<jupyter_code>model = Word2Vec(tokens, min_count=min_count, vector_size=size,window=window)
model.
# vector of 'a'
model.wv['a']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08317677198975235, 'fraction_numerical': 0.01372046683746086, 'mean_word_length': 5.522836984775344, 'pattern_counts': {'":': 1, '<': 94, '<?xml version=': 0, '>': 98, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '534328', 'license': 'MIT', 'n_tokens_mistral': 5451, 'n_tokens_neox': 5104, 'n_words': 2017, 'path': '05-Getting Started With NLP.ipynb', 'repo_name': 'Muhammad-Elgendi/nlp', 'size': '17608'}