<jupyter_start><jupyter_text>Natural Language Processing What is NLP> "*Natural language processing (NLP) is a collective term referring to automatic computational processing of human languages. This includes both algorithms that take human-produced text as input, and algorithms that produce natural looking text as outputs.*" *(Goldberg, 2017) Where NLP is used? **Text Classification**    *is the process of assigning tags or categories to text according to its content.** **Sentiment analysis**      *is the process of analyzing emotions within a text and classifying them as positive, negative, or neutral*            Example: By running sentiment analysis on social media posts, product reviews, NPS surveys, and customer feedback, businesses can gain valuable insights about how customers perceive their brand. See this Zoom customer and product reviews, e.g.:                      A sentiment classifier tries to detect the emotions people express with their words, and classifies texts into Positive, Negative, or Neutral. From image above it can understand the nuance of each opinion and automatically tag 1st review as Negative and 2nd one as Positive.                         * **Topic classification**      *is the process of identifying the main themes or topics within a text and assigning predefined tags.*      For training a topic classifier, it's needed to be familiar with the data which is analyzing, so to define relevant categories.        Example: you might work for a software company, and receive a lot of customer support tickets that mention technical issues, usability, and feature requests.In this case, you might define your tags as Bugs, Feature Requests, and UX/IX.* **Intent detection**      *is the process of identifying the purpose, goal, or intention behind a text.*              Example: sorting outbound sales email responses by Interested, Need Information, Unsubscribe, Bounce.The tag Interested could help to spot a potential sale opportunity as soon as an email enters an inbox!                     Text Extraction  *is the process of extraction of specific information that is already in the text.*  * **Keyword extraction**      *is the process of automatically extracting the most important words and expressions within a text.*      This can provide with a sort of preview of the content and its main topics, without needing to read each piece.           Example: check out this feature request, below, processed with MonkeyLearn’s public keyword extractor     * **Named Entity Recognition (NER)**      *is the process of extracting the entities such as names of people, companies, places, etc.*      * **Machine Translation**      *is the process of translating speech and text to different languages*      One of the first problems addressed by NLP researchers. Online translation tools (like *Google Translate*) use different NLP techniques to achieve human-levels of accuracy in translating speech and text to different languages. Custom translators models can be trained for a specific domain to maximize the accuracy of the results. * **Topic Modeling**      *is the process of finding relevant topics in a text by grouping texts with similar words and expressions.*       This is similar to topic classification. * **Natural Language Generation (NLG)**      *is the process of analyzing unstructured data and using it as an input to automatically create content*       Can be used to generate automated answers, write emails, and even books!  Tasks & Techniques**Syntactic analysis**  or parsing/syntax analysis, identifies the syntactic structure of a text and the dependency relationships between words, represented on a diagram called a parse tree.**Semantic analysis**  focuses on identifying the meaning of language. However, since language is polysemic and ambiguous, semantics is considered one of the most challenging areas in NLP. Semantic tasks analyze the structure of sentences, word interactions, and related concepts, in an attempt to discover the meaning of words, as well as understand the topic of a text. * Tokenization* Part-of-speech tagging* Dependency Parsing* Constituency Parsing* Lemmatization & Stemming* Stopword Removal* Word Sense Disambiguation* Named Entity Recognition (NER)* Relationship extraction Tokenization  *is an essential task in NLP used to break up a string of words into semantically useful units called **tokens**.*Sentence tokenization splits sentences within a text, and word tokenization splits words within a sentence. Generally, word tokens are separated by blank spaces, and sentence tokens by stops. However, you can perform high-level tokenization for more complex structures, like words that often go together, otherwise known as collocations (e.g., *New York*).    Example of how word tokenization simplifies text:      Customer service couldn’t be better! = “customer service” “could” “not” “be” “better”.          Part-of-speech (PoS) tagging*involves adding a part of speech category to each token within a text.*  Some common PoS tags are verb, adjective, noun, pronoun, conjunction, preposition, intersection, among others. PoS tagging is useful for identifying relationships between words and, therefore, understand the meaning of sentences.    Example above would look like this:      “Customer service”: NOUN, “could”: VERB, “not”: ADVERB, be”: VERB, “better”: ADJECTIVE, “!”: PUNCTUATION Dependency Parsing  *refers to the way the words in a sentence are connected.*     Example: a dependency parser, therefore, analyzes how ‘head words’ are related and modified by other words too understand the syntactic structure of a sentence    Constituency Parsing  *aims to visualize the entire syntactic structure of a sentence by identifying phrase structure grammar.*  It consists of using abstract terminal and non-terminal nodes associated to words.    Example:    Lemmatization & Stemming  *aims to transform the words back to their root form.*The word as it appears in the dictionary – its **root form** – is called a **lemma**. When we speak or write, we tend to use inflected forms of a word (words in their different grammatical forms). To make these words easier for computers to understand, NLP uses lemmatization and stemming to transform them back to their root form.     Example:     The terms "is, are, am, were, and been,” are grouped under the lemma ‘be.’ So, if we apply this lemmatization to “African elephants have four nails on their front feet,” the result will look something like this:    African elephants have four nails on their front feet = “African,” “elephant,” “have,” “4”, “nail,” “on,” “their,” “foot”]This example is useful to see how the lemmatization changes the sentence using its base form (e.g., the word "feet"" was changed to "foot").When we refer to stemming, the root form of a word is called a stem. Stemming "trims" words, so word stems may not always be semantically correct.For example, stemming the words “consult,” “consultant,” “consulting,” and “consultants” would result in the root form “consult.”While lemmatization is dictionary-based and chooses the appropriate lemma based on context, stemming operates on single words without considering the context.     Example: “This is better”      The word “better” is transformed into the word “good” by a lemmatizer but is unchanged by stemming. Eventhough stemmers can lead to less-accurate results, they are easier to build and perform faster than lemmatizers. But lemmatizers are recommended if seeking more precise linguistic rules. Stopword Removal  *involves filtering out high-frequency words that add little or no semantic value to a sentence, for example, which, to, at, for, is, etc.*  Removing stop words is an essential step in NLP text processing. Lists of stopwords can be customized in order to include words that you want to ignore.    Example:      Let’s say you want to classify customer service tickets based on their topics.     In this example: “Hello, I’m having trouble logging in with my new password”, it may be useful to remove stop words like “hello”, “I”, “am”, “with”, “my”, so you’re left with the words that help you understand the topic of the ticket: “trouble”, “logging in”, “new”, “password”.     Word Sense Disambiguation (WSD)Depending on their context, words can have different meanings.     Example: the word “book”      * You should read this book; it’s a great novel!      * You should book the flights as soon as possible.      * You should close the books by the end of the year.      * You should do everything by the book to avoid potential complications.There are two main techniques that can be used for WSD: * knowledge-based (or dictionary approach)  tries to infer meaning by observing the dictionary definitions of ambiguous terms within a text* supervised approach  is based on NLP algorithms that learn from training data. Named Entity Recognition (NER)  *involves extracting entities from within a text (e.g., names, places)*  Entities can be names, places, organizations, email addresses, and more. Open-Source named entity recognition APIsOpen-source APIs are for developers: they are free, flexible, and entail a gentle learning curve. Here are a few options:   * **Stanford Named Entity Recognizer (SNER)**         JAVA tool developed by Stanford University is considered the standard library for entity extraction. It’s based on Conditional Random Fields (CRF) and provides pre-trained models for extracting person, organization, location, and other entities.    * **SpaCy**         Python framework known for being fast and very easy to use. It has an excellent statistical system that you can use to build customized NER extractors.   * **Natural Language Toolkit (NLTK)**         Python lib which is widely used for NLP tasks. NLKT has its own classifier to recognize named entities called ne_chunk, but also provides a wrapper to use the Stanford NER tagger in Python. Relationship extraction*finds relationships between two nouns.*      Example:      In the phrase “Susan lives in Los Angeles,” a person (Susan) is related to a place (Los Angeles) by the semantic category “lives in.”  NLP in everyday life11 of the most common and most powerful uses of natural language processing in everyday life:* Email filters* Virtual assistants, voice assistants, or smart speakers    * Amazon's Alexa, Yandex's Alice* Online search engines     * Google, Yandex, ...* Predictive text and autocorrect* Monitor brand sentiment on social media* Sorting customer feedback* Automating processes in customer support* Chatbots* Automatic summarization* Machine translation* Natural language generation   [**SpaCy**](https://spacy.io/usage)<jupyter_code>#!pip install -U spacy
#!python -m spacy download en_core_web_sm
#!python -m spacy validate

import sys
!{sys.executable} -m pip install spacy
!{sys.executable} -m spacy download en_core_web_sm
!{sys.executable} -m spacy validate
import spacy
nlp = spacy.load("en_core_web_sm")<jupyter_output><empty_output><jupyter_text>[SpaCy linguistic features](https://spacy.io/usage/linguistic-features)<jupyter_code># Named Entity Recognition
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)
"Apple is looking at buying U.K. startup for $1 billion"[27:31]<jupyter_output><empty_output><jupyter_text><jupyter_code># Spacy Visualizer
from spacy import displacy
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
displacy.render(doc, style="dep", jupyter=True)
#displacy.serve(doc, style="dep")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
displacy.render(doc, style="ent", jupyter=True)
#displacy.serve(doc, style="ent")
# Tokenization
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for token in doc:
    print(token.text)<jupyter_output>Apple
is
looking
at
buying
U.K.
startup
for
$
1
billion<jupyter_text>[NLTK](https://www.nltk.org/)<jupyter_code>import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')
nltk.download('treebank')
sentence = "Apple is looking at buying U.K. startup for $1 billion"
tokens = nltk.word_tokenize(sentence)
tokens
tagged = nltk.pos_tag(tokens)
tagged
entities = nltk.chunk.ne_chunk(tagged)
#entities
# Display parse trees
from nltk.corpus import treebank

t = treebank.parsed_sents('wsj_0001.mrg')[0]
t.draw()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0646630964751019, 'fraction_numerical': 0.0019982415474382544, 'mean_word_length': 4.565836298932385, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '334412', 'license': 'MIT', 'n_tokens_mistral': 3187, 'n_tokens_neox': 2999, 'n_words': 1691, 'path': '10_nlp_basics_lecture.ipynb', 'repo_name': 'ckorikov/mlcc', 'size': '12511'}