<jupyter_start><jupyter_text>Data AcquisitionHow to get the data?1. Use a public dataset2. Scrape data3. Product intervention: AI team work with the product team to collect more data by developing better instrumentation.4. Data augmentation    - Synonym replacement    - Back translation    - TF-IDF-based word replacement    - Bigram flipping    - Replacing entities    - Adding noises to data    - Advanced techniques: Snorkel, Easy Data Augmentation (EDA), Active learning  Text Extraction and Cleanup  HTML parsing and cleanup<jupyter_code>from bs4 import BeautifulSoup
from urllib.request import urlopen
myurl = "https://stackoverflow.com/questions/415511/how-to-get-the-current-time-in-python"

html = urlopen(myurl).read()
soupified = BeautifulSoup(html, "html.parser")

question = soupified.find("div", {"class": "question"})
questiontext = question.find("div", {"class": "s-prose js-post-body"})
print("Question: \n", questiontext.get_text().strip())

answer = soupified.find("div", {"class": "answer"})
answertext = answer.find("div", {"class": "s-prose js-post-body"})
print("Best answer: \n", answertext.get_text().strip())<jupyter_output>Question: 
 What is the module/method used to get the current time?
Best answer: 
 Use:
>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2009, 1, 6, 15, 8, 24, 78915)

>>> print(datetime.datetime.now())
2009-01-06 15:08:24.789150

And just the time:
>>> datetime.datetime.now().time()
datetime.time(15, 8, 24, 78915)

>>> print(datetime.datetime.now().time())
15:08:24.789150

See the documentation for more information.
To save typing, you can import the datetime object from the datetime module:
>>> from datetime import datetime

Then remove the leading datetime. from all of the above.<jupyter_text>Unicode normalization<jupyter_code>text = 'I love 🍕!  Shall we book a 🚗 to gizza?'
Text = text.encode("utf-8")
print(Text)<jupyter_output>b'I love \xf0\x9f\x8d\x95!  Shall we book a \xf0\x9f\x9a\x97 to gizza?'<jupyter_text>Spelling correction<jupyter_code># Need azure upgraded account
# Tutorial: https://docs.microsoft.com/en-us/azure/cognitive-services/bing-spell-check/quickstarts/python

import requests
import json

api_key = "<ENTER-KEY-HERE>"
example_text = "Hollo, wrld"
endpoint = "https://api.cognitive.microsoft.com/bing/v7.0/SpellCheck"

data = {'text': example_text}
params = {
    'mkt':'en-us',
    'mode':'proof'
    }
headers = {
    'Content-Type': 'application/x-www-form-urlencoded',
    'Ocp-Apim-Subscription-Key': api_key,
    }
response = requests.post(endpoint, headers=headers, params=params, data=data)
json_response = response.json()
print(json.dumps(json_response, indent=4))<jupyter_output>{
    "error": {
        "code": "401",
        "message": "Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource."
    }
}<jupyter_text>System-specific error correction<jupyter_code>from PIL import Image
import pytesseract
from pytesseract import image_to_string

# need to follow this steps to use tesseract
# https://stackoverflow.com/questions/50951955/pytesseract-tesseractnotfound-error-tesseract-is-not-installed-or-its-not-i
pytesseract.pytesseract.tesseract_cmd = r'D:\Program Files\Tesseract-OCR\tesseract.exe'

filename = "scanned_document.png"
text = image_to_string(Image.open(filename))
print(text)<jupyter_output>in the nineteenth century the only Kind of linguistics considered
seriously was this comparative and historical study of words in languages
known or believed to be cognate—say the Semitic languages, or the Indo-
European languages. It is significant that the Germans who really made
the subject what it was, used the term Indo-germanisch. Those who know
the popular works of Otto Jespersen will remember how fitmly he
declares that linguistic science is historical. And those who have noticed<jupyter_text>Pre-Processing- Preliminaries:    - Sentence segmentation, word tokenization- Frequent steps:    - stop word removal, stemming and lemmatization, removing digits/punctuation, lowercasing.- Other steps:    - normalization, language detection, code mixing, transliteration, etc.- Advanced processing:    - POS tagging, parsing, coreference resolution, etc.<jupyter_code># Sentence segmentation
from nltk.tokenize import sent_tokenize, word_tokenize

mytext = "In the previous chapter, we saw examples of some common NLP \
applications that we might encounter in everyday life. If we were asked to \
build such an application, think about how we would approach doing so at our \
organization. We would normally walk through the requirements and break the \
problem down into several sub-problems, then try to develop a step-by-step \
procedure to solve them. Since language processing is involved, we would also \
list all the forms of text processing needed at each step. This step-by-step \
processing of text is known as pipeline. It is the series of steps involved in \
building any NLP model. These steps are common in every NLP project, so it \
makes sense to study them in this chapter. Understanding some common procedures \
in any NLP pipeline will enable us to get started on any NLP problem encountered \
in the workplace. Laying out and developing a text-processing pipeline is seen \
as a starting point for any NLP application development process. In this \
chapter, we will learn about the various steps involved and how they play \
important roles in solving the NLP problem and we’ll see a few guidelines \
about when and how to use which step. In later chapters, we’ll discuss \
specific pipelines for various NLP tasks (e.g., Chapters 4–7)."

my_sentences = sent_tokenize(mytext)
my_sentences
# word tokenization
for sentence in my_sentences:
    print(sentence)
    print(word_tokenize(sentence))
# frequent steps
from nltk.corpus import stopwords
from string import punctuation

def preprocess_corpus(texts):
    mystopwords = set(stopwords.words("english"))
    def remove_stops_digits(tokens):
        return [token.lower() for token in tokens if token not in mystopwords and 
                not token.isdigit() and token not in punctuation]
    return [remove_stops_digits(word_tokenize(text)) for text in texts]
preprocess_corpus([mytext])
# stemming and lemmatization
import nltk
nltk.download('wordnet')
    
from nltk.stem.porter import PorterStemmer

stemmer = PorterStemmer()
word1, word2 = "cars", "revolution"
print(stemmer.stem(word1), stemmer.stem(word2))


from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("better", pos="a")) #a is for adjective

import spacy

sp = spacy.load('en_core_web_sm')
token = sp(u'better')
for word in token:
    print(word.text,  word.lemma_)
import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp(u'<NAME> was born on 16 April 1889 to<NAME> (born <NAME>) and <NAME> Sr')
for token in doc:
    print(token.text, token.lemma_, token.pos_,
          token.shape_, token.is_alpha, token.is_stop)<jupyter_output><NAME> PROPN Xxxxx True False
Spencer Spencer PROPN Xxxxx True False
Chaplin Chaplin PROPN Xxxxx True False
was be AUX xxx True True
born bear VERB xxxx True False
on on ADP xx True True
16 16 NUM dd False False
April April PROPN Xxxxx True False
1889 1889 NUM dddd False False
toHannah toHannah PROPN xxXxxxx True False
Chaplin Chaplin PROPN Xxxxx True False
( ( PUNCT ( False False
born bear VERB xxxx True False
<NAME> PROPN Xxxxx True False
Harriet Harriet PROPN Xxxxx True False
Pedlingham Pedlingham PROPN Xxxxx True False
Hill Hill PROPN Xxxx True False
) ) PUNCT ) False False
and and CCONJ xxx True True
Charles Charles PROPN Xxxxx True False
Chaplin Chaplin PROPN Xxxxx True False
Sr Sr PROPN Xx True False
{'fraction_non_alphanumeric': 0.08795999487113733, 'fraction_numerical': 0.01513014489037056, 'mean_word_length': 4.769230769230769, 'pattern_counts': {'":': 7, '<': 23, '<?xml version=': 0, '>': 41, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '200276', 'license': 'MIT', 'n_tokens_mistral': 2493, 'n_tokens_neox': 2335, 'n_words': 875, 'path': '02_NLP Pipeline/NLP Pipeline.ipynb', 'repo_name': 'yasirabd/practical-nlp', 'size': '7873'}