<jupyter_start><jupyter_text>![MLU Logo](../data/MLU_Logo.png)  Machine Learning Accelerator - Natural Language Processing - Lecture 1 Text ProcessingIn this notebook, we go over some simple techniques to clean and prepare text data for modeling with machine learning.1. Simple text cleaning processes2. Lexicon-based text processing    * Stop words removal     * Stemming       * Lemmatization  1. Simple text cleaning processes(Go to top)In this section, we will do some general purpose text cleaning. The following methods for cleaning can be extended depending on the application.<jupyter_code>text = "   This is a message to be cleaned. It may involve some things like: <br>, ?, :, ''  adjacent spaces and tabs     .  "
print(text)<jupyter_output>This is a message to be cleaned. It may involve some things like: <br>, ?, :, ''  adjacent spaces and tabs     .<jupyter_text>Let's first lowercase our text.<jupyter_code>text = text.lower()
print(text)<jupyter_output>this is a message to be cleaned. it may involve some things like: <br>, ?, :, ''  adjacent spaces and tabs     .<jupyter_text>We can get rid of leading/trailing whitespace with the following:<jupyter_code>text = text.strip()
print(text)<jupyter_output>this is a message to be cleaned. it may involve some things like: <br>, ?, :, ''  adjacent spaces and tabs     .<jupyter_text>Remove HTML tags/markups:<jupyter_code>import re

text = re.compile('<.*?>').sub('', text)
print(text)<jupyter_output>this is a message to be cleaned. it may involve some things like: , ?, :, ''  adjacent spaces and tabs     .<jupyter_text>Replace punctuation with space. Be careful with this one, depending on the application, punctuations can actually be useful. For example positive vs negative meanining of a sentence.<jupyter_code>import re, string

text = re.compile('[%s]' % re.escape(string.punctuation)).sub(' ', text)
print(text)<jupyter_output>this is a message to be cleaned  it may involve some things like              adjacent spaces and tabs<jupyter_text>Remove extra space and tabs<jupyter_code>import re

text = re.sub('\s+', ' ', text)
print(text)<jupyter_output>this is a message to be cleaned it may involve some things like adjacent spaces and tabs<jupyter_text>2. Lexicon-based text processing(Go to top)We saw some general purpose text pre-processing methods in the previous section. Lexicon based methods are usually applied after the common text processing methods. They are used to normalize sentences in our dataset. By normalization, here, we mean putting words into a similar format that will also enhace similarities (if any) between sentences. We need to install some packages for this example. Run the following cell.<jupyter_code>import nltk

nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('wordnet')<jupyter_output>[nltk_data] Downloading package punkt to /home/ec2-user/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/ec2-user/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package wordnet to /home/ec2-user/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>Stop word removalThere can be some words in our sentences that occur very frequently and don't contribute too much to the overall meaning of the sentences. We usually have a list of these words and remove them from each our sentences. For example: "a", "an", "the", "this", "that", "is", "it", "to", "and" in this example.<jupyter_code># We will use a tokenizer from the NLTK library
import nltk
from nltk.tokenize import word_tokenize

filtered_sentence = []

# Stop word lists can be adjusted for your problem
stop_words = ["a", "an", "the", "this", "that", "is", "it", "to", "and"]

# Tokenize the sentence
words = word_tokenize(text)
for w in words:
    if w not in stop_words:
        filtered_sentence.append(w)
text = " ".join(filtered_sentence)
print(text)<jupyter_output>message be cleaned may involve some things like adjacent spaces tabs<jupyter_text>StemmingStemming is a rule-based system to convert words into their root form. It removes suffixes from words. This helps us enhace similarities (if any) between sentences. Example:"jumping", "jumped" -> "jump""cars" -> "car"<jupyter_code># We will use a tokenizer and stemmer from the NLTK library
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import SnowballStemmer

# Initialize the stemmer
snow = SnowballStemmer('english')

stemmed_sentence = []
# Tokenize the sentence
words = word_tokenize(text)
for w in words:
    # Stem the word/token
    stemmed_sentence.append(snow.stem(w))
stemmed_text = " ".join(stemmed_sentence)
print(stemmed_text)<jupyter_output>messag be clean may involv some thing like adjac space tab<jupyter_text>You can see above that stemming operation is NOT perfect. We have mistakes such as "messag", "involv", "adjac". It is a rule based method that sometimes mistakely remove suffixes from words. Nevertheless, it runs fast.  LemmatizationIf we are not satisfied with the result of stemming, we can use the Lemmatization instead. It usually requires more work, but gives better results. As mentioned in the class, lemmatization needs to know the correct word position tags such as "noun", "verb", "adjective", etc. and we will use another NLTK function to feed this information to the lemmatizer.<jupyter_code># Importing the necessary functions
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet
from nltk.stem import WordNetLemmatizer

# Initialize the lemmatizer
wl = WordNetLemmatizer()

# This is a helper function to map NTLK position tags
# Full list is available here: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
def get_wordnet_pos(tag):
    if tag.startswith('J'):
        return wordnet.ADJ
    elif tag.startswith('V'):
        return wordnet.VERB
    elif tag.startswith('N'):
        return wordnet.NOUN
    elif tag.startswith('R'):
        return wordnet.ADV
    else:
        return wordnet.NOUN

lemmatized_sentence = []
# Tokenize the sentence
words = word_tokenize(text)
# Get position tags
word_pos_tags = nltk.pos_tag(words)
# Map the position tag and lemmatize the word/token
for idx, tag in enumerate(word_pos_tags):
    lemmatized_sentence.append(wl.lemmatize(tag[0], get_wordnet_pos(tag[1])))

lemmatized_text = " ".join(lemmatized_sentence)
print(lemmatized_text)<jupyter_output>message be clean may involve some thing like adjacent space tabs
{'fraction_non_alphanumeric': 0.0854687971058185, 'fraction_numerical': 0.0025625565269822128, 'mean_word_length': 4.856134157105031, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '156017', 'license': 'MIT-0', 'n_tokens_mistral': 2056, 'n_tokens_neox': 1984, 'n_words': 763, 'path': 'notebooks/MLA-NLP-Lecture1-Text-Process.ipynb', 'repo_name': 'dsMOOC/MLA-NLP', 'size': '6634'}