<jupyter_start><jupyter_text>Text CleaningThis is a code walkthrough for self-starters on most common text cleaning tasks.After the end of this notebook, you will be able to: 1. Understand tokenization, do it manually yourself - and using spaCy2. Understand why stop word removal and case standardization work - with spaCy examples3. Differentiate between Stemming and Lemmatization - with spaCy lemmatization examples I have always liked **The Adventures of Sherlock Holmes** by _<NAME>_. Let's download the book and save it locally:<jupyter_code>url = 'http://www.gutenberg.org/ebooks/1661.txt.utf-8'
file_name = 'sherlock.txt'
import urllib.request
# Download the file from `url` and save it locally under `file_name`:

with urllib.request.urlopen(url) as response:
    with open(file_name, 'wb') as out_file:
        data = response.read() # a `bytes` object
        out_file.write(data)
!ls {*.txt}
!head -2 sherlock.txt<jupyter_output>ï»¿Project Gutenberg's The Adventures of Sherlock Holmes, by <NAME><jupyter_text>The file contains header and footer information from Project Gutenberg. We are not interested in the same and will discard the copyright and other legal notices. Todo: - Open the file and delete the header and footer information and save the file as ```sherlock_clean.txt```I opened the text file to see that I need to remove the first 33 lines. Let's do that using shell commands - which also work on Windows inside Jupyter notebook:<jupyter_code>!sed -i 1,33d sherlock.txt<jupyter_output><empty_output><jupyter_text>I use the ```sed``` syntax. The ```-i``` flag tells to make the changes in place.  ```1,33d``` instructs to delete lines 1 to 33.<jupyter_code>!head -5 sherlock.txt<jupyter_output>THE ADVENTURES OF SHERLOCK HOLMES

by

SIR <NAME><jupyter_text>What do I see? Before I continue to text cleaning for any Natural Language Processing Task, I like to spend a few seconds taking a quick glance at the data itself. I noted down some of the things I spotted below, of course a trained eye can see a lot more than I did: 1. Dates are written in a mixed format: `twentieth of March, 1888`, times are too: `three o'clock`1. Text is wrapped at around 70 columns, or no line can be longer than 70 characters 1. There are lot of proper nouns. These include names such as `Atkinson`, `Trepoff` in addition to locations such as `Trinconmalee` and `Baker Street` etc.1. The index is in Roman numerals such as `I` and `IV` and not `1` or `4`1. There are lot of dialogues such as: "You have carte blanche." with no narrative around them. This storytelling style switches freely from a narrative to a dialogue driven. 1. The grammar and vocabulary is slightly unusual because of the time when Doyle wrote.    Load Data<jupyter_code>#let's the load data to RAM
text = open(file_name, 'r', encoding='utf-8').read()  # note that I add an encoding='utf-8' parameter to preserve information
print(text[:5])
print(f'The file is loaded as datatype: {type(text)} and has {len(text)} characters in it')<jupyter_output>The file is loaded as datatype: <class 'str'> and has 581204 characters in it<jupyter_text>Exploring Loaded Data<jupyter_code># how many unique characters do we see? 
# For reference, ASCII has 127 characters in it - so we expect this to have at most 127 characters
unique_chars = list(set(text))
unique_chars.sort()
print(unique_chars)
print(f'There are {len(unique_chars)} unique characters, including both ASCII and Unicode character')<jupyter_output>['\n', ' ', '!', '"', '$', '%', '&', "'", '(', ')', '*', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'à', 'â', 'è', 'é']
There are 85 unique characters, including both ASCII and Unicode character<jupyter_text>For our machine learning models, we often need the words to occur as individual tokens or single words. This process is called: Tokenization We convert the raw text into a list of words. This preserves the original ordering of the text.  Split by Whitespace<jupyter_code>words = text.split()
print(len(words))
print(words[90:200])  #start with the first chapeter, ignoring the index for now
# Let's look at another example: 
'red-headed woman on the street'.split()<jupyter_output><empty_output><jupyter_text>Notice how the words red-headed were not split. This is something we may or may not want to keep always.  *Problem:* Punctuations are often appearing with the word itself, like: `Adler.` and `emotions,`.*Solution:* Simply extract words and discard everything else. This means we will discard all non-ASCII characters and punctuations. Split by Word Extraction**Introducing Regex**<jupyter_code>import re
re.split('\W+', 'Words, words, words.')<jupyter_output><empty_output><jupyter_text>Regular expressions can be daunting at first, but are very powerful. The regular expression `\W+` means *a word character (A-Z etc.) repeated one or more times*.<jupyter_code>words_alphanumeric = re.split('\W+', text)
len(words_alphanumeric), len(words)
print(words_alphanumeric[90:200])<jupyter_output>['BOHEMIA', 'I', 'To', 'Sherlock', 'Holmes', 'she', 'is', 'always', 'THE', 'woman', 'I', 'have', 'seldom', 'heard', 'him', 'mention', 'her', 'under', 'any', 'other', 'name', 'In', 'his', 'eyes', 'she', 'eclipses', 'and', 'predominates', 'the', 'whole', 'of', 'her', 'sex', 'It', 'was', 'not', 'that', 'he', 'felt', 'any', 'emotion', 'akin', 'to', 'love', 'for', 'Irene', 'Adler', 'All', 'emotions', 'and', 'that', 'one', 'particularly', 'were', 'abhorrent', 'to', 'his', 'cold', 'precise', 'but', 'admirably', 'balanced', 'mind', 'He', 'was', 'I', 'take', 'it', 'the', 'most', 'perfect', 'reasoning', 'and', 'observing', 'machine', 'that', 'the', 'world', 'has', 'seen', 'but', 'as', 'a', 'lover', 'he', 'would', 'have', 'placed', 'himself', 'in', 'a', 'false', 'position', 'He', 'never', 'spoke', 'of', 'the', 'softer', 'passions', 'save', 'with', 'a', 'gibe', 'and', 'a', 'sneer', 'They', 'were', 'admirable']<jupyter_text>We notice how `Adler` no longer has the punctuation with her. This is what we wanted. Mission Accomplished.  **What was the tradeoff we made here?** To understand that, let's look at another example:<jupyter_code>words_break = re.split('\W+', "Isn't he coming home for dinner with the red-headed girl?")
print(words_break)<jupyter_output>['Isn', 't', 'he', 'coming', 'home', 'for', 'dinner', 'with', 'the', 'red', 'headed', 'girl', '']<jupyter_text>We have split `Isn't` to `Isn` and `t`. This is not good if you were working with say email or Twitter data, because you would've a lot more of such contractions. As a minor annoyance, we have an extra empty token at the end. Similarly, because we neglected punctuation `red-headed` is broken into two words: `red` and `headed`We can write custom rules in our tokenization strategy to cover all these cases. Or, use something which already has been written for us.  spaCy for Tokenization<jupyter_code>%%time
import spacy
nlp = spacy.load('en')
doc = nlp(text)<jupyter_output><empty_output><jupyter_text>The above syntax creates a spaCy object `doc`. The object pre-computes a lot of linguistic features, including tokens. We can retrieve them by calling the object iterator. Below, we call the iterator and `list` it.<jupyter_code>print(list(doc)[150:200])<jupyter_output>[whole, of, her, sex, ., It, was, not, that, he, felt, 
, any, emotion, akin, to, love, for, <NAME>, ., All, emotions, ,, and, that, 
, one, particularly, ,, were, abhorrent, to, his, cold, ,, precise, but, 
, admirably, balanced, mind, ., He, was, ,, I, take, it, ,]<jupyter_text>Conveniently, spaCy tokenizes all *punctuations and words* and returned those as individual tokens as well. Let's try the example which we didn't like earlier:<jupyter_code>words = nlp("Isn't he coming home for dinner with the red-headed girl?")
print([token for token in words])<jupyter_output>[Is, n't, he, coming, home, for, dinner, with, the, red, -, headed, girl, ?]<jupyter_text>*Observations*:- spaCy got the `Isn't` split as we wanted - `red-headed` was broken into 3 tokens: `red`, `-`, `headed`. Since the punctuation information isn't lost, we can restore the original `red-headed` token if we want to**How does the spaCy tokenizer work ?**> First, the raw text is split on whitespace characters, similar to text.split(' '). Then, the tokenizer processes the text from left to right. On each substring, it performs two checks:> > - **Does the substring match a tokenizer exception rule?** For example, "don't" does not contain whitespace, but should be split into two tokens, "do" and "n't", while "U.K." should always remain one token.> - **Can a prefix, suffix or infix be split off?** For example punctuation like commas, periods, hyphens or quotes.>> ![caption](https://spacy.io/assets/img/tokenization.svg)> from [spaCy-101](https://spacy.io/usage/spacy-101) docs **Sentence Tokenization**: We can also use spaCy to extract one sentence at a time, instead of one-word-at-a-time.<jupyter_code>sentences = list(doc.sents)
print(sentences[13:18])<jupyter_output>[I. A SCANDAL IN BOHEMIA

I.

To <NAME>, she is always THE woman., I have seldom heard
him mention her under any other name., In his eyes she eclipses
and predominates the whole of her sex., It was not that he felt
any emotion akin to love for <NAME>.]<jupyter_text>STOP WORD REMOVAL & CASE CHANGEThese simple ideas are widespread and fairly effective for a lot of tasks. They are particularly useful in reducing the number of unique tokens in a document for your processing.   spaCy has already marked each token as a stop word or not and stored it in `is_stop` attribute of each token. This makes it very handy for text cleaning. Let's take a quick look:<jupyter_code>sentence_example = "the AI/AGI uprising cannot happen without the progress of NLP"
[(token, token.is_stop, token.is_punct) for token in nlp(sentence_example)]
for token in doc[:5]:
    print(token, token.is_stop, token.is_punct)<jupyter_output>THE False False
ADVENTURES False False
OF False False
SHERLOCK False False
HOLMES False False<jupyter_text>Interesting, while `the` and `of` were marked as stop words, `THE` and `OF` were not. This is not a bug but by design. spaCy doesn't remove words which are probably important because of their CAPS or Title Case automatically. We can instead force this behaviour by converting our original text to lower case before we pass it to spaCy.<jupyter_code>text_lower = text.lower()  # native python function
doc_lower = nlp(text_lower)
for token in doc_lower[:5]:
    print(token, token.is_stop)
from spacy.lang.en.stop_words import STOP_WORDS
f'spaCy has a dictionary of {len(list(STOP_WORDS))} stop words'<jupyter_output><empty_output><jupyter_text>You can also extend the STOP WORDS dictionary on the fly for your domain. For instance, if you were using this code to process text of a NLP book, we might want to add words like `NLP`, `Processing`, `AGI`, `Data` etc. to stop words list.<jupyter_code>domain_stop_words = ["NLP", "Processing", "AGI"]
for word in domain_stop_words:
    STOP_WORDS.add(word)
[(token, token.is_stop, token.is_punct) for token in nlp(sentence_example)]<jupyter_output><empty_output><jupyter_text>Exactly as expected, `NLP` and `AGI` are now marked as stop words too. Let's pull out string tokens which are not stop words into a Python list or similar data structure. Most NLP tasks which come after text pre-processing expect string tokens and not spaCy token objects as datatype. We will remove both stop words and punctuations here for demonstration:<jupyter_code>[str(token) for token in nlp(sentence_example) if not token.is_stop and not token.is_punct]
[str(token) for token in nlp(sentence_example) if not token.is_stop]<jupyter_output><empty_output><jupyter_text>Stemming and Lemmatization> **Stemming** usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. > **Lemmatization** usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma. > If confronted with the token `saw`, stemming might return just `s`, whereas lemmatization would attempt to return either `see` or `saw` depending on whether the use of the token was as a verb or a noun.> - <NAME> et al, 2008, [IR-Book](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html)   spaCy for Lemmatization**spaCy only supports lemmatization**. As discussed by spaCy core contributor in [issue 327](https://github.com/explosion/spaCy/issues/327) on Github, stemmer's are rarely a good idea. We want to treat `meet/NOUN` as different from `meeting/VERB`. Unlike Stanford NLTK which was created to introduce as many NLP ideas as possible, spaCy takes an opinionated stand against stemming.  An underscore at end, such as `lemma_` tells spaCy we are looking for something which is human readable. spaCy stores the internal hash or identifier which spaCy stores in `token.lemma`.<jupyter_code>lemma_sentence_example = "Their Apples & Banana fruit salads are amazing. Would you like meeting me at the cafe?"
[(token, token.lemma_, token.lemma, token.pos_ ) for token in nlp(lemma_sentence_example)]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12328166130447499, 'fraction_numerical': 0.0076045627376425855, 'mean_word_length': 5.276732446076181, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 82, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '426290', 'license': 'MIT', 'n_tokens_mistral': 4420, 'n_tokens_neox': 4166, 'n_words': 1829, 'path': 'Part-02-A.ipynb', 'repo_name': 'theainerd/nlp-python-deep-learning', 'size': '13745'}