<jupyter_start><jupyter_text>NLP Pre-Processing<NAME>  04/05/2018  The Purpose of NLP Pre-ProcessingNatural language processing (NLP) is often thought of as one of the main areas in Artificial Intellegence.  NLP techniques are at the core of AI-based products we use every day - chat bots, Google Translate, article summarizers, and the like.  However, NLP actually sits at the crossroads of AI/CS and computational linguistics - it's applications are more widespread than the obvious applications listed above.  NLP techniques allow us to derive things as complex as sentiment from text data, or to find patterns in text for any number of applications (fraud detection, topic segmentation, etc).NLP is characterized as a difficult problem in computer science, due mostly to the ambiguity of human language. Human speech is seldom precise or direct.  Understanding natural language means you need to understand the concepts beneath the words, how they go together, and how the words/order/concepts come together to create meaning.  Before embarking on any NLP techniques (i.e. Sentiment Analysis) we need to make sure the text data is in the proper format.If not, the text won't be accepted into any models or processes.  Transforming your text data into something that an algorithm is able to ingest can be complicated, and it's helpful to have a solid grasp on the text data you're working with.  Generally, there are four stages within NLP pre-processing:- _Cleaning_  Just like with non-text data, cleaning involves excluding the irrelevant or corrupt data points.  In NLP, this typically consists of removing stop words, punctuation, and other extraneous text.  Other cleaning tasks might involve dealing with capitalization rules, or other non-alphanumeric characters.- _Annotation_  Typically annotation include things like parts-of-speech (POS) tagging, and is generally thought of as the application of a scheme to text data.- _Normalization_  The translation or mapping or text within the scheme through Stemming, Lemmatization, or another method of standardization.- _Analysis_  Applying basic statistical techniques to manipulate the data for more in depth analysis.  Tools and MethodsThere are a lot methods and tools available for pre-processing text data. This article is meant to give you a starting point, and is not an exhaustive list of all the options available. Like with all data analysis, the analyst must understand the drawbacks and best uses of each technique and choose a method appropriate for the given dataset. Two well cooonly used Python packages for NLP are `spaCy` and `NLTK` - both have pros and cons.  `NLTK` is highly customizable, but wasn't built to be quick and simple.  `spaCy`, on the other hand, was designed specifically with efficiency in mind and as such is quick and easy to use.<jupyter_code>import pandas as pd
import spacy
from spacy.lang.en import English
parser = English()
import en_core_web_sm  # or any other model you downloaded via spacy download or pip
nlp = en_core_web_sm.load()<jupyter_output><empty_output><jupyter_text>Let's pull in some text data to work with. We're going to use a subset of the 20 Newsgroups dataset, via Sci-Kit Learn. By using the Pandas package we can enforce a tabular structure on the data. This is especially helpful if you're used to working in SQL, SAS, or Excel.<jupyter_code>from sklearn.datasets import fetch_20newsgroups

newsgroups = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'))
df = pd.DataFrame([newsgroups.data, newsgroups.target.tolist()]).T
df.columns = ['text', 'target']
targets = pd.DataFrame( newsgroups.target_names)
targets.columns=['title']
news_data = pd.merge(df, targets, left_on='target', right_index=True)<jupyter_output><empty_output><jupyter_text>We can print the first 5 rows of data to make sure our dataframe looks like we expected it to.<jupyter_code>news_data.head()<jupyter_output><empty_output><jupyter_text>Punctuation The first thing you notice might be that there are some extra characters and puntuation in the text column. It's typical that the data you get will be in less than perfect condition for whatever you're trying to use it for. Removing puctuation is often necessary, and luckily it's pretty easy using spaCy.  CapitalizationText data typically contains a various capitalizations - the beginning of sentences, proper nouns, etc.  The most common approach is to reduce everything to lower case, though this can sometimes affect the fidelity of your data - changing something like "US" to "us" can alter the meaning or affect how the string is tagged (this is where that inside knowledge of the data you're working with and the analysis you're running comes in handy).  Stop Words  Most words within text data are connectors, which do little to show the subject, object, or other information within a sentence. Depending on the type of analysis you're running, excluding these stop words is a necessary step.  Stop words are equivilant to noise within the data.  There are pre-fab stop words lists out there, which can be used as is or altered to fit your particular data. For now we can use the English stop words list from spaCy.<jupyter_code>from spacy.lang.en import STOP_WORDS
print(STOP_WORDS)<jupyter_output>{'call', 'must', 'whereupon', 'less', 'not', 'whether', 'because', 'ca', 'sometime', 'itself', 'per', 'are', 'amongst', 'against', 'alone', 'seem', 'bottom', 'nine', 'only', 'while', 'but', 'becoming', 'would', 'amount', 'by', 'well', 'often', 'ourselves', 'most', 'has', 'ever', 'serious', 'if', 'fifteen', 'such', 'she', 'either', 'front', 'once', 'or', 'every', "'m", 'be', 'enough', 'though', 'your', 'could', 'so', 'there', 'over', 'between', 'seeming', 'six', "'s", 'also', 'out', 'regarding', 'still', 'perhaps', 'toward', 'n’t', 'herself', 'all', "'ll", 'everywhere', 'various', 'towards', 'see', 'my', 'he', 'become', 'himself', 'twenty', 'via', 'made', 'five', 'whence', 'me', 'throughout', 'became', 'former', 'thereby', 'wherever', 'will', '’s', 'herein', 'last', 'before', 're', 'noone', 'using', 'thereafter', '‘s', 'top', 'cannot', 'done', 'does', 'on', '‘m', 'afterwards', 'across', 'as', 'side', 'beyond', 'during', 'part', 'therefore', 'always', 'anyhow', 'meanwhile', 'others', 'fo[...]<jupyter_text>Tokenization  Tokenization is used for splitting sentences into individual words and/or splitting paragraphs into sentences. Splitting sentences into individual words and punctuation is most often done by splitting across white space or punctuation. This might cause problems when you're working with abbreviations, possessives, or proper nouns that use puntuation (like O'Brien or Sackville-West).  Splitting paragraphs into sentences accurately is equally challenging, largely due to the ambiguity of puntuation in the English language. The period alone can be used to denote the end of a sentence, an abbreviation, or be included in an email address. To accurately identify the boundaries of sentences a pre-trained algorithm, like NLTK's Punkt Models, should be used.  Parts of Speech TaggingParts of Speech (POS) tags are useful for understanding the meaning of a sentence, or identifying speech patterns in text. POS tagging typically entails looking at the neighboring words using either a stochastic or rule absed method.    StemmingStemming is a process where words are reduced to their root, removing whatever inflextion is present.  This is usually done by removing the suffix. There are a variety of models available for stemming, including Porter and Snowball. The main drawback to stemming is that words are often overstemmed to the point of uselessness. This happens when words are structurally similar but have vastly different meanings (i.e. "universe" and "university" both stem to "univers"). Since stemming doesn't take into consideration the inflection it's removing, this technique can be useful if you're planning to match words based on origin but really not for much else.  LemmatizationLemmazation is an alternative to stemming, that can produce better end results depending on your specific research or business question. Lemmazation is a more intensive process involving POS tags, which is more accurate than stemming.  For example, broken/ADJ yields broken but broken/VERB yields break. This increased accuracy comes at a slight time cost, but I've found this to be worth the trade off even for very large datasets.  Generally, lemmazatiztion is a better choice when trying to determine sentiment or do any sort of linguistic analysis. One thing to keep in mind is, as you'll see below, lemmas are lowercase.  Let's put this all together.In one block of code we can remove punctuation, stop words, extra white space, and URLs, generate lemmas, tokens, and parts of speech (POS) tags. Then we can add that data to our Pandas dataframe as additional columns.Depending on the size of your dataset, a for loop like I have below might not be the most efficient method. However, even very efficient methods can feel slow because text data often takes considerable time to process compared to more structured data.<jupyter_code>tokens = []
lemma = []
pos = []

for doc in nlp.pipe(news_data['text'].astype('unicode').values, batch_size=100,
                        n_threads=3):
    if doc.is_parsed:
        tokens.append([n.text for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.like_url])
        lemma.append([n.lemma_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.like_url])
        pos.append([n.pos_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.like_url])
    else:
        # We want to make sure that the lists of parsed results have the
        # same number of entries of the original Dataframe, so add some blanks in case the parse fails
        tokens.append(None)
        lemma.append(None)
        pos.append(None)

news_data['tokens'] = tokens
news_data['lemmas'] = lemma 
news_data['pos'] = pos

news_data.head()<jupyter_output><empty_output><jupyter_text>Word Counts One of the more basic, but still powerful, tools for feature engineering is to calculate word, sentence, punctuation, character, and keyword counts. Again, this is where that knowledge of your data will serve you well - you can create your own list of keywords and then calculate the count of those specific words to store as a feature.<jupyter_code>news_data['totalwords'] = news_data['text'].str.split().str.len()
news_data.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07736009847552315, 'fraction_numerical': 0.0017990720575703058, 'mean_word_length': 4.9337078651685395, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '635464', 'license': 'MIT', 'n_tokens_mistral': 2842, 'n_tokens_neox': 2690, 'n_words': 1468, 'path': 'NLP Pre-processing.ipynb', 'repo_name': 'LizMGagne/NLP', 'size': '10568'}