<jupyter_start><jupyter_text>[LEGALST-190] Preprocessing Text - Lab 3-1---This lab will provide an introduction to manipulating strings and chunking sentences.*Estimated Time: 30-40 minutes*--- Topics Covered- How to tokenize text- How to stem text- How to chunk text Table of Contents[The Data](section data)1 - [Tokenization](section 1)2 - [Stemming](section 2)3 - [Chunking](section 3)<jupyter_code>! pip install nltk<jupyter_output>Requirement already satisfied: nltk in c:\users\anike\anaconda3\lib\site-packages
Requirement already satisfied: six in c:\users\anike\anaconda3\lib\site-packages (from nltk)<jupyter_text>--- The Data In this notebook, you'll be working with the text of each country’s statement from the General Debate in annual sessions of the United Nations General Assembly. This dataset is separated by country, session and year and tagged for each, and has over forty years of data from different countries.  Visualizing data Run the below cells and take a look at a sample of the data that we'll be working with.<jupyter_code>import pandas as pd
data = pd.read_csv("../data/un-general-debates.zip", compression='zip')
data.head()<jupyter_output><empty_output><jupyter_text>Tokenization   Tokenization is defined as the process of segmenting running text into words and sentences.  Why do we need to tokenize text Electronic text is a linear sequence of symbols. Before any processing is to be done, text needs to be segmented into linguistic units, and this process is called tokenization. We usually look at grammar and meaning at the level of words, related to each other within sentences, within each document. So if we're starting with raw text, we first need to split the text into sentences, and those sentences into words -- which we call "tokens".  How to tokenize You might imagine that the easiest way to identify sentences is to split the document at every period '.', and to split the sentences using white space to get the words.<jupyter_code># using the split function to create tokens
paragraph = data['text'][0]
sentences = paragraph.split(".")
for s in sentences[:5]:
    print(s + '\n')<jupyter_output>﻿It is indeed a pleasure for me and the members of my delegation to extend to Ambassador Garba our sincere congratulations on his election to the presidency of the forty-fourth session of the General Assembly

 His election to this high office is a well-deserved tribute to his personal qualities and experience

 I am fully confident that under his able and wise leadership the Assembly will further consolidate the gains achieved during the past year


My delegation associates itself with previous speakers in expressing its appreciation of the dedicated efforts of his predecessor, His Excellency Mr

 <NAME>, for the exemplary manner in which he discharged his duties as President of the forty-third session of the General Assembly<jupyter_text>Then to split sentences further into words.<jupyter_code>sentence = "What kind of patterns do you see in this graph?"
tokens = sentence.split(" ")
tokens<jupyter_output><empty_output><jupyter_text>We'll stop here as NLTK provides handy tools for us to use.  NLTK NLTK (Natural Language Toolkit) is a platform for building Python programs to work with human language data<jupyter_code>import nltk
# run the below commented command if error
#nltk.download('punkt')
# create sentence tokens
speech = data['text'][4]
sents = nltk.sent_tokenize(speech)
sents[:3]
s4 = "At eight o'clock on Thursday morning Arthur didn't feel very good."
nltk.word_tokenize(s4)<jupyter_output><empty_output><jupyter_text>nltk recognized that "o'clock" is one word and separated "didn't" into "did" and "n't" For more complicated metrics, it's easier to use NLTK's classes and methods.<jupyter_code># Find the 10 most common tokens
tokens = nltk.word_tokenize(speech)
fd = nltk.collocations.FreqDist(tokens)
fd.most_common()[:10]<jupyter_output><empty_output><jupyter_text>Not so interesting as the most common words seem to be words that have no particular meanings. A common step in text analysis is to remove noise. *However*, what you deem "noise" is not only very important but also dependent on the project at hand. For the purposes of today, we will discuss two common categories of strings often considered "noise". - Punctuation: While important for sentence analysis, punctuation will get in the way of word frequency and n-gram analyses. They will also affect any clustering on topic modeling.- Stopwords: Stopwords are the most frequent words in any given language. Words like "the", "a", "that", etc. are considered not semantically important, and would also skew any frequency or n-gram analysis. Question Write a function below that takes a string as an argument and returns a list of words without punctuation or stopwords.`punctuation` is a list of punctuation strings, and we have created the list `stop_words` for you.Hint: first you'll want to remove punctuation, then tokenize, then remove stop words. Make sure you account for upper and lower case!<jupyter_code>def rem_punc_stop(text):
    
    from string import punctuation
    from nltk.corpus import stopwords
    
    stop_words = set(stopwords.words("english"))
    punctuation = set(punctuation)
    
    punc_free = "".join([ch for ch in text if ch not in punctuation])
    
    words = nltk.word_tokenize(punc_free)
    
    noise_free = [word for word in words if word not in stop_words]
    
    return noise_free
from nltk.corpus import stopwords
from string import punctuation
punctuation<jupyter_output><empty_output><jupyter_text>Now we can rerun our frequency analysis without the noise:<jupyter_code>#nltk.download('stopwords')
tokens_reduced = rem_punc_stop(speech)
tokens_reduced
#type(tokens_reduced)
#fd_reduced = nltk.collocations.FreqDist(tokens_reduced)
#fd_reduced.most_common()[:10]<jupyter_output><empty_output><jupyter_text>Now our analysis is much more informational and revealing. POS tagging The process of classifying words into their parts of speech and labeling them accordingly is known as part-of-speech tagging.<jupyter_code>#nltk.download('averaged_perceptron_tagger')
tagged = nltk.pos_tag(tokens[2:8])
tagged<jupyter_output><empty_output><jupyter_text>Stemming  In NLP it is often the case that the specific form of a word is not as important as the idea to which it refers. For example, if you are trying to identify the topic of a document, counting 'running', 'runs', 'ran', and 'run' as four separate words is not useful. Reducing words to their stems is a process called stemming.A popular stemming implementation is the Snowball Stemmer, which is based on the Porter Stemmer. Its algorithm looks at word forms and does things like drop final 's's, 'ed's, and 'ing's.Just like the tokenizers, we first have to create a stemmer object with the language we are using. Refer to [this documentation](http://www.nltk.org/howto/stem.html) to create a snowball stemmer.<jupyter_code>snowball = nltk.SnowballStemmer('english')<jupyter_output><empty_output><jupyter_text>Now, we can try stemming some words<jupyter_code>snowball.stem('running')
snowball.stem('eats')
snowball.stem('embarassed')<jupyter_output><empty_output><jupyter_text>Snowball is a very fast algorithm, but it has a lot of edge cases. In some cases, words with the same stem are reduced to two different stems<jupyter_code>snowball.stem('cylinder'), snowball.stem('cylindrical')<jupyter_output><empty_output><jupyter_text>Sometimes two different words are reduced to the same stem.<jupyter_code>snowball.stem('vacation'), snowball.stem('vacate')<jupyter_output><empty_output><jupyter_text>Question How would the above two situations affect our text analysis Your answer here  Chunking We may want to work with larger segments of text than single words (but still smaller than a sentence). For instance, in the sentence "The black cat climbed over the tall fence", we might want to treat "The black cat" as one thing (the subject), "climbed over" as a distinct act, and "the tall fence" as another thing (the object). The first and third sequences are noun phrases, and the second is a verb phrase.We can separate these phrases by "chunking" the sentence, i.e. splitting it into larger chunks than individual tokens. This is also an important step toward identifying entities, which are often represented by more than one word. You can probably imagine certain patterns that would define a noun phrase, using part of speech tags. For instance, a determiner (e.g. an article like "the") could be concatenated onto the noun that follows it. If there's an adjective between them, we can include that too.To define rules about how to structure words based on their part of speech tags, we use a grammar (in this case, a "chunk grammar"). NLTK provides a RegexpParser that takes as input a grammar composed of regular expressions (which define patterns in text, we'll learn it in later labs). The grammar is defined as a string, with one line for each rule we define. Each rule starts with the label we want to assign to the chunk (e.g. NP for "noun phrase"), followed by a colon, then an expression in regex-like notation that will be matched to tokens' POS (part-of-speech) tags.We can define a single rule for a noun phrase like this. The rule allows 0 or 1 determiner, then 0 or more adjectives, and finally at least 1 noun. (By using 'NN.*' as the last POS tag, we can match 'NN', 'NNP' for a proper noun, or 'NNS' for a plural noun.) If a matching sequence of tokens is found, it will be labeled 'NP'. Take a look at different [POS tags](http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)<jupyter_code>grammar = "NP: {<DT>?<JJ>*<NN.*>+}"<jupyter_output><empty_output><jupyter_text>We create a chunk parser object by supplying this grammar, then use it to parse a sentence into chunks. The sentence we want to parse must already be POS-tagged, since our grammar uses those POS tags to identify chunks. Let's try this on the second sentence of the speech we generated above.<jupyter_code>from nltk import RegexpParser

cp = RegexpParser(grammar)

sent_tagged = nltk.pos_tag(sents[1])
sent_chunked = cp.parse(sent_tagged)

print(sent_chunked)<jupyter_output>(S
  (NP T/NNP h/NN e/NN  /NNP)
  u/JJ
  n/FW
  (NP a/DT n/NN i/NN)
  m/VBP
  (NP i/NN)
  t/VBP
  (NP y/NN  /NNP)
  o/VBZ
  (NP f/JJ  /NNP t/NN)
  h/VBD
  (NP a/DT t/NN  /NNP d/NN e/NN c/NN i/NN)
  s/VBP
  (NP i/NN)
  o/VBP
  (NP n/JJ  /NNP r/NN e/NN f/NN l/NN e/NN)
  c/VBP
  (NP t/NN s/NN  /NNP)
  n/CC
  (NP o/JJ t/NN  /NNP)
  o/VBZ
  (NP n/JJ l/NN y/NN  /NNP y/NNP)
  o/MD
  u/VB
  (NP r/NN  /NNP)
  o/VBZ
  (NP w/JJ n/JJ  /NNP d/NN i/NN)
  s/VBP
  (NP t/NN i/NN)
  n/VBP
  (NP g/NN u/NN i/NN)
  s/VBP
  (NP h/NN e/NN d/NN  /NNP r/NN e/NN c/NN o/NN r/NN d/NN)
   /VBZ
  (NP a/DT s/JJ  /NN F/NNP)
  o/VBZ
  (NP r/NN e/NN i/NN)
  g/VBP
  (NP n/JJ  /NNP M/NNP i/NN)
  n/VBP
  (NP i/NN)
  s/VBP
  (NP t/NN e/NN r/NN)
   /VBZ
  (NP a/DT n/JJ d/NN  /NNP P/NNP e/NN r/NN)
  m/VBD
  (NP a/DT n/JJ e/NN)
  (NP n/JJ t/NN  /NNP R/NNP e/NN p/NN r/NN e/NN s/NN e/NN)
  n/JJ
  t/VBZ
  (NP a/DT t/NN i/NN)
  v/VBP
  (NP e/NN  /NNP)
  o/VBZ
  (NP f/JJ  /NNP y/NN o/NN)
  (NP u/JJ r/NN  /NNP)
  c/VBZ
  (NP
    o/[...]<jupyter_text>When we called print() on this chunked sentence, it printed out a nested list of nodes.<jupyter_code>type(sent_chunked)<jupyter_output><empty_output><jupyter_text>The tree object has a number of methods we can use to interact with its components. For instance, we can use the method draw() to see a more graphical representation. This will open a separate window.The tree is pretty flat, because we defined a grammar that only grouped words into non-overlapping noun phrases, with no additional hierarchy above them. This is sometimes referred to as "shallow parsing".<jupyter_code>sent_chunked.draw()<jupyter_output><empty_output><jupyter_text>Combining it all Write a function that takes in a strubg, tokenizes it, removes noise, turns everything to lower-case, and returns a string of stems of all tokens.Hint: any function from above that we can just grab and use? As a reminder, this is what our table looks like<jupyter_code>data.head()
def does_it_all(text):
    tokens = ""
    
    not_stemmed = rem_punc_stop(text)
    stemmed = [snowball.stem(word.lower()) for word in not_stemmed]
    for word in not_stemmed:
        tokens += snowball.stem(word) + " "
    
    return tokens<jupyter_output><empty_output><jupyter_text>Let's apply our function to speeches from 2001.First create a table that includes all 2001 speeches. Refer to [this doc](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html)<jupyter_code>speech_2001 = data.loc[data['year'] == 2001]
speech_2001.head()<jupyter_output><empty_output><jupyter_text>Then create a new column in speech_2001 which contains the tokenized string.<jupyter_code>speech_2001_with_tokens = speech_2001.copy()
speech_2001_with_tokens['tokens'] = speech_2001['text'].apply(does_it_all)
speech_2001_with_tokens<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0802724211815121, 'fraction_numerical': 0.006657483930211203, 'mean_word_length': 4.53771186440678, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '486410', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4200, 'n_tokens_neox': 3969, 'n_words': 1708, 'path': 'labs/15_Text Preprocessing/.ipynb_checkpoints/3-1_preprocessing_text-checkpoint.ipynb', 'repo_name': 'jdmarshl/Legal-123-Sp20', 'size': '13074'}