<jupyter_start><jupyter_text>SIT742: Modern Data Science **(Week 04: Text Analysis)**---- Materials in this module include resources collected from various open-source online repositories.- You are free to use, change and distribute this package.- If you found any issue/bug for this document, please submit an issue at [tulip-lab/sit742](https://github.com/tulip-lab/sit742/issues)Prepared by **SIT742 Teaching Team**--- Session 4B - Exploring Pre-Processed text and Generating Features Table of Content* Part 1. Counting Vocabulary by Selecting Tokens of Interest* Part 2. Building Vector Representation * Part 3. Saving Pre-processed Text to a File* Part 4. Extracting Other Features* Part 5. Summary* Part 6. Reading Materials* Part 7. Exercises---One of the challenges of text analysis is to convert unstructured and semi-structured text into a structured representation. This must be done prior to carrying out any text analysis tasks. This chapter will show you how to put some of those basic steps discussed in the previous chapter together to generate different vectorrepresentations for some given text. You will learn how to compute some basic statistics for text, and how to extract features rather than unigrams. Part 1. Counting Vocabulary by Selecting Tokens of InterestTwo important concepts that should be mentioned first are **type** and **token**.Here are the definitions of the two terms, quoted from "[tokenization](http://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html)",  >a **token** is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing;> a **type** is the class of all tokens containing the same character sequence. A *type* is also a vocabulary entry. In other words, a vocabulary consists of a number of word types.The distinction between a type and its tokens is a distinction that separates a descriptive concept fromits particular concrete instances. This is quite similar to the distinction in object-oriented programming between classes and objects.In this section, you are going to learn how to count types in a given corpus by further processing the text.The document collection that we are going to use is a set of Reuters articles that comes with NLTK.It contains 10788 Reuters articles in total and has been split into two subsets, training and testing.Although this collection has already been pre-processed (e.g., you can access the text at different levels, like raw text, tokens, and sentences),we would still like to demonstrate how to put some of the basis text preprocessing steps together and process the raw Reuters articles step by step.First, import the main Python libraries.<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline 
import nltk
from nltk.corpus import reuters<jupyter_output><empty_output><jupyter_text>Since the tokenizer works on a per document level, we can parallelize the process of tokenization with Python's multi-processing module. Please refer to its official documentation [here](https://docs.python.org/2/library/multiprocessing.html).In the following code, we wrap tokenization in a Python function, and thencreate a pool of four worker processes with the Python Pool class.The Pool.map(), a parallel equivalent of the  built-in  map() function, takes one iterable argument.The iterable will be split into a number of chunks, each of which will be submitted to a process in the process pool.Each process will apply a callable function to each element in the chunk it has received.Note that you can replace the NLTK tokenizer with the one you implement.<jupyter_code>def tokenizeRawData(fileid):
    """
        This function tokenizes a raw text document.
    """
    raw_article = reuters.raw(fileid).lower() # cover all words to lowercase
    tokenised_article = nltk.tokenize.word_tokenize(raw_article) # tokenize each Reuters articles
    return (fileid, tokenised_article) 
nltk.download('reuters')
tokenized_reuters =  dict(tokenizeRawData(fileid) for fileid in reuters.fileids())<jupyter_output><empty_output><jupyter_text>1.1. Removing Words with Non-alphabetic CharactersThe NLTK's built-in  word_tokenize function tokenizes a string to split off punctuation other than periods.Not only does it return words with alphanumerical characters, but also punctuations. Let's take a look at one Reuters articles,<jupyter_code>tokenized_reuters['training/1684']<jupyter_output><empty_output><jupyter_text>Let's Assume that we are interested in words containing alphabetic characters only and would like to remove all the other tokensthat contain digits, punctuation and the other symbols.Removing all the non-alphabetic words from the vocabulary isusually required in some text analysis tasks, such as Topic Modelling thatlearns the semantic meaning of documents.It can be easily done with the  isalpha() function. isalpha()checks whether the string consists of alphabetic characters only or not.This method returns true if all characters in the string are in the alphabet and there is at least one character, false otherwise.If you would like to keep all words with alphanumeric characters, you can use isalnum(). Refer to Python's [built-in types](https://docs.python.org/2/library/stdtypes.html) for more detail.Indeed, you can construct your tokenizer in a way such that the tokenizer only extracts words with either alphabetic or alphanumerical characters, as we discussed in the previous chapter.We will leave this as a simple exercise for you to do on your own.<jupyter_code>for k, v in tokenized_reuters.items():
    tokenized_reuters[k] = [word for word in v if word.isalpha()]

tokenized_reuters['training/1684']<jupyter_output><empty_output><jupyter_text>Now you should have derived much cleaner text for each Reuters article.Let's check how many types we have in the whole corpus and the lexical diversity (i.e., the average number of times a type apprearing in the collection.)<jupyter_code>from __future__ import division
from itertools import chain

words = list(chain.from_iterable(tokenized_reuters.values()))
vocab = set(words)
lexical_diversity = len(words)/len(vocab)
print ("Vocabulary size: ",len(vocab),"\nTotal number of tokens: ", len(words), \
"\nLexical diversity: ", lexical_diversity)<jupyter_output><empty_output><jupyter_text>There are about 1.27 million word tokens in the tokenized Reuters corpus.The vocabulary size is 27,944, which is still quite large according to our knowledge of this corpus.The lexical diversity tells us that words occur on average about 46 times each.You might think thatthere could still be words that occur very frequently, such as stopwords,and those that only occur once or twice.For example, if an article "the" appears in almostevery document in a corpus,it might not help you at all and would only contribute noise.Similarly if a word appears only once in a corpus or only in one document of the corpus,it could carry little useful information for downstream analysis.Therefore, we would better remove those words from the vocabulary, whichwill benefit the text analysis algorithms in terms of reducing running time andmemory requirement, and improving their performance.To do so, we need to further explore the corpus by computing some simplestatistics.Note that we introduced two new Python libraries in the code above.They are[`__future__`](https://docs.python.org/2/library/__future__.html) and [`itertools`](https://docs.python.org/2/library/itertools.html). The first statement in the code makes sure that Python switches to always yielding a real result.Thus if you divide two integer values, you will not get for example. ````    1/2 = 0    3/2 = 1````Instead, you will have```    1/2 = 0.5    3/2 = 1.5```The second statement imported a  chain() iterator from the  itertools module.We use the iterator to join all the words in all the Reuters articles together.It works as```python   for wordList in tokenized_reuters.values():       for word in wordList:           yield word```  1.2. Removing the Most and Less Frequent WordsIt is quite useful for us to identify the words that are most informative about the sematic meaning of the text regardless of syntax.One common statistics often used in text processing is frequency distribution.It can tell us how frequent a word is in a given corpus in terms of either term frequency or document frequency.Term frequency counts the number of times a word occurs in the whole corpus regardless which document it is in.Frequency distribution based on term frequency tells us how the total number of word tokens are distributed across all the types.NLTK provides a built-in function `FreqDist` to compute this distribution directly from a set of word tokens.<jupyter_code>from nltk.probability import *
fd_1 = FreqDist(words)<jupyter_output><empty_output><jupyter_text>What are the most frequent words in the corpus?we can use the  most_common function to print out the most frequent words together with their frequencies.<jupyter_code>fd_1.most_common(25)<jupyter_output><empty_output><jupyter_text>The list above contains the 25 most frequent words.You can see that it is mostly dominated by the little words of the English language which have important grammatical roles.Those words are articles, prepositions, pronouns, auxiliary webs, conjunctions, etc.They are usually referred to as function words in linguistics, which tell us nothing about the meaning of the text.What proportion of the text is taken up with such words?We can generate a cumulative frequency plot for themusing  fd.plot(25, cumulative=True).If you set  cumulative to  False, it will plot the frequencies of these 25 words.These 25 words account for about 33% of the while Reuters corpus.<jupyter_code>fd_1.plot(25, cumulative=True)<jupyter_output><empty_output><jupyter_text>What are the most frequent words in terms of document frequency?Here we are going to count how many documents a word appears in, which is referred to as document frequency.Instead of writing nested FOR loops to count the document frequency for each word,we can use  FreqDist() jointly with  set() as follows:1. Apply  set() to each Reuters article to generate a set of unique words in the article and save all sets in a list```python    [set(value) for value in tokenized_reuters.values()]```2. Similar to what we have done before, we put all the words in a list using  chain.from_iterable and pastit to  FreqDist.The first step makes sure that each word in an article appears only once, thus the total number of times a word appears in all the sets is equal to the number of documents containing that word.<jupyter_code>words_2 = list(chain.from_iterable([set(value) for value in tokenized_reuters.values()]))
fd_2 = FreqDist(words_2)
fd_2.most_common(25)<jupyter_output><empty_output><jupyter_text>What you will find is that the majority of the most frequent words according to their document frequecy are still functional words.Therefore, the next step is to remove all the stopwords.  1.2.1 Ignoring StopwordsWe often remove function words from the text completely for most text analysis tasks.Instead of using the built-in stopword list of NLTK, we use a much rich stopword list.<jupyter_code>!pip install wget
import wget

link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/stopwords_en.txt'

DataSet = wget.download(link_to_data)

!ls
stopwords = []
with open('stopwords_en.txt') as f:
    stopwords = f.read().splitlines()
tokenized_reuters_1 = {}
for fileid in reuters.fileids():
    tokenized_reuters_1[fileid] = [w for w in tokenized_reuters[fileid] if w not in stopwords]<jupyter_output><empty_output><jupyter_text>The list comprehension ```python    [w for w in tokenized_reuters[fileid] if w not in stopwords]```says: For each word in each Reuters article, keep the word if the word is not contained in the stopword list.Checking for membership of a value in a list takes time proportional to the list's length in the average and worst cases. It causes the above code to run quite slow as we need to do the check for every word in each Reuters articleand the size of the stopword list is large.However, if you have hashable items, which means both the item order and duplicates are disregarded, Python `set` is better choice than `list`. The former runs much faster than the latter in terms of searchinga large number of hashable items. Indeed, `set` takes constant time to check the membership.Let's try converting the stopword list into a stopword set, then search to remove all the stopwords.Please also note that if you try to perform iteration, `list` is much better than `set`.<jupyter_code>stopwordsSet = set(stopwords)
for fileid in reuters.fileids():
    tokenized_reuters[fileid] = [w for w in tokenized_reuters[fileid] if w not in stopwordsSet]<jupyter_output><empty_output><jupyter_text>In the above stopping process, 481 stopwords have been removed from the vocabulary. You might wonder what those removed words are. It is quite easy to check those words by differentiating the vocabulary before and after stopping.<jupyter_code>words_3 = list(chain.from_iterable(tokenized_reuters.values()))
fd_3 = FreqDist(words_3)
list(vocab - set(fd_3.keys()))<jupyter_output><empty_output><jupyter_text>Beside stopwords, there might some other words that occur quite often as well.<jupyter_code>fd_3.most_common(10)<jupyter_output><empty_output><jupyter_text>Before we decide to remove those words from our vocabulary, it might be worth checking what those words mean and the context of those words. Fortunately NLTK provides a `concordance`function in the `nltk.text` module. A concordance view shows us every occurrence of a given word, together with the corresponding context. For example,<jupyter_code>nltk.Text(reuters.words()).concordance('mln')
nltk.Text(reuters.words()).concordance('net')<jupyter_output><empty_output><jupyter_text>After reviewing those words, you might also want to remove them from the vocabulary. We will leave it as an excersie for you to do on your own.  1.2.2 Remove Less Frequent WordsIf the most common words do not benefit the downstream text analysis tasks, except for contributing noises,how about the words that occur once or twice?Here another interesting statistic to look at is the frequency of the frequencies of word types in a given corpus.We would like to see how many words appear only once, how many words appear twice, how manywords appear three times, and so on.<jupyter_code>ffd = FreqDist(fd_3.values())
from pylab import *
y = [0]*14
for k, v in ffd.items():
     if k <= 10:
        y[k-1] = v
     elif k >10 and k <= 50:
        y[10] =  y[10] + v
     elif k >50 and k <= 100:
        y[11] =  y[11] + v
     elif k > 100 and k <= 500:
        y[12] =  y[12] + v
     else:
        y[13] =  y[13] + v
x = range(1, 15) # generate integer from 1 to 14
ytks =list(map(str, range(1, 11))) # covert a integer list to a string list
ytks.append('10-50')
ytks.append('51-100')
ytks.append('101-500')
ytks.append('>500')
barh(x,y, align='center')
yticks(x, ytks)
xlabel('Frequency of Frequency')
ylabel('Word Frequency')
grid(True)<jupyter_output><empty_output><jupyter_text>The horizontal bar chart generated above shows how many word types occur with a certain frequency.There are 241 types occurring over 500 times and therefore individually accounting for about 1% ofthe vocabulary. However, on the other extreme, more than one-third of the word types occur only once in the Reuters corpus.Note that the majority of word types occur quite infrequently given the size of the whole corpus (i.e., 721,371 word tokens):about 78% of the word types occur 10 times or less.Similarly, you can also look at the bar chart based on the document frequency. Try it by yourself!Let's further remove those words that occur only once. To get those words, you can write the code like```python    lessFreqWords = set([k for k, v in fdist.items() if v < 2])```or choose to use `hapaxes()` function.<jupyter_code>lessFreqWords = set(fd_3.hapaxes())

def removeLessFreqWords(fileid):
    return (fileid, [w for w in tokenized_reuters[fileid] if w not in lessFreqWords])

#pool = mp.Pool(4)
#tokenized_reuters = dict(pool.map(removeLessFreqWords, reuters.fileids()))
tokenized_reuters = dict(removeLessFreqWords(fileid) for fileid in reuters.fileids())<jupyter_output><empty_output><jupyter_text>Now, you should have a pretty clean set of Reuters articles, each of which is stored as a list of word tokens.Let's further print out some statistics that summarize this corpus.<jupyter_code>import numpy as np
words = list(chain.from_iterable(tokenized_reuters.values()))
vocab = set(words)
print ("Vocabulary size: ",len(vocab))
print ("Total number of tokens: ", len(words))
print ("Lexical diversity: ", lexical_diversity)
print ("Total number of articles:", len(tokenized_reuters))
lens = [len(value) for value in tokenized_reuters.values()]
print ("Average document length:", np.mean(lens))
print ("Maximun document length:", np.max(lens))
print ("Minimun document length:", np.min(lens))
print ("Standard deviation of document length:", np.std(lens))<jupyter_output><empty_output><jupyter_text>It is interesting that the minimun document length is 0. There must be some Reuters articles that are extremely short,after tokenization and stopping, there are no words left. Can you check those documents to see what they look like?  Part 2. Building Vector RepresentationAfter text pre-processing has been completed, each individual document needs to be transformed into some kind of numeric representation that can be input into most NLP and text mining algorithms.For example, classification algorithms, such as Support Vector Machine, can only take data in a structured and numerical form. They do not accept free languge text.The most popular structured representation of text is the vector-space model, which represents textas a vector where the elements of the vector indicate the occurence of words within the text.The vector-space model makes an implicit assumption that the order of words in a text document are not asimportant as words themselves, and thus disregarded.This assumpiton is called [**Bag-of-words**](https://en.wikipedia.org/wiki/Bag-of-words_model).Given a set of documents and a pre-defined list of words appearing in those documents (i.e., a vocabulary), you can compute a vector representation for each document.This vector representation can take one of the following three forms:* a binary representation,* an integer count,* and a float-valued weighted vector.To highlight the difference among the three approaches, we use a very simple example as follows:```    document_1: "Data analysis is important."    document_2: "Data wrangling is as important as data analysis."    document_3: "Data science contains data analysis and data wrangling."```The three documents contain 20 tokens and 9 unique words.Those unique words are sorted alphabetically with total counts:```     'analysis': 3,     'and': 1,     'as': 2,     'contains': 1,     'data': 6,     'important': 2,     'is': 2,     'science': 1,     'wrangling': 2```Given the vocabulary above, both the binary and the integer count vectors are easy to compute.A binary vector stores 1s for the word that appears in a document and 0s for the other words inthe vocabulary,whereas a count vector stores the frequency of each word appearing in the document.Thus, the binary vector representations for the three documents above are      ||'analysis'|'and'|'as'|'contains'|'data'|'important'|'is'|'science'|'wrangling'|   |-|-|-|-|-|-|-|-|-|   |document 1:|1|0|0|0|1|1|1|0|0|   |document 2:|1|0|1|0|1|1|1|0|1|   |document 3:|1|1|0|1|1|0|0|1|1|The count vector representations for the same documents would look as follows:   ||'analysis'|'and'|'as'|'contains'|'data'|'important'|'is'|'science'|'wrangling'|   |-|-|-|-|-|-|-|-|-|   |document 1:|1|0|0|0|1|1|1|0|0|   |document 2:|1|0|2|0|2|1|1|0|1|   |document 3:|1|1|0|1|3|0|0|1|1|Instead of using the two vector representations above, most existing text analysis algorithms, like document classification and information retrieval, prefer representing documents as weighted vectors.The raw term frequency is often replaced with a weighted term frequencythat indicates how important a word is in a particular document.There are many different term weighting schemes online.To store each document as a weighted vector, we first need to choose a weighting scheme. The most popular scheme is the TF-IDF weighting approach. TF-IDF stands for term frequency-inverse document frequency. The term frequency for a word is the number of times the word appears in a document. In the preceding example, the term frequency in Document 2 for “data” is 2, since it appears twice in the document. Document frequency for a word is the number of documents that contain the word; it would also be 3 for “data” in the collection of the three preceding documents. The Wikipidia entry on [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) lists a number of variants of TF-IDF. One variant is reproduced here$$tf\cdot idf(w,d) = tf(w, d) * idf(w)$$where $$tf(w,d)\,=\, \sum_{i}^{|d|} 1_{w = w_{d,i}}$$and$$idf(w) = log\left(\frac{|D|}{|d \in D: w \in d |}\right)$$The assumption behind TF-IDF is that words with high term frequency should receive high weight unless they also have high document frequency. Stopwords are the most commonly occurring words in the English language. They often occur many times within a single document, but they also occur in nearly every document. These two competing effects cancel out to give them low weights,as those very common words carry very little meaningful information about the actual contents of the document.Therefore, the TF-IDF weights for stopwords are almost always 0.With the TF-DF formulas above,the weighted vector representations for the example documents are computed as||'analysis'|'and'|'as'|'contains'|'data'|'important'|'is'|'science'|'wrangling'|   |-|-|-|-|-|-|-|-|-|   |document 1:|0|0|0|0|0|0.176|0.176|0|0|   |document 2:|0|0|0.954|0|0|0.176|0.176|0|0.176|   |document 3:|0|0.477|0|0.477|0|0|0|0.477|0.176| Given the cleaned up Reuters documents, how can we generate those vectors for each documents? Unfortunately, NLTK does not implement methods that directly produce those vectors.Therefore, we will either write our own code to compute them or appeal to other data analysis libraries.Here we are going to use [scikit-learn](http://scikit-learn.org/stable/index.html), an open source machine learning library for Python.If you use Anaconda, you should already have scikit-learn installed, otherwise you will need to [install it](http://scikit-learn.org/stable/install.html) by following the instruction on its official website.Although scikit-learn features various classification, regression and clustering algorithmswe are particularly interested in its feature extraction module, [sklearn.feature_extraction](http://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.feature_extraction).This module is often used to "extract features in a format supported by machine learning algorithms from datasets consisting of formats such as text and image." Please refer to its documentation on text feature extraction,section 4.2.3 of [Feature Extraction](http://scikit-learn.org/stable/modules/feature_extraction.htmltext-feature-extraction). We will demonstrate the usage of the following two classes:* [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.htmlsklearn.feature_extraction.text.CountVectorizer): It converts a collection of text documents to a matrix of token counts. * [TfidfVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.htmlsklearn.feature_extraction.text.TfidfVectorizer):It converts a collection of raw documents to a matrix of TF-IDF features. 2.1 Creating Count VectorsLet's start with generating the count vector representation for each Reuters document.Initialise the "CountVector" object: since we have pre-processed all the Reuters documents, the parameters, "tokenizer", "preprocessor" and "stop_words" are set to their default value, i.e., None.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(analyzer = "word")<jupyter_output><empty_output><jupyter_text>Next, transform Reuters articles into feature vectors. `fit_transform` does two things: First, it fits the model and learns the vocabulary; second it transforms the text data into feature vectors. Please note the input to `fit_transform` should be a list of strings. Since we have stored each tokenised article as a list of words, we concatenate all the words in the list and separatethem with white spaces. The following code will do that:```python[' '.join(value) for value in tokenized_reuters.values()]```Then, we input this list of strings into `fit_transform`,<jupyter_code>data_features = vectorizer.fit_transform([' '.join(value) for value in tokenized_reuters.values()])
print (data_features.shape)<jupyter_output><empty_output><jupyter_text>The shape of document-by-word matrix should be 10788 * 17403. However, in order to save such a matrix in memory but also to speed up algebraic operations on the matrix,scikit-learn implements matrix/vector in a sparse representation.Let's check the count vector for the first article, i.e., 'training/1684'.<jupyter_code>vocab2 = vectorizer.get_feature_names()
for word, count in zip(vocab, data_features.toarray()[0]):
    if count > 0:
        print (word, ":", count)<jupyter_output><empty_output><jupyter_text>Another way to get the count list above is to use `FreqDist`.<jupyter_code>FreqDist(tokenized_reuters['training/1684'])<jupyter_output><empty_output><jupyter_text>Note that the vocabulary you just got with `vectorizer.get_feature_names()`  shoud be exactly the sameas the one you got in section 1.<jupyter_code>list(vocab-set(vocab2))<jupyter_output><empty_output><jupyter_text>2.2 Creating TF-IDF VectorsSimilar to the use of `CountVector`, we first initialise a `TfidfVectorizer` object by only specifying the value of "analyzer", and then covert the Reuters data into a list of strings, each of which correspondsto a Reuters articles.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(analyzer = "word")
tfs = tfidf.fit_transform([' '.join(value) for value in tokenized_reuters.values()])
tfs.shape<jupyter_output><empty_output><jupyter_text>Let's print out the weighted vector for the first document.<jupyter_code>vocab = vectorizer.get_feature_names()
for word, weight in zip(vocab, tfs.toarray()[0]):
    if weight > 0:
        print (word, ":", weight)<jupyter_output><empty_output><jupyter_text>So now we have converted all the Reuters articles into feature vectors. We can use those vectors to, for example,* compute the similarity between two articles, * search articles for a given query* do other advance text analysis, such as document classification and clustering.Assume that we have a new document, how can we get its TF-IDF vector.We do this by using the transform function as follows.We have randomly chosen a sentence from [a recent Reuters news](http://www.reuters.com/article/us-usa-election-idUSKCN0W346T).<jupyter_code>str = """
the former secretary of state hoped to win enough states to take a big step toward wrapping up her nomination fight
with a democratic senator from Vermont.
"""
response = tfidf.transform([str])
for col in response.nonzero()[1]:
    print (vocab[col], ' - ', response[0, col])<jupyter_output><empty_output><jupyter_text>Note that the text above is not included in the trained TF-IDF model with the 'transform' function, unless the `fit_transform` function is called,Both `CountVectorizer` and `TfidfVectorizer` come with their own options to automatically do pre-processing, tokenization, and stop word removal -- for each of these, instead of using their default value (i.e., None),we could customise the two vectorizer classes by either using a built-in method or specifying our own function.See the function documentation for more details.However, we wanted to write our own function for clean the text data in this chapter to show you how it's done step by step.  Part 3. Saving Pre-processed Text to a FileThe pre-processed text needs to be saved in a proper format so that it can be easily used by the downstream analysis algorithm. There are a couple of ways of dumping the pre-processed text data into txt files. For example, use one txt file to store the tokenized documents. The tokens in a document are stored in one row in the txt file, and are separated with a given delimiter, e.g., whitespace. In this case, the downstream text analyser needs to re-construct the vocabulary.<jupyter_code>import wget

link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/reuters_1.txt'

DataSet = wget.download(link_to_data)

# import wget

link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/reuters_2.txt'

DataSet = wget.download(link_to_data)

!ls
out_file = open("reuters_1.txt", 'w')
for d in tokenized_reuters.values():
    out_file.write(' '.join(d) + '\n')
out_file.close()<jupyter_output><empty_output><jupyter_text>You can also save vocabulary in a separate file, and assign a fixed integer id to each word in the vocabulary. What text analysers usually do is to use the index of each word in the vocabulary as its integer id.Given the vocabulary, each document can be represented as a sequence of integers that correspond to the tokens,or in the following sparse form:```    word_index:word count```for example,<jupyter_code>out_file = open("reuters_2.txt", 'w')
vocab = list(vocab)

vocab_dict = {}
i = 0
for w in vocab:
    vocab_dict[w] = i
    i = i + 1

for d in tokenized_reuters.values():
    d_idx = [vocab_dict[w] for w in d]
    for k, v in FreqDist(d_idx).items():
        out_file.write("{}:{} ".format(k,v))
    out_file.write('\n')
out_file.close()<jupyter_output><empty_output><jupyter_text>Part 4. Extracting Other FeaturesIt is common for most text analysis tasks to treat documents as bags-of-words, which can significantly simplify the inference procedure of text analysis algorithms. However, things always have pros and cons. The bag-of-words representation loses lots of information encoded in either syntax or word order (i.e., dependencies between adjacent words in sentences.). For example, representing a document as a collection of unigrams effectively disregards any word order dependence,which fails to capture phrases and multi-word expressions. A similar issue has been mentioned in section 2.1. of Chapter 2. In this section, we are going to show you how to* use Part-of-Speeching (POS) tagging to extract specific word groups, such as all nouns, verbs, etc.,* extract n-grams,*  and extract collocationsThese features can be further used to enrich the representation of a document. 4.1 Extracting Nouns and VerbsIt is easy for human to tell the difference between nouns, verbs, adjectives and adverbs, as we have learnt them back in elementary school.However, how can we automatically classify words into their parts of speech (i.e., lexical categories or word classes) and label them accordingly with computer program? This section is not going to discuss how to determine the category of a word from a linguistic perspective.Instead it demonstrates the use of some existing POS taggers to extract words in a specific lexical category.It has been proven that words together with their part-of-speech (POS) are quite useful for many language processing tasks. In NLP, the process of labelling words with their corresponding part-of-speech (POS) tags is known as [POS tagging](https://en.wikipedia.org/wiki/Part-of-speech_tagging).A POS tagger processes a sequence of words and attaches a POS tag to each word based on both its definition and its context. There are many POS taggers available online, such as [Sandford POS tagger](http://nlp.stanford.edu/software/tagger.shtml). We are going to use the one implemented by NLTK.<jupyter_code>example_sent = 'A POS tagger processes a sequence of words and attaches a POS tag to each \
word based on both its definition and its context'
text = nltk.word_tokenize(example_sent)
tagged_sent = nltk.tag.pos_tag(text)
print (tagged_sent)<jupyter_output><empty_output><jupyter_text>If you are seeing these tags for the first time, you will wonder what these tags mean. You can find the specification of all the tags [here](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). NLTK provides documentation for each tag, which can be queried using the tag, e.g.,<jupyter_code>nltk.download('tagsets')
print (nltk.help.upenn_tagset('NNP'))
print (nltk.help.upenn_tagset('IN'))
print (nltk.help.upenn_tagset('PRP$'))<jupyter_output><empty_output><jupyter_text>The example sentence has been processed by `pos_tag` into a list of tuples, each of which is a pair of a word and its POS tag. We see that 'a' is 'DT', a determiner; 'its' is 'PRP$', a possessive pronoun; 'and' is 'CC', a coordinating conjunction, 'words' is 'NNS', a noun in the plural form, and so on. Note that several of the corpora included in NLTK have been tagged for their POS. Please click [here](http://www.nltk.org/howto/corpus.htmltagged-corpora) to see how to access those tagged corpora.Here is an example of using the `tagged_words` function to retrieve all words in Brown corpus with their tags.<jupyter_code>nltk.download('brown')
nltk.corpus.brown.tagged_words()<jupyter_output><empty_output><jupyter_text>Please note that the collection of tags is known as a tag set. There are many different conventions for tagging words.Therefore, tag sets can vary among different tasks.What we used above is the Penn Treebank tag set.Let's change the tag set to the Universal POS tag set, and print the Brown corpus again.You will find different tags are used.<jupyter_code>nltk.download('universal_tagset')
nltk.corpus.brown.tagged_words(tagset='universal')<jupyter_output><empty_output><jupyter_text>If you would like to learn more about POS tagging, please refer to [1].Given the tagged text, you can easily identify all the nouns, verbs, etc.Nouns generally refer to people, places, things, or concepts, e.g., Monash, Melbourne, university, data, and science. Nouns can appear after determiners and adjectives, and can be the subject or object of the verb.Now how can we extract all the nouns from a text?Assume we use the Penn Treebank tag set.Here are all the tags for nouns:```    NN    Noun, singular or mass    NNS   Noun, plural    NNP   Proper noun, singular    NNPS  Proper noun, plural```It is not hard to see all the tags above start with 'NN'.Thus, we can iterate over all the words and check if their tag string starts with 'NN'.<jupyter_code>all_nouns = [w for w,t in tagged_sent if t.startswith('NN')]
all_nouns<jupyter_output><empty_output><jupyter_text>Similarly, you will find that all the verb tags start with 'VB', see```    VB	Verb, base form    VBD   Verb, past tense    VBG   Verb, gerund or present participle    VBN   Verb, past participle    VBP   Verb, non-3rd person singular present    VBZ   Verb, 3rd person singular present```Thus,<jupyter_code>all_verbs = [w for w,t in tagged_sent if t.startswith('VB')]
all_verbs<jupyter_output><empty_output><jupyter_text>Unfortunately, the Reuters corpus that we have been using, has no built-in POS tags. But you can get sentences from Reuters corpus, and then you can get the POS tags.  4.2 Extracting N-grams and CollocationsBesides unigrams that we have been working on so far,N-grams of texts are also extensively used in various text analysis tasks.They are basically contiguous sequences of `n` words from a given sequence of text.When computing the n-grams you typically move a fixed size window of size nwords forward.For example, for the sentence"Laughter is like a windshield wiper."if N = 2 (known as bigrams), the n-grams would be:```    Laughter is     is like     like a     a windshield     windshield wiper```So you have 5 bigrams in this case. Notice that the generative process aboveessentially moves one word forward to generate the next bigram.If N = 3 (known as trigrams), the n-grams would be:```    Laughter is like     is like a     like a  windshield    a  windshield wiper```What are N-grams used for? They can be used to build n-gram language model thatcan be further used for speech recognition, spelling correction, entity detection, etc.In terms of text mining tasks, n-grams is used for developing features for classification algorithms, such as SVMs, MaxEnt models, Naive Bayes, etc.The idea is to expand the unigram feature space with n-grams.But please notice thatthe use of bigrams and trigrams in your feature space may not necessarily yield significant performanceimprovement. The only way to know this is to try it! Extracting from a text a list of n-gram can be easily accomplished with function `ngram()`:<jupyter_code>from nltk.util import ngrams
bigrams = ngrams(reuters.words(), n = 2)
fdbigram = FreqDist(bigrams)
fdbigram.most_common()<jupyter_output><empty_output><jupyter_text>Collocations are expressions of multiple words that commonly co-occur. >Finding collocations requires first calculating the frequencies of words andtheir appearance in the context of other words. Often the collection of wordswill then requiring filtering to only retain useful content terms. Each ngramof words may then be scored according to some association measure, in orderto determine the relative likelihood of each ngram being a collocation. (Quoted from [here](http://www.nltk.org/_modules/nltk/collocations.html))For example, to extract bigram collocations, we can firstly extract bigrams then get the commonly co-occurring ones by ranking the bigrams by some measures. A commonly used measure is [Pointwise Mutual Information](https://en.wikipedia.org/wiki/Pointwise_mutual_information) (PMI). The following code will find the best 50 bigrams using the PMI scores.<jupyter_code>bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = nltk.collocations.BigramCollocationFinder.from_words(reuters.words())
finder.nbest(bigram_measures.pmi, 50)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07379267277268943, 'fraction_numerical': 0.011084512905911741, 'mean_word_length': 5.178938906752411, 'pattern_counts': {'":': 2, '<': 146, '<?xml version=': 0, '>': 150, 'https://': 13, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '50330', 'license': 'MIT', 'n_tokens_mistral': 11055, 'n_tokens_neox': 10314, 'n_words': 4885, 'path': 'Jupyter/SIT742P04B-TextFeatures.ipynb', 'repo_name': 'jilliant/sit742', 'size': '38432'}