<jupyter_start><jupyter_text>ISRC Python Workshop: Text Analytics in Python___Text Analytics in Python___  @author: <NAME>@email: <EMAIL> ---  Introduction Text data has been one of the most important sources across areas. Doctoral notes, students comments/feedback, product reviews, as well as social media text (tweets/facebook posts), are valuable resources. In this workshop, we will use a toy dataset to go though a common text analytics procedure in Python. Let's first load a toy dataset (I did a little bit of preprocessing) used in the paper: > <NAME>, <NAME>, <NAME>. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015).This is actually the testing set (for algorithm evaluation) but we will just use this as an illustration for text analytics procedures.<jupyter_code>import pandas as pd
df = pd.read_csv('sample-data/ag_news.csv')
df.head(2)<jupyter_output><empty_output><jupyter_text>There are four columns: - class index and class names that annotate the content type- title and description of each news piece Let's combine the title and description into a column called `content` and drop unneccessary columns.<jupyter_code>df['content'] = df['title'] + '. ' + df['description']
df.drop(columns=['title', 'description'], inplace=True)
df.head(2)<jupyter_output><empty_output><jupyter_text>---  Document Representation While we as human can understand the text, machines can hardly know text as it is. A classic way to mathematically represent text data is to convert them into [vector spaces](https://en.wikipedia.org/wiki/Vector_space_model). Specifically, each documeent will be stored as a vector, where each element is a ___weight___ for one term. A simple and commonly used approach is to tokenize documents and use term frequencies or [term frequency-inverse document frequency (TFIDF)](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) as weights.  Bag-of-Words (BOW) While more sophisticated approaches are available, BOW representation is still very popular due to its simplicity. In this style, we can forget word orders and treat each document as ___a bag of words___. While this is a very strong assumption, it still makes sense -- we can understand a sentence even if the words are randomly ordered. For example, we can easily understand the following sentence:> stting a chair is cat there on In Python, we can use a `numpy.ndarray` or `list` to save this information:<jupyter_code>sentence = ['stting', 'a', 'chair', 'is', 'cat', 'there', 'on']
sentence<jupyter_output><empty_output><jupyter_text>Tokenize Let's now apply this onto our toy dataset. In English text, we can simply split each document by spaces. Besides, we usually may want to remove puncutations because they do not provide valuable information. Last but not least, we may want to exclude some very common words (e.g., `the`, `we`, `you`, `is`, etc.), called [___stop words___](https://en.wikipedia.org/wiki/Stop_words).<jupyter_code>from string import punctuation
print(punctuation)<jupyter_output>!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~<jupyter_text>If there's no special needs, we can simply use a common stop word list from `nltk`.<jupyter_code>from nltk.corpus import stopwords
eng_stopwords = stopwords.words('english')
eng_stopwords[:10]<jupyter_output><empty_output><jupyter_text>Now we can start to convert the contents into bag of words! To do this, we may utilize the package [`nltk`'s tokenize package](http://www.nltk.org/api/nltk.tokenize.html)<jupyter_code>import nltk
df.head(2)
# also convert them to lower case
bow = [nltk.word_tokenize(content.lower()) for content in df['content'].values]
# show the first 1
print('; '.join(bow[0]))
len(bow)
bow[0]<jupyter_output><empty_output><jupyter_text>Let's remove the stop words and punctuations. Note that there are some very short words of length 1. I will typically remove them as well if they do not have special meanings. Also, I will remove pure numbers by `str.isdigit`. Note that this may not work well but for simplicity, we will just go with it. See more dicussion on this issue [here](https://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-float?page=1&tab=votestab-top).<jupyter_code>min_length = 3 # define the customized minimum length
bow = [[w for w in d if w not in punctuation and w not in eng_stopwords and not w.isdigit()] for d in bow]
print('; '.join(bow[0]))<jupyter_output>fears; n; pension; talks; unions; representing; workers; turner; newall; say; 'disappointed; talks; stricken; parent; firm; federal; mogul<jupyter_text>As we can, sometimes we need further data cleaning to remove punctuations in words. In this example, we want to remove the quote in the word "disappointed". In this case, we can utilize [`string.translate`](https://stackoverflow.com/questions/34293875/how-to-remove-punctuation-marks-from-a-string-in-python-3-x-using-translate):<jupyter_code># do not translate anything, except for removing all punctuations
trans = str.maketrans('', '', punctuation)
bow = [[w.translate(trans).strip() for w in d] for d in bow]
bow = [[w for w in d if len(w) >= min_length] for d in bow]
print('; '.join(bow[0]))
for i in range(1, 5):
    print('; '.join(bow[i]))
    print('---------------')<jupyter_output>race; second; private; team; sets; launch; date; human; spaceflight; spacecom; spacecom; toronto; canada; secondteam; rocketeers; competing; million; ansari; prize; contest; forprivately; funded; suborbital; space; flight; officially; announced; firstlaunch; date; manned; rocket
---------------
company; wins; grant; study; peptides; company; founded; chemistry; researcher; university; louisville; grant; develop; method; producing; better; peptides; short; chains; amino; acids; building; blocks; proteins
---------------
prediction; unit; helps; forecast; wildfires; barely; dawn; mike; fitzpatrick; starts; shift; blur; colorful; maps; figures; endless; charts; already; knows; day; bring; lightning; strike; places; expects; winds; pick; moist; places; dry; flames; roar
---------------
calif; aims; limit; farmrelated; smog; southern; california; smogfighting; agency; went; emissions; bovine; variety; friday; adopting; nation; first; rules; reduce; air; pollution; dairy; cow; manure
-------[...]<jupyter_text>Stemming/Lemmetization Finally, it is noteworthy that one word can take different forms. For example, run can be ___run___, ___runs___, ___ran___, and ___running___. While they are different, they mean the same thing. There are two common methods to reduce a word back to the root form. The first one is called ___stemming___, where each word will bee reduced to its "stem". For example, the stem of the word ___fly___ will be ___fli___<jupyter_code>from nltk import PorterStemmer
stemmer = PorterStemmer()
for w in ['fly', 'papers', 'communication', 'community']:
    print(w, ': ', stemmer.stem(w))<jupyter_output>fly :  fli
papers :  paper
communication :  commun
community :  commun<jupyter_text>While this seems to be okay, sometimes it is hard to ___reverse___ the stemming result (e.g., both "communication" and "community" are transformed to "commun", although they mean very different things). A second choice can be ___lemmatization___, which makes a word to its [___lemma__ ](https://en.wikipedia.org/wiki/Lemma_(morphology)), or say canonical or dictionary form.<jupyter_code>from nltk import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
for w in ['flies', 'papers', 'communication', 'communities']:
    print(w, ': ', lemmatizer.lemmatize(w))<jupyter_output>flies :  fly
papers :  paper
communication :  communication
communities :  community<jupyter_text>For simplicity, let's just use stemming.<jupyter_code>bow = [[stemmer.stem(w) for w in d] for d in bow]
for i in range(1, 5):
    print('; '.join(bow[i]))
    print('---------------'*10)<jupyter_output>race; second; privat; team; set; launch; date; human; spaceflight; spacecom; spacecom; toronto; canada; secondteam; rocket; compet; million; ansari; prize; contest; forpriv; fund; suborbit; space; flight; offici; announc; firstlaunch; date; man; rocket
------------------------------------------------------------------------------------------------------------------------------------------------------
compani; win; grant; studi; peptid; compani; found; chemistri; research; univers; louisvil; grant; develop; method; produc; better; peptid; short; chain; amino; acid; build; block; protein
------------------------------------------------------------------------------------------------------------------------------------------------------
predict; unit; help; forecast; wildfir; bare; dawn; mike; fitzpatrick; start; shift; blur; color; map; figur; endless; chart; alreadi; know; day; bring; lightn; strike; place; expect; wind; pick; moist; place; dri; flame; roar
-----------------------------[...]<jupyter_text>Vector Space Model (VSM) Now that we have a bag of words, we can create vectors based on these items. Instead of using tokens/text, it is easier and better to just use integer indices. For example, `race` is the first word and therefore the number `1` maps to `race`. For this task, I like to use [`gensim`](https://radimrehurek.com/gensim/index.html), which has a library of very well written and convenient APIs, especially for [topic modeling](https://en.wikipedia.org/wiki/Topic_model) and [word2vec](https://rare-technologies.com/word2vec-tutorial/) algorithms:<jupyter_code>import gensim
dictionary = gensim.corpora.Dictionary(bow)
print(dictionary)
dictionary.token2id<jupyter_output><empty_output><jupyter_text>Mapping of tokens<jupyter_code>dictionary.token2id['disappoint'], dictionary.token2id['california']<jupyter_output><empty_output><jupyter_text>Upon creation of a dictionary that maps words to integers (and vice versa), we can transform our bag of words. Each document will be a list of tuples that contain token indices and frequencies.<jupyter_code>corpus = [dictionary.doc2bow(d) for d in bow]
corpus[0]<jupyter_output><empty_output><jupyter_text>TFIDF Sometimes term frequencies can be misleading. Just like the same reason we remove stop words, words that occur a lot across the whole corpus may not be informative. On the other hand, a word that shows up a lot in only a small portion instead of all can provide valuable informaiton on the contents of these texts. One way to resolve this potential problem is called term frequency-inverse document frequency (TFIDF), which is a product of TF and IDF. A common weighting scheme is $TFIDF(t,d, D) = freq_{t,d}\times log_2~\dfrac{N_D}{N_t}$, where $N_D$ is the number of documents; $N_t$ is the number of documents that contain the term $t$. See [here](https://en.wikipedia.org/wiki/Tf%E2%80%93idfTerm_frequency) for more details. We can try this on our data with [`gensim.models.tfidfmodel`](https://radimrehurek.com/gensim/models/tfidfmodel.html):<jupyter_code>from gensim.models import TfidfModel
model = TfidfModel(corpus)<jupyter_output><empty_output><jupyter_text>Apply the model on one document<jupyter_code>model[corpus[0]]<jupyter_output><empty_output><jupyter_text>---  Topic Modeling  Latent Dirichet Allocation (LDA) A very commonly used dimensionality reduction technique family is called ___topic modeling___. It assumes that each document is a mixture of topics, where each topic is a mixture of terms. One of the most successful algorithms is [___latent Dirichlet allocation___](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) (LDA), whose corresponding paper is:> <NAME>., <NAME>., & <NAME>. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), 993-1022. LDA is a generative model that does the reverse engineerging of document generation. It can be represented as a probablistic graphical model:![lda](https://upload.wikimedia.org/wikipedia/commons/4/4d/Smoothed_LDA.png) The generative process can be described as follows:- For each topic $k$, sample a multinomial distribution $\phi_k$ over words from the Dirichlet prior with parameter $\beta$- For each document $m$, sample a multinomial distribution $\theta_m$ over topics from the Dirichlet prior with parameter $\alpha$    - For each word $n$ in $m$:        - Sample a topic $z_{m,n}$ from the correponding topic distribution parameterized by $\theta_m$        - Sample a word $w_{m,n}$ from the correponding topic $z_{m,n}$'s word distribution parameterized by $\phi_{z_{m,n}}$  Parameters in LDA Generally, we need to control two hyperparamters of a LDA model:- Topic-word Dirichlet prior $\beta$- Document-topic Dirichlet prior $\alpha$The selection of these parameters are application dependent. Heuristically, people will choose $\alpha=\dfrac{50}{K}$ and $\beta=0.01$, as described in> <NAME>., and <NAME>. 2004. “Finding Scientific Topics,” Proceedings of the National Academy of Sciences (101:Supplement 1), National Academy of Sciences, pp. 5228–5235.It is also possible to infer these two hyperparameters given the data. The selection of $K$ totally depends on the context. It is also possible to select a topic number based on quantitative measures of topic modeling quality, but this is beyond the scope of this tutorial.For our toy sample set, we will just select $K=4$ because there are 4 labels:<jupyter_code>df.class_name.unique()<jupyter_output><empty_output><jupyter_text>Run LDA! Thanks to the convenient APIs by `gensim`, we can easily run [LDA in Python](https://radimrehurek.com/gensim/models/ldamodel.html):<jupyter_code>from gensim.models import LdaModel
lda = LdaModel(corpus, num_topics=4, id2word=dictionary, passes=10, 
               minimum_probability=0)<jupyter_output><empty_output><jupyter_text>Analysis on LDA results Let's take a look at the output of LDA. First, we can check if the topics make sense<jupyter_code>for _, topic_str in lda.show_topics():
    print(topic_str)
    print('------------'*10)<jupyter_output>0.012*"new" + 0.009*"reuter" + 0.009*"said" + 0.008*"price" + 0.007*"oil" + 0.007*"inc" + 0.007*"compani" + 0.006*"stock" + 0.006*"share" + 0.006*"report"
------------------------------------------------------------------------------------------------------------------------
0.007*"quot" + 0.007*"compani" + 0.007*"new" + 0.006*"said" + 0.005*"servic" + 0.005*"microsoft" + 0.005*"plan" + 0.004*"million" + 0.004*"say" + 0.004*"deal"
------------------------------------------------------------------------------------------------------------------------
0.010*"game" + 0.008*"win" + 0.007*"team" + 0.007*"first" + 0.007*"new" + 0.006*"season" + 0.005*"world" + 0.005*"one" + 0.004*"year" + 0.004*"final"
------------------------------------------------------------------------------------------------------------------------
0.009*"said" + 0.007*"reuter" + 0.007*"kill" + 0.006*"iraq" + 0.006*"presid" + 0.004*"offici" + 0.004*"two" + 0.004*"minist" + 0.004*"say" + 0.004*"afp"
--------------------[...]<jupyter_text>While we probably cannot say the topics are perfect, they are okay. We can interpret the topics as: business, sports, world, and sci/tech. For each document, we can check their topic distributions:<jupyter_code>i = 1000
lda.get_document_topics(corpus[i])<jupyter_output><empty_output><jupyter_text>We can see that topic 0, which is interpreted "business" topics dominate this document. We can check to see if this makes sense:<jupyter_code>df.loc[i]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15554404815845685, 'fraction_numerical': 0.017412130234966666, 'mean_word_length': 4.110817069136619, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 94, 'https://': 14, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '306672', 'license': 'MIT', 'n_tokens_mistral': 4703, 'n_tokens_neox': 4357, 'n_words': 1707, 'path': 'python-tutorial-uiowa/7-Text-Analytics-in-Python.ipynb', 'repo_name': 'ValRCS/RCS_Data_Analysis_Python_2019', 'size': '15503'}