<jupyter_start><jupyter_text>Latent Dirichlet Allocation application to topic modeling LDA is used to classify text in a document to a particular topic. It builds a topic per document model and words per topic model, modeled as Dirichlet distributions. * Each document is modeled as a multinomial distribution of topics and each topic is modeled as a multinomial distribution of words.* LDA assumes that the every chunk of text we feed into it will contain words that are somehow related. Therefore choosing the right corpus of data is crucial. * It also assumes documents are produced from a mixture of topics. Those topics then generate words based on their probability distribution. This is an **unsupervised learning** task The dataset is a list of over one million news headlines published over a period of 15 years.<jupyter_code>import pandas as pd
data = pd.read_csv('abcnews-date-text.csv', error_bad_lines=False);
# We only need the Headlines text column from the data
data_text = data[:300000][['headline_text']];
data_text['index'] = data_text.index

documents = data_text<jupyter_output><empty_output><jupyter_text>Looking into the dataset:<jupyter_code>print(len(documents))
documents[:5]<jupyter_output><empty_output><jupyter_text>Data Preprocessing * **Tokenization**: Split the text into sentences and the sentences into words. Lowercase the words and remove punctuation.* Words that have fewer than 3 characters are removed.* All **stopwords** are removed.* Words are **lemmatized** - words in third person are changed to first person and verbs in past and future tenses are changed into present.* Words are **stemmed** - words are reduced to their root form.<jupyter_code>'''
Loading Gensim and nltk libraries
'''
# conda install -c anaconda gensim
import gensim
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
from nltk.stem import WordNetLemmatizer, SnowballStemmer
from nltk.stem.porter import *
import numpy as np
np.random.seed(400)
import nltk
nltk.download('wordnet')<jupyter_output>[nltk_data] Downloading package wordnet to /home/thojo/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>Lemmatizer ExampleBefore preprocessing our dataset, let's first look at an lemmatizing example. What would be the output if we lemmatized the word 'went':<jupyter_code>print(WordNetLemmatizer().lemmatize('went', pos = 'v')) # past tense to present tense<jupyter_output>go<jupyter_text>Stemming<jupyter_code>stemmer = SnowballStemmer("english")
original_words = ['caresses', 'flies', 'dies', 'mules', 'denied','died', 'agreed', 'owned', 
           'humbled', 'sized','meeting', 'stating', 'siezing', 'itemization','sensational', 
           'traditional', 'reference', 'colonizer','plotted']
singles = [stemmer.stem(plural) for plural in original_words]

pd.DataFrame(data={'original word':original_words, 'stemmed':singles })
'''
Function to perform the pre processing steps on the entire dataset
'''
def lemmatize_stemming(text):
    return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos='v'))

# Tokenize and lemmatize
def preprocess(text):
    result=[]
    for token in gensim.utils.simple_preprocess(text) :
        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
            # Apply lemmatize_stemming() on the token, then add to the results list
            result.append( lemmatize_stemming( token ) )
    return result
'''
Preview a document after preprocessing
'''
document_num = 4310
doc_sample = documents[documents['index'] == document_num].values[0][0]

print("Original document: ")
words = []
for word in doc_sample.split(' '):
    words.append(word)
print(words)
print("\n\nTokenized and lemmatized document: ")
print(preprocess(doc_sample))<jupyter_output>Original document: 
['rain', 'helps', 'dampen', 'bushfires']


Tokenized and lemmatized document: 
['rain', 'help', 'dampen', 'bushfir']<jupyter_text>Time to preprocess all the news headlines using the [map](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.map.html) function from pandas to apply `preprocess()` to the `headline_text` column<jupyter_code># from pandas import map
#note: this is taking a few mins
docs = documents['headline_text']
processed_docs = docs.map( preprocess)
'''
Preview 'processed_docs'
'''
processed_docs[:10]<jupyter_output><empty_output><jupyter_text>Bag of words on the datasetCreate a dictionary from 'processed_docs' containing the number of times a word appears in the training set. `processed_docs` is passed to [`gensim.corpora.Dictionary()`](https://radimrehurek.com/gensim/corpora/dictionary.html).<jupyter_code>'''
Create a dictionary from 'processed_docs' containing the number of times a word appears 
in the training set using gensim.corpora.Dictionary
'''
dictionary = gensim.corpora.Dictionary(processed_docs)
'''
Checking dictionary created
'''
count = 0
for k, v in dictionary.iteritems():
    print(k, v)
    count += 1
    if count > 10:
        break<jupyter_output>0 broadcast
1 communiti
2 decid
3 licenc
4 awar
5 defam
6 wit
7 call
8 infrastructur
9 protect
10 summit<jupyter_text>** Gensim filter_extremes **[`filter_extremes(no_below=5, no_above=0.5, keep_n=100000)`](https://radimrehurek.com/gensim/corpora/dictionary.htmlgensim.corpora.dictionary.Dictionary.filter_extremes)Filtering out tokens that appear in* less than no_below documents (absolute number) or* more than no_above documents (fraction of total corpus size, not absolute number).* after (1) and (2), keep only the first keep_n most frequent tokens (or keep all if None).<jupyter_code>'''
Remove very rare and very common words:

- words appearing less than 15 times
- words appearing in more than 10% of all documents
'''
dictionary.filter_extremes(no_below=5, no_above=0.5, keep_n=100000)<jupyter_output><empty_output><jupyter_text>** Gensim doc2bow **[`doc2bow(document)`](https://radimrehurek.com/gensim/corpora/dictionary.htmlgensim.corpora.dictionary.Dictionary.doc2bow)* Converting document (a list of words) into the bag-of-words format = list of (token_id, token_count) 2-tuples. Each word is assumed to be a tokenized and normalized string (either unicode or utf8-encoded). No further preprocessing is done on the words in document; apply tokenization, stemming etc. before calling this method.<jupyter_code>'''
Creating the Bag-of-words model for each document i.e for each document we create a dictionary reporting how many
words and how many times those words appear. 
'''
# TODO
bow_corpus = [dictionary.doc2bow( doc ) for doc in processed_docs]
# for doc in processed_docs:
#     bow_corpus.append( dictionary.doc2bow( doc ) )
'''
Checking Bag of Words corpus for sample document --> (token_id, token_count)
'''
bow_corpus[document_num]
'''
Preview BOW for sample preprocessed document
'''
# Here document_num is document number 4310 which was checked earlier in the notebook 
bow_doc_4310 = bow_corpus[document_num]

for i in range(len(bow_doc_4310)):
    print("Word {} (\"{}\") appears {} time.".format(bow_doc_4310[i][0], 
                                                     dictionary[bow_doc_4310[i][0]], 
                                                     bow_doc_4310[i][1]))<jupyter_output>Word 76 ("bushfir") appears 1 time.
Word 113 ("help") appears 1 time.
Word 482 ("rain") appears 1 time.
Word 4016 ("dampen") appears 1 time.<jupyter_text>TF-IDF on document set While performing TF-IDF on the corpus is not necessary for LDA implemention using the gensim model, it is useful. TF-IDF expects a bag-of-words (integer values) training corpus during initialization. During transformation, it will take a vector and return another vector of the same dimensionality.*The author of Gensim dictates the standard procedure for LDA to be using the Bag of Words model.* ** TF-IDF stands for "Term Frequency, Inverse Document Frequency".*** It is a way to score the importance of words (or "terms") in a document based on how frequently they appear across multiple documents.* If a word appears frequently in a document, it's important. Give the word a high score. But if a word appears in many documents, it's not a unique identifier. Give the word a low score.* Therefore, common words like "the" and "for", which appear in many documents, will be scaled down. Words that appear frequently in a single document will be scaled up.In other words:* TF(w) = `(Number of times term w appears in a document) / (Total number of terms in the document)`.* IDF(w) = `log_e(Total number of documents / Number of documents with term w in it)`.<jupyter_code>'''
Create tf-idf model object using models.TfidfModel on 'bow_corpus' and save it
'''
from gensim import corpora, models

tfidf = models.TfidfModel(bow_corpus)
'''
Apply transformation to the entire corpus and call it 'corpus_tfidf'
'''

corpus_tfidf = [ tfidf[ doc ] for doc in bow_corpus ]
'''
Preview TF-IDF scores for our first document --> --> (token_id, tfidf score)
'''
from pprint import pprint
for doc in corpus_tfidf:
    pprint(doc)
    break<jupyter_output>[(0, 0.5959813347777092),
 (1, 0.39204529549491984),
 (2, 0.48531419274988147),
 (3, 0.5055461098578569)]<jupyter_text>LDA using Bag of Words Trying 10 topics in the document corpus.** Running LDA using all CPU cores to parallelize and speed up model training.**Some of the parameters to be tweaked are:* **num_topics** is the number of requested latent topics to be extracted from the training corpus.* **id2word** is a mapping from word ids (integers) to words (strings). It is used to determine the vocabulary size, as well as for debugging and topic printing.* **workers** is the number of extra processes to use for parallelization. Uses all available cores by default.* **alpha** and **eta** are hyperparameters that affect sparsity of the document-topic (theta) and topic-word (lambda) distributions. We will let these be the default values for now(default value is `1/num_topics`)    - Alpha is the per document topic distribution.        * High alpha: Every document has a mixture of all topics(documents appear similar to each other).        * Low alpha: Every document has a mixture of very few topics    - Eta is the per topic word distribution.        * High eta: Each topic has a mixture of most words(topics appear similar to each other).        * Low eta: Each topic has a mixture of few words.* ** passes ** is the number of training passes through the corpus. For  example, if the training corpus has 50,000 documents, chunksize is  10,000, passes is 2, then online training is done in 10 updates:     * `1 documents 0-9,999 `    * `2 documents 10,000-19,999 `    * `3 documents 20,000-29,999 `    * `4 documents 30,000-39,999 `    * `5 documents 40,000-49,999 `    * `6 documents 0-9,999 `    * `7 documents 10,000-19,999 `    * `8 documents 20,000-29,999 `    * `9 documents 30,000-39,999 `    * `10 documents 40,000-49,999`<jupyter_code># LDA mono-core -- fallback code in case LdaMulticore throws an error on your machine
# lda_model = gensim.models.LdaModel(bow_corpus, 
#                                    num_topics = 10, 
#                                    id2word = dictionary,                                    
#                                    passes = 50)

# LDA multicore 
'''
Train lda model using gensim.models.LdaMulticore and save it
'''

lda_model = gensim.models.LdaMulticore(bow_corpus, 
                                       id2word=dictionary, 
                                       num_topics=10,
                                       workers=6, #if running on my local quadcore machine
                                       passes=2)
'''
For each topic, explore the words occuring in that topic and its relative weight
'''
for idx, topic in lda_model.print_topics(-1):
    print("Topic: {} \nWords: {}".format(topic, idx ))
    print("\n")<jupyter_output>Topic: 0.020*"probe" + 0.019*"death" + 0.016*"polic" + 0.014*"govt" + 0.011*"threat" + 0.010*"action" + 0.010*"say" + 0.007*"call" + 0.007*"hick" + 0.006*"inquiri" 
Words: 0


Topic: 0.017*"water" + 0.013*"nation" + 0.011*"park" + 0.011*"guilti" + 0.010*"fight" + 0.008*"worri" + 0.007*"plead" + 0.007*"river" + 0.007*"season" + 0.007*"staff" 
Words: 1


Topic: 0.019*"crash" + 0.011*"die" + 0.011*"reject" + 0.010*"continu" + 0.008*"highway" + 0.008*"claim" + 0.008*"govt" + 0.008*"england" + 0.007*"critic" + 0.007*"condit" 
Words: 2


Topic: 0.024*"fund" + 0.018*"boost" + 0.013*"govt" + 0.013*"indigen" + 0.012*"coast" + 0.011*"open" + 0.009*"health" + 0.008*"north" + 0.008*"gold" + 0.008*"feder" 
Words: 3


Topic: 0.033*"plan" + 0.028*"council" + 0.028*"urg" + 0.016*"chang" + 0.015*"govt" + 0.015*"water" + 0.012*"road" + 0.011*"public" + 0.011*"polic" + 0.011*"power" 
Words: 4


Topic: 0.016*"concern" + 0.012*"price" + 0.012*"fear" + 0.010*"sale" + 0.010*"market" + 0.010*"high" + 0.009*"h[...]<jupyter_text>Classification of the topics Using the words in each topic and their corresponding weights, a rough identification of categories as I see it , would be as follows, but I see some ambiguity with just 2 passes* 0: politics / investigation* 1: government* 2: Traffic / infrastructure* 3: Funding / natural resources* 4: Insfrastructure* 5: Finance* 6: Australia* 7: War / international* 8: Labor* 9: Crime  Running LDA using TF-IDF<jupyter_code>'''
Define lda model using corpus_tfidf, using gensim.models.LdaMulticore()
'''

lda_model_tfidf = gensim.models.LdaMulticore(corpus_tfidf, 
                                       id2word=dictionary, 
                                       num_topics=10,
                                       workers=6, #if running on my previous local quadcore machine
                                       passes=5)
'''
For each topic, we will explore the words occuring in that topic and its relative weight
'''
for idx, topic in lda_model_tfidf.print_topics(-1):
    print("Topic: {} Word: {}".format(idx, topic))
    print("\n")<jupyter_output>Topic: 0 Word: 0.028*"closer" + 0.008*"england" + 0.008*"bird" + 0.007*"australia" + 0.006*"test" + 0.006*"wind" + 0.006*"world" + 0.005*"lebanon" + 0.005*"lanka" + 0.004*"pont"


Topic: 1 Word: 0.007*"open" + 0.006*"final" + 0.004*"hewitt" + 0.004*"win" + 0.004*"burn" + 0.004*"drown" + 0.004*"grain" + 0.004*"brawl" + 0.004*"swim" + 0.004*"world"


Topic: 2 Word: 0.012*"kill" + 0.011*"blaze" + 0.010*"bomb" + 0.008*"iraq" + 0.008*"hick" + 0.007*"blast" + 0.007*"firefight" + 0.006*"soldier" + 0.006*"baghdad" + 0.006*"crew"


Topic: 3 Word: 0.008*"market" + 0.006*"isra" + 0.006*"tiger" + 0.006*"palestinian" + 0.006*"israel" + 0.005*"kill" + 0.005*"bank" + 0.005*"bomber" + 0.005*"gaza" + 0.005*"lead"


Topic: 4 Word: 0.014*"govt" + 0.009*"plan" + 0.009*"council" + 0.009*"fund" + 0.009*"health" + 0.008*"urg" + 0.007*"chang" + 0.006*"labor" + 0.006*"group" + 0.006*"union"


Topic: 5 Word: 0.011*"coast" + 0.009*"drought" + 0.009*"north" + 0.008*"rain" + 0.008*"gold" + 0.008*"east" + 0.007*"so[...]<jupyter_text>Classification of the topics As we can see, when using tf-idf, heavier weights are given to words that are not as frequent which results in nouns being factored in. That makes it harder to figure out the categories as nouns can be hard to categorize. This goes to show that the models we apply depend on the type of corpus of text we are dealing with. Using the words in each topic and their corresponding weights, I now see these categories:* 0: International* 1: Sports* 2: War* 3: International /conflict* 4: Labor* 5: Weather* 6: Local / accident* 7: Health* 8: Business* 9: Resources   Performance evaluation by classifying sample document using LDA Bag of Words modelCheck to see where our test document would be classified.<jupyter_code>'''
Text of sample document 4310
'''
processed_docs[4310]
lda_model[bow_corpus[document_num]]
'''
Check which topic our test document belongs to using the LDA Bag of Words model.
'''
document_num = 4310
# test document is document number 4310

for index, score in sorted(lda_model[bow_corpus[document_num]], key=lambda tup: -1*tup[1]):
    print("\nScore: {}\t \nTopic: {}".format(score, lda_model.print_topic(index, 10)))<jupyter_output>Score: 0.5037757754325867	 
Topic: 0.016*"concern" + 0.012*"price" + 0.012*"fear" + 0.010*"sale" + 0.010*"market" + 0.010*"high" + 0.009*"howard" + 0.008*"trade" + 0.008*"fuel" + 0.007*"plan"

Score: 0.33620792627334595	 
Topic: 0.024*"fund" + 0.018*"boost" + 0.013*"govt" + 0.013*"indigen" + 0.012*"coast" + 0.011*"open" + 0.009*"health" + 0.008*"north" + 0.008*"gold" + 0.008*"feder"

Score: 0.020005660131573677	 
Topic: 0.015*"worker" + 0.012*"plan" + 0.011*"record" + 0.010*"farm" + 0.009*"break" + 0.009*"rescu" + 0.009*"anti" + 0.008*"strike" + 0.008*"blaze" + 0.008*"govt"

Score: 0.020002705976366997	 
Topic: 0.033*"plan" + 0.028*"council" + 0.028*"urg" + 0.016*"chang" + 0.015*"govt" + 0.015*"water" + 0.012*"road" + 0.011*"public" + 0.011*"polic" + 0.011*"power"

Score: 0.020002439618110657	 
Topic: 0.020*"probe" + 0.019*"death" + 0.016*"polic" + 0.014*"govt" + 0.011*"threat" + 0.010*"action" + 0.010*"say" + 0.007*"call" + 0.007*"hick" + 0.006*"inquiri"

Score: 0.02000201866030693	 [...]<jupyter_text>It has the highest probability (`x`) to be  part of the topic that we assigned as X, which is the accurate classification.   Performance evaluation by classifying sample document using LDA TF-IDF model<jupyter_code>'''
Check which topic our test document belongs to using the LDA TF-IDF model.
'''
# Our test document is document number 4310
for index, score in sorted(lda_model_tfidf[bow_corpus[document_num]], key=lambda tup: -1*tup[1]):
    print("\nScore: {}\t \nTopic: {}".format(score, lda_model_tfidf.print_topic(index, 10)))<jupyter_output>Score: 0.42938441038131714	 
Topic: 0.025*"hospit" + 0.021*"plan" + 0.018*"power" + 0.016*"green" + 0.013*"push" + 0.012*"sale" + 0.012*"welcom" + 0.012*"delay" + 0.011*"group" + 0.011*"meet"

Score: 0.21999306976795197	 
Topic: 0.021*"help" + 0.019*"coast" + 0.017*"urg" + 0.015*"famili" + 0.015*"gold" + 0.012*"water" + 0.011*"forc" + 0.011*"sign" + 0.010*"save" + 0.010*"public"

Score: 0.2106127291917801	 
Topic: 0.017*"protest" + 0.017*"begin" + 0.012*"guilti" + 0.012*"arrest" + 0.012*"releas" + 0.012*"alleg" + 0.012*"highlight" + 0.011*"terror" + 0.011*"bushfir" + 0.011*"author"

Score: 0.020006049424409866	 
Topic: 0.016*"open" + 0.013*"farmer" + 0.011*"look" + 0.011*"back" + 0.011*"elect" + 0.010*"time" + 0.009*"confid" + 0.009*"play" + 0.009*"promis" + 0.008*"challeng"

Score: 0.020003490149974823	 
Topic: 0.031*"council" + 0.024*"fund" + 0.021*"plan" + 0.020*"water" + 0.018*"seek" + 0.016*"boost" + 0.015*"govt" + 0.014*"rise" + 0.013*"urg" + 0.010*"centr"

Score: 0.020000103861[...]<jupyter_text>It has the highest probability (`x%`) to be  part of the topic that we assigned as X.   Testing model on unseen document<jupyter_code>unseen_document = "My favorite sports activities are running and swimming."

# Data preprocessing step for the unseen document
bow_vector = dictionary.doc2bow(preprocess(unseen_document))

for index, score in sorted(lda_model[bow_vector], key=lambda tup: -1*tup[1]):
    print("Score: {}\t Topic: {}".format(score, lda_model.print_topic(index, 5)))<jupyter_output>Score: 0.6043497323989868	 Topic: 0.015*"worker" + 0.012*"plan" + 0.011*"record" + 0.010*"farm" + 0.009*"break"
Score: 0.23562777042388916	 Topic: 0.017*"water" + 0.013*"nation" + 0.011*"park" + 0.011*"guilti" + 0.010*"fight"
Score: 0.02000717632472515	 Topic: 0.024*"fund" + 0.018*"boost" + 0.013*"govt" + 0.013*"indigen" + 0.012*"coast"
Score: 0.02000698447227478	 Topic: 0.020*"probe" + 0.019*"death" + 0.016*"polic" + 0.014*"govt" + 0.011*"threat"
Score: 0.020005160942673683	 Topic: 0.016*"concern" + 0.012*"price" + 0.012*"fear" + 0.010*"sale" + 0.010*"market"
Score: 0.020001428201794624	 Topic: 0.033*"plan" + 0.028*"council" + 0.028*"urg" + 0.016*"chang" + 0.015*"govt"
Score: 0.020000752061605453	 Topic: 0.032*"kill" + 0.030*"polic" + 0.019*"miss" + 0.018*"iraq" + 0.015*"investig"
Score: 0.020000535994768143	 Topic: 0.033*"charg" + 0.032*"court" + 0.029*"face" + 0.019*"jail" + 0.016*"murder"
Score: 0.020000221207737923	 Topic: 0.019*"crash" + 0.011*"die" + 0.011*"reject" + 0.010*"cont[...]
{'fraction_non_alphanumeric': 0.13692884634362973, 'fraction_numerical': 0.08867067995657751, 'mean_word_length': 4.223453608247422, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 55, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '659589', 'license': 'MIT', 'n_tokens_mistral': 8184, 'n_tokens_neox': 6701, 'n_words': 2072, 'path': 'topic-modeling/LatentDirichletA_BoW_TF-IDF.ipynb', 'repo_name': 'gotamist/nlp', 'size': '20266'}