<jupyter_start><jupyter_text>Topic Modeling As with other forms of natural language processing, *topic modeling* allows us to quickly and systematically analyze vast quantities of unstructured text. With topic modelling, we are able to uncover some of the more abstract, underlying themes contained within large collections of text without the need to assign pre-determined categories or tags to the corpus. This lesson will go over what, exactly, topic modeling does, walk through how to run Latent Dirichlet allocation (LDA) topic models in Python, and introduce a number of fit statistics to help us better understand the topic models we'll be generating. We'll end with a discussion of several useful visualization tools for topic modeling.   What is Topic Modeling?As a form of unsupervised machine learning, *topic modeling* allows for the classification of large collections of textual documents into natural groups, without the need for extensive human supervision. Employed in text mining and natural language processing, topic models can uncover the hidden, or *latent*, meanings of language patterns within texts.  [Source](https://www.tidytextmining.com/topicmodeling.html)  SetupIn addition to `matplotlib inline` and `pandas`, we'll also be importing `CountVectorizer` from [scikit-learn](https://scikit-learn.org/stable/), a machine learning library for Python. For more information on the usage and relavent parameters taken by `CountVectorizer`, see the chapter on Text Classification. We'll also want to use `pandas` to increase our maximum column width to 120 characters, over the default 50-character column width. This will help us more easily glance over text in the dataframe.<jupyter_code>%matplotlib inline

import pandas as pd

from sklearn.feature_extraction.text import CountVectorizer


pd.set_option('display.max_colwidth', 120)<jupyter_output><empty_output><jupyter_text>In this chapter we'll be looking at [transcriptions](https://www.kaggle.com/unitednations/un-general-debates) of United Nations General Debates, from 1980 to 1999. We've previously looked at the same dataset when disucssing word frequencies.<jupyter_code>un_df = pd.read_json('un-general-debates.json')
print(len(un_df))<jupyter_output>3214<jupyter_text>With 3,214 complete transcriptions of UN members' statements to work with, this is an ideal dataset to use as we begin playing around with topic models. Let's look at a small random sample of 5 texts from the dataframe to get a feel for what we're dealing with:<jupyter_code>un_df.sample(5)<jupyter_output><empty_output><jupyter_text>Topic Modeling Exercise 1Take a look at the text of the UN speeches. When delivering an address, what are the different topics that are covered? Make a list of four topics and provide three example words from each topic. ![](images/lda.jpg)  Latent Dirichlet allocation (LDA) We'll narrow our focus for the time being to one particular form of topic modeling, *Latent Dirichlet allocation*, or *LDA* for short. With LDA topic modeling, we'll be able to treat every document in our corpus as a mixture of topics. Each document in our corpus can contain words associated with any number of topics in varying proportions. At the same time, each topic can be treated as a mixture of words. Any given word can be associated with any number of topics. Considering our documents and topics as these sorts of "mixtures" helps us to mimic the thematic subtleties inherent in natural language.[Source](https://www.tidytextmining.com/topicmodeling.htmllatent-dirichlet-allocation) ![](images/lda.jpg) We can import `LatentDirichletAllocation` from scikit-learn to run our own LDA topic models in Python.<jupyter_code>from sklearn.decomposition import LatentDirichletAllocation<jupyter_output><empty_output><jupyter_text>Converting Documents to VectorsIn order to run our topic models, we'll need to convert each document in the corpus into a fixed-length vector of token counts. We can accomplish this using the [CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html?highlight=vectorizersklearn.feature_extraction.text.CountVectorizer) function we imported earlier in the chapter. For now, let's set our parameters so that we convert all text to lower case, only look at unigrams, only look at terms with a document frequency of .90 or below, use the default 'english' stopwords list, and only consider the top 1,000 terms in our corpus.<jupyter_code>vectorizer = CountVectorizer(lowercase   = True,
                             ngram_range = (1,1),
                             max_df      = .90,
                             stop_words   = 'english',
                             max_features = 1000)<jupyter_output><empty_output><jupyter_text>After setting our parameters, we can `fit` the vectorizer to the `speech_text` key in our UN General Debate dataframe to build a vocabulary out of the raw documents. *Note*: You'll run into an Attribute Error if the key you plan to fit the vectorizer to contains any missing values. While we don't have to worry about this with our UN dataframe, if you encounter such an error in the future, be sure to [clean your dataframe](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.htmlmissing-data) before attempting to fit the vectorizer.<jupyter_code>vectorizer.fit(un_df['speech_text'])<jupyter_output><empty_output><jupyter_text>We can use the `len` function, along with `get_feature_names`, to ensure we're dealing with a vocabulary composed of the top 1,000 highest-frequency terms in our corpus.<jupyter_code>len(vectorizer.get_feature_names())<jupyter_output><empty_output><jupyter_text>Now we'll want to use the vectorizor to `transform` the raw documents into a document-term matrix.<jupyter_code>un_word_counts = vectorizer.transform(un_df['speech_text'])<jupyter_output><empty_output><jupyter_text>Running the LDA ModelNow that we've vectorized our dataset, we're just about ready to run our first LDA model in Python. Before we do, though, we'll want to set our parameters. Below is some information on the parameters we'll set.   `LatentDirichletAllocation` Parameters- **n_components**: Sets the number of topics generated. We can set this as high or as low as we like, depending on the size and character of the texts in our corpus. - **max_iter**: Sets the maximum number of iterations. By default, `max_iter` is set to 10.> - ***Note***: In almost every case, we'll want to set `max_iter` above 10. It's highly unlikely our models will converge in 10 iterations or less. In the following example, we'll set `max_iter` to 50, a more reasonable maximum iteration threshold.- **evaluate_every**: Lets us adjust how frequently we gauge the perplexity of our model across iterations. By default, `evaluate_every` is set to 0.> - ***Note***: Leaving `evaluate_every` at 0 leaves our model without any built-in goodness of fit measure. We'll discuss other measurements of model fit later in the chapter, but for now it's useful to set `evaluate_every` to some positive number. In the following example, we'll `evaluate_every` 5 iterations.- **n_jobs**: Sets the number of concurrently running processes. When set to -1, we'll use all processors. To use all processors but one, set `n_jobs` to -2. - **verbose**: Lets us determine whether or not every step of the process is logged. If > 0, we'll be able to see what's going on with our LDA model through the output in real time.<jupyter_code>lda_model = LatentDirichletAllocation(n_components = 10,
                                      max_iter     = 50,
                                      evaluate_every = 5,
                                      n_jobs       = -1,
                                      verbose      = 1)<jupyter_output><empty_output><jupyter_text>With our parameters set, we can `fit` the LDA model to our document-term matrix of the UN General Debate transcripts.*Note*: It's going to take a while to work our way through up to 50 iterations. That's alright.<jupyter_code>lda_model.fit(un_word_counts)<jupyter_output>iteration: 1 of max_iter: 50
iteration: 2 of max_iter: 50
iteration: 3 of max_iter: 50
iteration: 4 of max_iter: 50
iteration: 5 of max_iter: 50, perplexity: 711.0737
iteration: 6 of max_iter: 50
iteration: 7 of max_iter: 50
iteration: 8 of max_iter: 50
iteration: 9 of max_iter: 50
iteration: 10 of max_iter: 50, perplexity: 704.3359
iteration: 11 of max_iter: 50
iteration: 12 of max_iter: 50
iteration: 13 of max_iter: 50
iteration: 14 of max_iter: 50
iteration: 15 of max_iter: 50, perplexity: 702.4462
iteration: 16 of max_iter: 50
iteration: 17 of max_iter: 50
iteration: 18 of max_iter: 50
iteration: 19 of max_iter: 50
iteration: 20 of max_iter: 50, perplexity: 701.4847
iteration: 21 of max_iter: 50
iteration: 22 of max_iter: 50
iteration: 23 of max_iter: 50
iteration: 24 of max_iter: 50
iteration: 25 of max_iter: 50, perplexity: 700.7602
iteration: 26 of max_iter: 50
iteration: 27 of max_iter: 50
iteration: 28 of max_iter: 50
iteration: 29 of max_iter: 50
iteration: 30 of max_iter: 50[...]<jupyter_text>Congrats, you've run your first topic model on Python!  Some fit statisticsWhile we can intuitively "eyeball" topic quality as a first step, it's hard to do so objectively.  Calculating some fit statistics can help us to evaluate our topics' quality numerically.`LatentDirichletAllocation` includes a few handy methods for calculating fit statistics:- **`score()`** lets us calculate the approximate logged likelihood of the model parameters we've set, given our data. The higher this number is, the better our topic fit. - **`perplexity()`**, another (normalized) form of logged likelihood, calculates the amount of "surprise" our model experiences if we introduce some previously unseen data. The lower this number is, the better our topic fit.To learn more about evaluating fit for LDA topic models in Python, see [here](https://towardsdatascience.com/evaluate-topic-model-in-python-latent-dirichlet-allocation-lda-7d57484bb5d0).<jupyter_code>print("Log Likelihood: ", lda_model.score(un_word_counts))

print("Perplexity: ", lda_model.perplexity(un_word_counts))<jupyter_output>Log Likelihood:  -17539022.088006437
Perplexity:  699.3729778593845<jupyter_text>Guidelines on topic fit1. Low perplexity on test data.    - Remeber that the lower our perplexity score, the better our topic fit to our data.2. Topical coherence    - On the other hand, we want our models to receive the highest possible scores for topical coherence. 3. Best fit in a classification task.    - We'll discuss classification tasks in more detail in another chapter. These tasks provide an additional means of determining our models' goodness of fit.4. Extract more and then bin them yourself.     - It's also possible to just extract more topic models, and use your human intuition to bin them based on contextual or thematic similarity.<jupyter_code>print(lda_model.get_params())<jupyter_output>{'batch_size': 128, 'doc_topic_prior': None, 'evaluate_every': 5, 'learning_decay': 0.7, 'learning_method': 'batch', 'learning_offset': 10.0, 'max_doc_update_iter': 100, 'max_iter': 50, 'mean_change_tol': 0.001, 'n_components': 10, 'n_jobs': -1, 'perp_tol': 0.1, 'random_state': None, 'topic_word_prior': None, 'total_samples': 1000000.0, 'verbose': 1}<jupyter_text>Visualizing Topics In this section we'll discuss some of the ways we can visualize the topics we've created with our LDA model.   pdtext First, we can import `topic_words` from the `pdtext` package in order to create a easily interpretable matrix of our topics. Below, we'll look at the first 10:<jupyter_code>from pdtext.tm import topic_words

topic_words(lda_model, vectorizer).head(10)<jupyter_output><empty_output><jupyter_text>Also from `pdtext`, `topic_pred` will let us see which documents are associated with which topics.<jupyter_code>from pdtext.tm import topic_pred
un_topics = topic_pred(lda_model, un_word_counts, vectorizer)
un_topics<jupyter_output><empty_output><jupyter_text>We can now use our topics as features in order to get a better handle on topic patterns across texts. One way to do this is to generate a new key in our United Nations dataframe. For exmaple, we can create a `post_soviet` key to divide our general debate speeches between those that occured prior to the fall of the Soviet Union (where `post_soviet` = False), and those that occured after the fall of the Soviet Union (where `post_soviet` = True).<jupyter_code>un_df['post_soviet'] = un_df['speech_year'] > 1991
un_topics.groupby(un_df['post_soviet']).mean()
un_topics.groupby(un_df['post_soviet']).mean().T<jupyter_output><empty_output><jupyter_text>pyLDAvis The [pyLDAvis](https://pyldavis.readthedocs.io/en/latest/readme.html) library, a port of the [LDAvis](https://github.com/cpsievert/LDAvis) package for R, provides a variety of tools for interactive topic model visualization.<jupyter_code>pip install pyldavis<jupyter_output><jupyter_text>`pyLDAvis` is conveniently compatible with `LatentDirichletAllocation` from scikit-learn.<jupyter_code>import pyLDAvis
import pyLDAvis.sklearn<jupyter_output><empty_output><jupyter_text>We'll also import `pyplot` from matplotlib to allow for the generation of interactive plots in Python.<jupyter_code>import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Now we'll want to use the `enable_notebook()` function to allow us to display visualizations in our notebook.<jupyter_code>pyLDAvis.enable_notebook()<jupyter_output><empty_output><jupyter_text>Finally, we can use the `prepare()` function to transform the data from our LDA model into interactive visualizations!<jupyter_code>pyLDAvis.display(pyLDAvis.sklearn.prepare(lda_model, un_word_counts, vectorizer, mds='tsne'))<jupyter_output><empty_output><jupyter_text>Topic Modeling Exercise 2In your group, do 1 and 2 in 10_Topic_Modeling_group  Text Classification and Sentiment Analysis  We'll be using the following to apply classify texts and analyze sentiment within our corpus:- [seaborn](https://seaborn.pydata.org/), a data visualization library based on `matplotlib`. `seaborn` is also discussed in a previous chapter on text classification- The `SentimentIntensityAnalyzer` available through [vaderSentiment](https://github.com/cjhutto/vaderSentiment), a lexicon for sentiment analysis discussed in the previous chapter on Word Lists and Sentiment Analysis. With this, we'll be able to determine the intensity of positive, negative, and neutral sentiments contained within the documents in our corpus.- [Afinn](https://pypi.org/project/afinn/), another sentiment analysis tool discussed in the previous chapter on Word Lists and Sentiment Analysis. `Afinn` will produce a single numerical sentiment score, from -5 (negative sentiment) to 5 (positive sentiment).<jupyter_code>%matplotlib inline

import pandas as pd
import seaborn as sns
pip install afinn
pip install vaderSentiment
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
from afinn import Afinn<jupyter_output><empty_output><jupyter_text>In addition to `Afinn` and the `SentimentIntensityAnalyzer`, we'll also import a number of new functions from `sklearn`.<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.decomposition import LatentDirichletAllocation<jupyter_output><empty_output><jupyter_text>First among these imported functions is [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html?highlight=logisticregressionsklearn.linear_model.LogisticRegression), scikit-learn's logicstic regression classifier. While we've used `LogisticRegression` earlier in the chapter on Classification, at that point we didn't specify any parameters. In this exercise, we'll specify two:- **solver**: In the following example we'll be setting this to `lbfgs`, or [Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm](http://aria42.com/blog/2014/12/understanding-lbfgs). If not otherwise specified, `solver` will default to `lbfgs`. There are a number of other solvers available, discussed in more detail in the `LogisticRegression`[User Guide](https://scikit-learn.org/stable/modules/linear_model.htmllogistic-regression).- **max_iter**: Similar to what we saw earlier with `LatentDirichletAllocation`, here `max_iter` allows us to set a maximum threshold on the number of iterations in our regression before the solvers converge.  We'll set the maximum iterations in our own `lr_classifier` to 5,000.<jupyter_code>lr_classifier = LogisticRegression(solver = 'lbfgs', max_iter= 5000)<jupyter_output><empty_output><jupyter_text>Let's `fit` the model to our data based on the `post_soviet` key we've created.<jupyter_code>lr_classifier.fit(un_topics, un_df['post_soviet'])<jupyter_output><empty_output><jupyter_text>Our classifier can now be used to `predict` an accuracy score for our topics.<jupyter_code>prediction = lr_classifier.predict(un_topics)
print(accuracy_score(un_df['post_soviet'], prediction))<jupyter_output>0.940883634100809<jupyter_text>We're also now able to produce a classification report:<jupyter_code>print(classification_report(un_df['post_soviet'], prediction))<jupyter_output>precision    recall  f1-score   support

       False       0.96      0.94      0.95      1803
        True       0.92      0.94      0.93      1411

    accuracy                           0.94      3214
   macro avg       0.94      0.94      0.94      3214
weighted avg       0.94      0.94      0.94      3214<jupyter_text>Finally, we can use `seaborn` to generate a confusion matrix using the `heatmap` function.<jupyter_code>import seaborn as sns

cm = confusion_matrix(un_df['post_soviet'], prediction)
sns.heatmap(cm, annot=True, cmap="Greens", fmt='g')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0739539231319074, 'fraction_numerical': 0.02188376312750866, 'mean_word_length': 5.209627859337658, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 106, 'https://': 14, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '583392', 'license': 'MIT', 'n_tokens_mistral': 5634, 'n_tokens_neox': 5084, 'n_words': 2080, 'path': 'Notebooks/topic_modeling.ipynb', 'repo_name': 'nealcaren/Text-Mining-with-Python-Notebooks', 'size': '18187'}