<jupyter_start><jupyter_text>Topic Modeling IntroductionTopic modeling is another popular text analysis technique. The goal of topic modeling is to find various topics that are present in your corpus. Each document in the corpus will be made up of at least one topic, if not multiple topics.In this notebook, I will apply Latent Dirichlet Allocation (LDA), which is one of many topic modeling techniques. It was specifically designed for text data.Latent means hidden (topics) and Dirichlet refers to a probability distrubution technique. To use a topic modeling technique, we need to provide1. a document-term matrix and 2. the number of topics you would like the algorithm to pick up.Once the topic modeling technique is applied, the job is to interpret the results and see if the mix of words in each topic make sense. If they don't make sense, we can try changing up the number of topics, the terms in the document-term matrix, model parameters, or even try a different model. Topic Modeling - Attempt 1 (All Text)<jupyter_code># reading the document term matrix
import pandas as pd
import pickle

data = pd.read_pickle('pickle files/dtm_stop.pkl')
data.head()
# importing necessary modules for LDA with gensim
from gensim import matutils, models
import scipy.sparse

# we can also use log to debug that helps in later hyperpaarameter tuning
# import logging
# logging.basicConfig(format = '%(asctime)s : %(levelname)s : %(message)s',level = logging.INFO)
# we need term document matrix 
tdm = data.transpose()
tdm.head()
# putting tdm into a new gensim format
# df -----> sparse matrix ----->gensim corpus

sparse_count = scipy.sparse.csr_matrix(tdm)
corpus = matutils.Sparse2Corpus(sparse_count)
# Gensim also requires dictionary of all the terms and
# their respective location in the tdm

cv = pickle.load(open("pickle files/cv_stop.pkl", "rb"))

id2word = dict((value, key) for key, value in cv.vocabulary_.items())<jupyter_output><empty_output><jupyter_text>Now that we have the corpus(term document matrix) and id2word(dictionary of (location: term), we need to specify two more parameters - the number of topics and the numbers of passes(iterations). Let's take the number of topics as 2, and check the results.<jupyter_code># specifying no. of topics ad no. of passes
lda = models.LdaModel(corpus = corpus,
                     id2word = id2word,
                     num_topics = 2, passes = 10)
lda.print_topics()
# increasing number of topics for better insights
lda = models.LdaModel(corpus = corpus,
                     id2word = id2word,
                     num_topics = 3, passes = 10)
lda.print_topics()<jupyter_output><empty_output><jupyter_text>Clearly, there are no meaningful results.This time I had modified the parameters, now I have tried to modify the term list as well. Topic modeling - Attempt 2(Nouns only)<jupyter_code># function to pull oout all the nouns from the text string
from nltk import word_tokenize, pos_tag

def nouns(text):
    # string of text ----> tokenize the text ----> pull out nouns
    is_noun = lambda pos: pos[:2] == 'NN'
    tokenized = word_tokenize(text)
    all_nouns = [word for (word, pos) in pos_tag(tokenized) if is_noun(pos)]
    return ' '.join(all_nouns)
# loading the cleaned text data before count vectorization
data_clean = pd.read_pickle('pickle files/clean.pkl')
data_clean
# applying nouns function to the trascripts to filter only on nouns
data_nouns = pd.DataFrame(data_clean.transcript.apply(nouns))
data_nouns
# creating a new document-term matrix using only nouns
from sklearn.feature_extraction import text
from sklearn.feature_extraction.text import CountVectorizer

# re- adding stop_words since we are creating new dtm
# defining some additional stop words
add_stop_words = ['like', 'im', 'know', 'just', 'dont', 'thats', 'right', 'people',
                  'youre', 'got', 'gonna', 'time', 'think', 'yeah', 'said']
stop_words = text.ENGLISH_STOP_WORDS.union(add_stop_words)

# creating a dtm with only nouns
cvn = CountVectorizer(stop_words = stop_words)
model = cvn.fit_transform(data_nouns.transcript)
data_new_dtm = pd.DataFrame(model.toarray(), columns = cvn.get_feature_names())
data_new_dtm.index = data_nouns.index
data_new_dtm
# creating gensim corpus 
new_corpus = matutils.Sparse2Corpus(scipy.sparse.csr_matrix(data_new_dtm.transpose()))

# creating vocabulary dictionary
new_id2word = dict((v, k) for k, v in cvn.vocabulary_.items())
# starting with no of topics = 2
new_lda = models.LdaModel(corpus = new_corpus, id2word = new_id2word,
                          num_topics = 2 ,passes = 10)
new_lda.print_topics()
# no of topics = 3
new_lda = models.LdaModel(corpus = new_corpus, id2word = new_id2word,
                          num_topics = 3 ,passes = 10)
new_lda.print_topics()<jupyter_output><empty_output><jupyter_text>Topic Modeling - Attempt 3 (Nouns and Adjectives)<jupyter_code># creating a function to pull out nouns from a string of text
def nouns_adj(text):
    '''Given a string of text, tokenize the text and pull out only the nouns and adjectives.'''
    is_noun_adj = lambda pos: pos[:2] == 'NN' or pos[:2] == 'JJ'
    tokenized = word_tokenize(text)
    nouns_adj = [word for (word, pos) in pos_tag(tokenized) if is_noun_adj(pos)] 
    return ' '.join(nouns_adj)
# Applying the nouns function to the transcripts to filter only on nouns
data_nouns_adj = pd.DataFrame(data_clean.transcript.apply(nouns_adj))
data_nouns_adj
# Create a new document-term matrix using only nouns and adjectives, also remove common words with max_df
cvna = CountVectorizer(stop_words=stop_words, max_df=.8)
data_cvna = cvna.fit_transform(data_nouns_adj.transcript)
data_dtmna = pd.DataFrame(data_cvna.toarray(), columns=cvna.get_feature_names())
data_dtmna.index = data_nouns_adj.index
data_dtmna
# Creating the gensim corpus
corpusna = matutils.Sparse2Corpus(scipy.sparse.csr_matrix(data_dtmna.transpose()))

# Creating the vocabulary dictionary
id2wordna = dict((v, k) for k, v in cvna.vocabulary_.items())
# no of topics = 2
ldana = models.LdaModel(corpus=corpusna, num_topics=2, id2word=id2wordna, passes=10)
ldana.print_topics()
# no of topics = 3
ldana = models.LdaModel(corpus=corpusna, num_topics=3, id2word=id2wordna, passes=10)
ldana.print_topics()
# no of topics = 4
ldana = models.LdaModel(corpus=corpusna, num_topics=4, id2word=id2wordna, passes=10)
ldana.print_topics()<jupyter_output><empty_output><jupyter_text>Identify Topics in Each DocumentOut of the various topic models we looked at, the nouns and adjectives, 4 topic one made the most sense. So pulling that down here and running it through some more iterations to get more fine-tuned topics.<jupyter_code># final LDA model (for now) with more no of passes
ldana = models.LdaModel(corpus=corpusna, num_topics=4, id2word=id2wordna, passes=80)
ldana.print_topics()<jupyter_output><empty_output><jupyter_text>Trying to settle on these 4 topics as of now - Topic 0: guns - Topic 1: husband - Topic 2: mom, parents - Topic 3: grandma, friends<jupyter_code># checking which topic does each transcript contains
corpus_transformed = ldana[corpusna]
list(zip([a for [(a,b)] in corpus_transformed], data_dtmna.index))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08875, 'fraction_numerical': 0.009444444444444445, 'mean_word_length': 4.826051779935275, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '385747', 'license': 'MIT', 'n_tokens_mistral': 2414, 'n_tokens_neox': 2306, 'n_words': 792, 'path': 'Natural Language Processing/4-Topic-Modelling.ipynb', 'repo_name': 'riddhi-jain/Way-to-Deep-Learning', 'size': '7200'}