<jupyter_start><jupyter_text>Topics and TransformationsWe will be using gensim for data pre-processing, computational linguistics and topic modelling.Based on [Gensim Tutorials](https://radimrehurek.com/gensim/tutorial.html):* [Corpora and Vector Spaces](https://radimrehurek.com/gensim/tut1.html)* [Pre-processing and stop-words](https://radimrehurek.com/gensim/parsing/preprocessing.html)* [Stemming Algorithms](https://radimrehurek.com/gensim/parsing/porter.html)<jupyter_code>import os
import sys
import gensim

from gensim import corpora
from gensim import models
from gensim.models.coherencemodel import CoherenceModel

print('Python Version: %s' % (sys.version))
dictionary = corpora.Dictionary.load('documents.dict')
corpus = corpora.MmCorpus('documents.mm')
print(dictionary)
print(corpus)
import pickle
#with open('documents', 'wb') as f: #save
#    pickle.dump(mylist, f)

with open('documents', 'rb') as f: #load
    documents = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Creating a transformationThe transformations are standard Python objects, typically initialized by means of a *training* corpus:<jupyter_code>tfidf = models.TfidfModel(corpus)  # step 1 -- initialize a model<jupyter_output><empty_output><jupyter_text>Transforming vectorsFrom now on, `tfidf` is treated as a read-only object that can be used to convert any vector from the old representation (bag-of-words integer counts) to the new representation (TfIdf real-valued weights):<jupyter_code>corpus_tfidf = tfidf[corpus] # step 2 -- use the model to transform vectors<jupyter_output><empty_output><jupyter_text>LDA`gensim` uses a fast implementation of online LDA parameter estimation based on `<NAME>. 2010. Online learning for Latent Dirichlet Allocation`, modified to run in [distributed mode](https://radimrehurek.com/gensim/distributed.html) on a cluster of computers.<jupyter_code>lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
                                           id2word=dictionary,
                                           num_topics=20, 
                                           random_state=100,
                                           update_every=1,
                                           chunksize=100,
                                           passes=10,
                                           alpha='auto',
                                           per_word_topics=True)<jupyter_output><empty_output><jupyter_text>View the topics in LDA modelThe above LDA model is built with 20 different topics where each topic is a combination of keywords and each keyword contributes a certain weightage to the topic.You can see the keywords for each topic and the weightage(importance) of each keyword using `lda_model.print_topics()` as shown next.<jupyter_code># Print the weight and token 
lda_model.print_topics() # all topics
lda_model.print_topic(1) # selected topic<jupyter_output><empty_output><jupyter_text>Compute Perplexity and Coherence ScoreA lower perplexity score indicates better generalization performance *<NAME>.; <NAME>.; <NAME>. Latent Dirichlet Allocation. Journal of Machine Learning Research, [s. l.], v. 3, n. Jan, p. 993–1022, 2003.*<jupyter_code>print('Perplexity:', lda_model.log_perplexity(corpus))
coherence_model_lda = CoherenceModel(model=lda_model, texts=corpus, dictionary=dictionary, coherence='c_v')
coherence_lda = coherence_model_lda.get_coherence()
print('Coherence Score: ', coherence_lda)
lda_model.save('lda_model')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09111552653988078, 'fraction_numerical': 0.009650865739426626, 'mean_word_length': 3.8273972602739725, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '413098', 'license': 'MIT', 'n_tokens_mistral': 1077, 'n_tokens_neox': 1007, 'n_words': 277, 'path': 'Notebooks/gensim-topics.ipynb', 'repo_name': 'storopoli/topic-modelling', 'size': '3548'}