<jupyter_start><jupyter_text>Coding AssignmentQ: Write a python class with different function to fit LDA model, evaluate optimal number of topics based on best coherence scores and predict new instances based on best LDA model with optimal number of topics based on best coherence score. Function should take 2darray of embeddings as input and return a LDA model, optimal number of topics and topics.<jupyter_code>"""
author: <NAME>
email: <EMAIL>
github: https://github.com/parikshitsaikia1619
date: 20-08-2021
"""<jupyter_output><empty_output><jupyter_text>Step 1: Import neccessary Libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import gensim
import gensim.corpora as corpora
from gensim.utils import simple_preprocess
from gensim.models import CoherenceModel

import spacy
from tqdm.notebook import tqdm
#spacy download en_core_web_sm
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
stop_words = stopwords.words('english')<jupyter_output>[nltk_data] Downloading package stopwords to C:\Users\Parikshit
[nltk_data]     Saikia\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>Step 2: Load the datasetThis dataset contains a set of research articles related to computer science, mathematics, physics and statistics. Each article is tagged into major and minor topics in the form one hot encoding.But for our task (topic modeling) we don't need the tags, we just need the articles text.From the dataset we will form a list of articles<jupyter_code>data = pd.read_csv('./data/research_articles/Train.csv/Processed_train.csv')
data.head()
articles_data = data.ABSTRACT.values.tolist()<jupyter_output><empty_output><jupyter_text>Step 3: Creating a Data Preprocessing PipelineThis is the most important step in this entire code . We cannot expect good results from a model trained on a uncleaned data.As the famous quote goes "garbage in,garbage out"We want our corpus consisting a list of representative words capturing the essence of each article, To achieve that we need to follow a sequence of steps:![nlp_preprocessing](image.PNG)<jupyter_code>def convert_lowercase(string_list):
    """
    Convert the list of strings to lowercase and returns the list
    """
    pbar = tqdm(total = len(string_list),desc='lowercase conversion progress')
    for i in range(len(string_list)):
        string_list[i] = string_list[i].lower()
        pbar.update(1)
    pbar.close()
    return string_list

def remove_punctuation(doc_list):
    """
    Tokenization and remove punctuation and return the list of tokens
    """
    doc_word_list = []
    pbar = tqdm(total = len(doc_list),desc='punctuation removal progress')
    for doc in doc_list:
        doc_word_list.append(gensim.utils.simple_preprocess(str(doc), deacc=True,min_len=4))  # deacc=True removes punctuations
        pbar.update(1)
    pbar.close()
    return doc_word_list

def remove_stopwords(texts):
    """
    Remove common occuring words from the list of tokens
    stop words : a,an ,the ,so ,from ...
    
    """
    docs_data=[]
    stop_words = stopwords.words('english')
    pbar = tqdm(total = len(texts),desc='stopword removal progress')
    for doc in texts:
        doc_data=[]
        for word in doc:
            if word not in stop_words:
                doc_data.append(word)
        docs_data.append(doc_data)
        pbar.update(1)
    pbar.close()
    return docs_data

def filter_word_len(texts,size):
    """
    Remove tokens if their length is smaller than threshold
    
    """
    docs_data=[]
    pbar = tqdm(total = len(texts),desc='word filtering progress')
    for doc in texts:
        doc_data=[]
        for word in doc:
            if len(word)>=size:
                doc_data.append(word)
        docs_data.append(doc_data)
        pbar.update(1)
    pbar.close()
    return docs_data

def make_bigrams(texts):
    """
    Combine words that are frequently occurs together
    eg: Good Person , Great Content , Ever Growing
    """
    # Build the bigram and models
    pbar = tqdm(total = len(texts),desc='Bigram progress')
    
    bigram = gensim.models.Phrases(texts, min_count=5, threshold=100) # higher threshold fewer phrases.
    bigram_mod = gensim.models.phrases.Phraser(bigram)
    
    pbar.update(len(texts))
    pbar.close()
    return [bigram_mod[doc] for doc in texts]


def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):
    """
    Only allows token which are noun,adjective,verb and adverb.
    Also converting the tokens into base form
    """
    nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])
    texts_out = []
    pbar = tqdm(total = len(texts),desc='lemmatization progress')
    
    for sent in texts:
        doc = nlp(" ".join(sent)) 
        texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])
        pbar.update(1)
    pbar.close()
    
    return texts_out

def preprocessing_pipeline(doc_list,size):
    """
    Converting each document into list of tokens that represents the document
    """
    doc_list = convert_lowercase(doc_list)
    doc_data_words = remove_punctuation(doc_list)
    # Remove Stop Words
    data_words_nostops = remove_stopwords(doc_data_words)
    
     # Form Bigrams
    data_words_bigrams = make_bigrams(data_words_nostops)

    data_lemmatized = lemmatization(data_words_bigrams, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])
    
    final_data_words = filter_word_len(data_lemmatized,size)
        
    return final_data_words
    
def corpus_embeddings(data):
    """
    Converting each token in each document into a tuple of (unique id, frequency of occuring in that document)
    """
    # Create Dictionary
    id2word = corpora.Dictionary(data)
    # Create Corpus
    texts = data
    # Term Document Frequency
    corpus = [id2word.doc2bow(text) for text in texts]
    
    return corpus,id2word
    
processed_data = preprocessing_pipeline(articles_data,4) # this will take some time
processed_data[0]<jupyter_output><empty_output><jupyter_text>Step 4: Finalizing the input dataIn this step we will form our the inputs of model, which are:* **Corpus**: A 2D embedded array of tuples, where each tuple is in the form of (token id, frequency of token in that document).* **dictionary**: A dictionary storing the mapping from token to id.<jupyter_code>new_corpus,id_word = corpus_embeddings(processed_data)
new_corpus
id_word[0]
word_freq = [[(id_word[id], freq) for id, freq in cp] for cp in new_corpus[:1]]
word_freq # A more human reable form of our corpus<jupyter_output><empty_output><jupyter_text>Step 5: Modeling and EvaluationIn this part we will fit our data to our the LDA model , some hyper parameter tuning , evaluate the results and select the optimal setting for our model.<jupyter_code>class LDA_model:
    """
    A LDA Class consist functions to fit the model, calculating coherence values
    and finding the optimal no. of topic
    
    input: 
     corpus : a 2D array of embedded tokens
     dictionary: A dictionary with id to token mapping
     
    """
    def __init__(self, corpus,dictionary):
        self.corpus = corpus
        self.dictionary = dictionary
    
    def LDA(self,no_topics, a ='auto', b = 'auto',passes=10):
        lda_model = gensim.models.ldamodel.LdaModel(corpus=self.corpus, id2word=self.dictionary,
                                                    num_topics=no_topics, random_state=100,chunksize=100,passes=passes,alpha=a,eta=b)
        return lda_model

    def compute_coherence_values(self,processed_data,no_topics,a ='auto', b = 'auto',passes=10):
        """
        Computes the coherence value of a fitted LDA model
        """
    
        pbar = tqdm(total = len(self.corpus),desc='LDA with '+str(no_topics)+' topics')
        
        lda_model = self.LDA(no_topics,a,b,passes)
        pbar.update(len(self.corpus)/2)
        coherence_model_lda = CoherenceModel(model=lda_model, texts=processed_data, dictionary=self.dictionary, coherence='c_v')
        pbar.update(len(self.corpus) -(len(self.corpus)/2))
        pbar.close()

        return coherence_model_lda.get_coherence()

    def compute_optimal_topics(self,processed_data,min_topics,max_topics,step_size,path):
        """
        Calucates the coherence value for a given range of topic size and forms a dataset
        """
        
        topics_range = list(np.arange(min_topics,max_topics,step_size))
        model_results = {'Topics': [],'Coherence': []}
        # Can take a long time to run
        for i in range(len(topics_range)):
            # get the coherence score for the given parameters
            cv = self.compute_coherence_values(processed_data,topics_range[i],a ='auto', b = 'auto',passes=10)
            # Save the model results
            model_results['Topics'].append(topics_range[i])
            model_results['Coherence'].append(cv)
        pd.DataFrame(model_results).to_csv(path, index=False)
    
    def Optimal_no_topics(self,path):
        """
        finds the topic size with max coherence score
        """
        coherence_data = pd.read_csv(path)
        x = coherence_data.Topics.tolist()
        y = coherence_data.Coherence.tolist()
        plt.xlabel('No. of topics')
        plt.ylabel('Coherence Score')
        plt.plot(x,y)
        plt.show()
        index = np.argmax(y)
        no_topics = x[index]
        c_v = y[index]
        print('Optimal number of topics: '+str(no_topics)+' coherence score: '+str(c_v))
        
        return no_topics,c_v
    
    def Optimal_lda_model(self,path):
        """
        Fits the LDA with optimal topic size
        """
        no_topic, c_v =  self.Optimal_no_topics(path)
        pbar = tqdm(total = len(self.corpus),desc='LDA with '+str(no_topic)+' topics')
        lda = self.LDA(no_topic)
        pbar.update(len(self.corpus))
        pbar.close()
        
        return lda
        
lda_model = LDA_model(new_corpus,id_word)
path = './dat.csv'<jupyter_output><empty_output><jupyter_text>**Note**: Computing the optimal number of topics for LDA will take a pretty long amount of timeThe time complexity of this process : **O( m * x * p)**where:* m = no. of tokens in the corpus* x = summation of no. of topics given in the range* p = no. of passes for LDA model<jupyter_code>lda_model.compute_optimal_topics(processed_data,2,4,1,path)
lda = lda_model.Optimal_lda_model('./results/lda_tuning_results1.csv')
lda.print_topics()<jupyter_output><empty_output><jupyter_text>Step 6: Visualization<jupyter_code>import pyLDAvis
import pyLDAvis.gensim_models as gensimvis
pyLDAvis.enable_notebook()
vis =  gensimvis.prepare(lda, new_corpus, id_word)
vis<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09429049655941975, 'fraction_numerical': 0.005393342012274502, 'mean_word_length': 3.5018836333193804, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '668183', 'license': 'MIT', 'n_tokens_mistral': 3527, 'n_tokens_neox': 3396, 'n_words': 948, 'path': 'LDA_New.ipynb', 'repo_name': 'parikshitsaikia1619/LDA_modeing_IQVIA', 'size': '10786'}