<jupyter_start><jupyter_text>Import libraries<jupyter_code># standard library
import sys,os
sys.path.append(os.getcwd())
from pprint import pprint

# data and nlp
import pandas as pd
import spacy
nlp = spacy.load("en_core_web_sm", disable=["ner"])

# visualisation
import pyLDAvis.gensim
pyLDAvis.enable_notebook()
import seaborn as sns
from matplotlib import rcParams
# figure size in inches
rcParams['figure.figsize'] = 20,10


# LDA tools
import gensim
import gensim.corpora as corpora
from gensim.models import CoherenceModel
from utils import lda_utils

# warnings
import logging, warnings
warnings.filterwarnings('ignore')
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.ERROR)<jupyter_output>/usr/local/lib/python3.8/site-packages/scipy/sparse/sparsetools.py:21: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!
scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.
  _deprecated()<jupyter_text>Read in text<jupyter_code>with open("Austen_Pride.txt") as f:
    text = f.read()<jupyter_output><empty_output><jupyter_text>__Split into individual sentences__<jupyter_code>doc = nlp(text)
sentences = [sent.string.strip() for sent in doc.sents]<jupyter_output><empty_output><jupyter_text>__Create chunks of 10 sentences at a time__<jupyter_code>chunks = []
for i in range(0, len(sentences), 10):
    chunks.append(' '.join(sentences[i:i+10]))<jupyter_output><empty_output><jupyter_text>Process using ```gensim```Here we're using ```gensim``` to produce efficiently procude a model of bigrams and trigrams in the data.We first create bigrams based on words appearing one after another frequently. These bigrams are then fed into a trigram generator, which takes the bigram as the second part of a bigram.- bigram model = (a,b)- trigram model = (x, (a,b))<jupyter_code># Build the bigram and trigram models
bigram = gensim.models.Phrases(chunks, min_count=3, threshold=100) # higher threshold fewer phrases.
trigram = gensim.models.Phrases(bigram[chunks], threshold=100)  
bigram_mod = gensim.models.phrases.Phraser(bigram)
trigram_mod = gensim.models.phrases.Phraser(trigram)<jupyter_output><empty_output><jupyter_text>We can use the ```lda_utils.process_words()``` function to further process the data using ```spaCy```.Make sure to check the util script and see what arguments this function takes!<jupyter_code>data_processed = lda_utils.process_words(chunks,nlp, bigram_mod, trigram_mod, allowed_postags=["NOUN"])<jupyter_output><empty_output><jupyter_text>We then need to create a ```gensim``` dictionary and a ```gensim``` corpus.The dictionary converts each word into an integer value; the corpus creates a 'bag of words' model for all of the data - in this case, chunks of 10 sentences at a time.<jupyter_code># Create Dictionary
id2word = corpora.Dictionary(data_processed)

# Create Corpus: Term Document Frequency
corpus = [id2word.doc2bow(text) for text in data_processed]<jupyter_output><empty_output><jupyter_text>We then feed all of this information into our LDA algorithm using ```gensim```.<jupyter_code># Build LDA model
lda_model = gensim.models.LdaMulticore(corpus=corpus,
                                       id2word=id2word,
                                       num_topics=10, 
                                       random_state=100,
                                       chunksize=10,
                                       passes=10,
                                       iterations=100,
                                       per_word_topics=True, 
                                       minimum_probability=0.0)<jupyter_output><empty_output><jupyter_text>We can then calculate model perplexity and coherence in order to interpret how well the model performs.<jupyter_code># Compute Perplexity
print('\nPerplexity: ', lda_model.log_perplexity(corpus))  # a measure of how good the model is. lower the better.

# Compute Coherence Score
coherence_model_lda = CoherenceModel(model=lda_model, 
                                     texts=data_processed, 
                                     dictionary=id2word, 
                                     coherence='c_v')

coherence_lda = coherence_model_lda.get_coherence()
print('\nCoherence Score: ', coherence_lda)<jupyter_output>Perplexity:  -7.316109127702134

Coherence Score:  0.34821244385018285<jupyter_text>Lastly, we can inspect the 'contents' of the model<jupyter_code>pprint(lda_model.print_topics())<jupyter_output>[(0,
  '0.017*"sister" + 0.016*"man" + 0.014*"time" + 0.013*"friend" + '
  '0.013*"marriage" + 0.013*"room" + 0.012*"hertfordshire" + 0.012*"daughter" '
  '+ 0.012*"manner" + 0.010*"world"'),
 (1,
  '0.025*"sister" + 0.021*"time" + 0.020*"letter" + 0.019*"family" + '
  '0.016*"friend" + 0.013*"day" + 0.013*"brother" + 0.009*"longbourn" + '
  '0.009*"room" + 0.009*"darcy"'),
 (2,
  '0.078*"door" + 0.029*"world" + 0.027*"mother" + 0.020*"parent" + '
  '0.012*"dearest" + 0.012*"kind" + 0.012*"lake" + 0.010*"idea" + '
  '0.009*"consideration" + 0.009*"yesterday"'),
 (3,
  '0.020*"room" + 0.018*"mother" + 0.015*"daughter" + 0.013*"time" + '
  '0.013*"day" + 0.012*"family" + 0.011*"spirit" + 0.011*"husband" + '
  '0.011*"place" + 0.010*"evening"'),
 (4,
  '0.022*"letter" + 0.013*"man" + 0.012*"manner" + 0.011*"father" + '
  '0.011*"way" + 0.011*"ease" + 0.010*"time" + 0.010*"distress" + '
  '0.010*"honour" + 0.009*"anger"'),
 (5,
  '0.016*"respect" + 0.015*"darcy" + 0.014*"summer" + 0.014*"y[...]<jupyter_text>Run model multiple times to find best fit<jupyter_code># Can take a long time to run.
model_list, coherence_values = lda_utils.compute_coherence_values(texts=data_processed,
                                                                  corpus=corpus, 
                                                                  dictionary=id2word,  
                                                                  start=5, 
                                                                  limit=40,  
                                                                  step=5)<jupyter_output><empty_output><jupyter_text>Create dataframe showing most dominant topic per chunk<jupyter_code>df_topic_keywords = lda_utils.format_topics_sentences(ldamodel=lda_model, 
                                                      corpus=corpus, 
                                                      texts=data_processed)

# Format
df_dominant_topic = df_topic_keywords.reset_index()
df_dominant_topic.columns = ['Document_No', 'Dominant_Topic', 'Topic_Perc_Contrib', 'Keywords', 'Text']
df_dominant_topic.sample(10)
# Display setting to show more characters in column
pd.options.display.max_colwidth = 100

sent_topics_sorteddf = pd.DataFrame()
sent_topics_outdf_grpd = df_topic_keywords.groupby('Dominant_Topic')

for i, grp in sent_topics_outdf_grpd:
    sent_topics_sorteddf = pd.concat([sent_topics_sorteddf, 
                                      grp.sort_values(['Perc_Contribution'], ascending=False).head(1)], 
                                      axis=0)

# Reset Index    
sent_topics_sorteddf.reset_index(drop=True, inplace=True)

# Format
sent_topics_sorteddf.columns = ['Topic_Num', "Topic_Perc_Contrib", "Keywords", "Representative Text"]

# Show
sent_topics_sorteddf.head(10)<jupyter_output><empty_output><jupyter_text>Visualise topics<jupyter_code>vis = pyLDAvis.gensim.prepare(lda_model, corpus, dictionary=lda_model.id2word)
vis<jupyter_output><empty_output><jupyter_text>To dataframe<jupyter_code>values = list(lda_model.get_document_topics(corpus))<jupyter_output><empty_output><jupyter_text>__Split tuples and keep only values per topic__<jupyter_code>split = []
for entry in values:
    topic_prevelance = []
    for topic in entry:
        topic_prevelance.append(topic[1])
    split.append(topic_prevelance)<jupyter_output><empty_output><jupyter_text>__Create document-topic matrix__<jupyter_code>df = pd.DataFrame(map(list,zip(*split)))
df<jupyter_output><empty_output><jupyter_text>Plot with seaborn<jupyter_code>sns.lineplot(data=df.T.rolling(50).mean())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1193532461120711, 'fraction_numerical': 0.039249568007899284, 'mean_word_length': 3.38474025974026, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '91502', 'license': 'MIT', 'n_tokens_mistral': 2955, 'n_tokens_neox': 2664, 'n_words': 528, 'path': 'notebooks/session8.ipynb', 'repo_name': 'sofieditmer/cds-language', 'size': '8102'}