<jupyter_start><jupyter_text>SynopsisCreate an LDA topic model from scratch using collapsed Gibbs Sampling. This notebook demonstrates the values of* Using TFIDF to preprocess tokens* Using distinct words degrades performance  Configuration<jupyter_code>src_file = '20news_01.csv'
n_docs = 100
n_topics = 15
n_iters = 100
alpha = 1
beta = 1<jupyter_output><empty_output><jupyter_text>Libraries<jupyter_code>import pandas as pd
import numpy as np
import sqlite3
import re
import random
import textman as tx
from sklearn.feature_extraction.text import TfidfVectorizer
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Pragmas<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Process  Get corpus from databaseGrab an sample from the 20news corpus, which was generated from the SQLite database using the following query:```sqlSELECT * FROM doc WHERE doc_label LIKE '%christian%'     OR doc_label LIKE '%baseball%'     OR doc_label LIKE '%politics%'     OR doc_label LIKE '%hardware%'    OR doc_label LIKE '%hockey%'ORDER BY RANDOM() LIMIT 100```<jupyter_code>docs = pd.read_csv(src_file, sep='\t')
docs = docs.set_index('doc_id')
docs.head()<jupyter_output><empty_output><jupyter_text>Show labelsGet a sense of what the corpus is about.<jupyter_code># docs.doc_label.value_counts()<jupyter_output><empty_output><jupyter_text>Convert corpus to tokens and vocabWe use a function from TextMan, a bespoke library that incorporates the text processing routines used in earlier notebooks.<jupyter_code>tokens, vocab = tx.create_tokens_and_vocab(docs, src_col='doc_content')
tokens['token_num'] = tokens.groupby(['doc_id']).cumcount()
tokens = tokens.reset_index()[['doc_id','token_num','term_id']]
tokens = tokens[tokens.term_id.isin(vocab[vocab.go].index)]
tokens = tokens.set_index(['doc_id','token_num'])<jupyter_output><empty_output><jupyter_text>We add the term strings for convenience<jupyter_code>tokens['term_str'] = tokens.term_id.map(vocab.term)
tokens.head()<jupyter_output><empty_output><jupyter_text>Remove redundant wordsWe actually *don't* do this because it ruins performance!<jupyter_code># tokens = tokens.reset_index().drop_duplicates(['doc_id','term_id']).set_index(['doc_id','token_num'])
# tokens.head()<jupyter_output><empty_output><jupyter_text>Remove insignificant wordsWe use SKlearn's TFIDF vectorizor to quicky get a TFIDF vector space, which we use only to filter the words in our corpus.<jupyter_code>vectorizer = TfidfVectorizer(use_idf=1, stop_words='english', token_pattern=r'[A-Za-z][A-Za-z][A-Za-z]+')
X = vectorizer.fit_transform(docs.doc_content.values.tolist())
v = pd.DataFrame(vectorizer.get_feature_names(), columns=['term_str'])
v['idf'] = vectorizer.idf_
v.sort_values('idf', ascending=False).head(20)<jupyter_output><empty_output><jupyter_text>We take a look at the distribution of significance<jupyter_code>v.idf.hist()<jupyter_output><empty_output><jupyter_text>We take only the most significant words<jupyter_code>cutoff = 4.5
v = v[v.idf > cutoff].sort_values('idf', ascending=False).sample(1000)
my_v = v.term_str.tolist()
# sorted(my_v)
tokens = tokens[tokens.term_str.isin(my_v)]
vocab = vocab[vocab.term.isin(my_v)]<jupyter_output><empty_output><jupyter_text>Create topics tableWe will use this to collect information about topics, and in our sampling routine.<jupyter_code>topics = pd.DataFrame(index=range(n_topics))
topics.index.name = 'topic_id'<jupyter_output><empty_output><jupyter_text>Define functions  Randomly assign topics to tokens<jupyter_code>def init_topics(tokens):
    tokens['topic_id'] = np.random.randint(0, n_topics, len(tokens))
    return tokens
# tokens = init_topics(tokens)
# tokens.head()<jupyter_output><empty_output><jupyter_text>Generate Count Matrices<jupyter_code>def get_DT(tokens):
    return tokens.groupby(['doc_id','topic_id']).topic_id.count()\
        .unstack().fillna(0).astype('int')
def get_WT(tokens):
    return tokens.groupby(['topic_id','term_id']).term_id.count()\
        .unstack().fillna(0).astype('int').T
def get_DTP(dt):
    return (dt.T / dt.T.sum()).T
def get_WTP(wt):
    wtp = wt / wt.sum()
    wtp['term_str'] = vocab.term
    wtp = wtp.set_index('term_str')
    return wtp
def get_topic_seeds(wt):
    seeds = pd.DataFrame([(t, wt[t].sort_values().idxmax()) for t in range(n_topics)], columns=['t','w'])
    seeds = seeds.set_index('t')
    seeds['term_str'] = seeds.w.map(vocab.term)
    seeds = seeds.sort_values('term_str')
    return seeds
def get_doc_seeds(dt):
    seeds = pd.DataFrame([(d, DT.loc[d].idxmax()) for d in DT.index], columns=['d','t'])
    seeds = seeds.set_index('d')
    return seeds
def add_topic_info(topics, dt, wt, n_words = 7):
    dtp = get_DTP(dt)
    wtp = get_WTP(wt)
    topics['top_words'] = ''
    topics['top_words'] = topics.apply(lambda x: ' '.join(wtp[x.name].sort_values(ascending=False).head(n_words).index.values), 1)    
    topics['df_max'] = topics.apply(lambda x: dtp[x.name].max(), 1)
    topics['df_mean'] = topics.apply(lambda x: dtp[x.name].mean(), 1)
    topics['df_sum'] = topics.apply(lambda x: dtp[x.name].sum(), 1)
    topics['n_words'] = topics.apply(lambda x: wt[x.name].sum(), 1)
    return topics<jupyter_output><empty_output><jupyter_text>Print topics<jupyter_code>def plot_topics(wtp, n=5):
    fig, axes = plt.subplots(nrows=n_topics, ncols=1, sharex=True)
    for t in range(n_topics):
        wtp[t].sort_values(ascending=True).tail(10).plot(kind='barh', ax=axes[t], figsize=(5,80))
        axes[t].set_title('Topic {}'.format(t))<jupyter_output><empty_output><jupyter_text>Use Collapsed Gibbs SamplingSee [this blog](http://brooksandrew.github.io/simpleblog/articles/latent-dirichlet-allocation-under-the-hood/).Or, better, see [Griffiths 2002](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.3760).  Define row function<jupyter_code>def get_new_topic(row):
    
    d = row.name[0] # doc_id
    k = row.name[1] # token_num (not used)
    z = row.topic_id
    w = row.term_id
    
    DT.at[d,z] -= 1
    WT.at[w,z] -= 1
    
    p_z = DT.loc[d] * (WT.loc[w] / WT.sum())
    z_weights = p_z / p_z.sum()

    z1 = topics.sample(weights=z_weights).index.values[0]

    DT.at[d,z1] += 1
    WT.at[w,z1] += 1
    
    return z1<jupyter_output><empty_output><jupyter_text>Generate model<jupyter_code>tokens = init_topics(tokens)
DT = get_DT(tokens) + alpha
WT = get_WT(tokens) + beta
topic_seeds = get_topic_seeds(WT)
doc_seeds = get_doc_seeds(WT)
topic_seeds
doc_seeds.t.value_counts()
for i in range(n_iters):
    print(i, end=' ')
    tokens['topic_id'] = tokens.apply(get_new_topic, 1)
print('Done')
topics = add_topic_info(topics, DT, WT)
topics.top_words
plot_topics(get_WTP(WT), 7)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14446751705725305, 'fraction_numerical': 0.010827647582319787, 'mean_word_length': 5.944387229660144, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '106299', 'license': 'MIT', 'n_tokens_mistral': 2609, 'n_tokens_neox': 2516, 'n_words': 467, 'path': 'labs/2019-03-21_Lab08/DS5559_LDA_Inferential.ipynb', 'repo_name': 'samarthsing/DSI', 'size': '6742'}