<jupyter_start><jupyter_text>4. Topic modeling - NMFBesides LDA, there are other matrix factorization techniques such as Latent Semantic Indexing (**LSI**) and non-negative Matrix Factorization (**NMF**).NMF is similar to Principal component analysis (**PCA**), but NMF models are interpretable. Vectors are non-negative; by factoring them into the lower-dimensional form, coefficients are also non-negative.- Documents are expressed as combinations of topics- Images are expressed as combinations of patternsNMF has components (dimension of components = dimension of samples)  > **sample = feature * components**  > (transcript = _____ * topics)<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer()
csr_mat = tfidf.fit_transform(speech_df['transcript'])
# print(csr_mat)
csr_mat.toarray()
words = tfidf.get_feature_names()
tfidf_dtm = pd.DataFrame(csr_mat.toarray(), columns=tfidf.get_feature_names())
tfidf_dtm.index = speech_df.index
tfidf_dtm.iloc[:,:-1]
# Handle words to remove
add_stop_words = ['like','youre','ive','im','really','id','ve','just','dont','didnt','thi','wa',
                  'say','know','make','people']

boring_words = ['say','like','just','dont','don','im',
                'ive','youll','youve','things','thing','youre','right','really','lot',
                'make','know','people','way','day','class',
                'little', 'maybe','niagara','university','dartmouth','woman', 'womens','wellesley',
                'shirtwaist','scripps','aidan','tuskegee','dr','colleges', 'guy', 'dave',
                'arts','montgomery','girls']

import pickle
with open("../dump/common_words.txt", "rb") as f:   # Unpickling
    common_words = pickle.load(f)
    
add_stop_words = add_stop_words + common_words + boring_words

stop_words = text.ENGLISH_STOP_WORDS.union(add_stop_words)
# stop_words
def tfidf_dtm(df,column_name,add_stop_words=[]):
    """
    Input: corpus (Ex: speech_clean_2, 'transcript')
    Output: Document-Term Matrix (rows: documents, columns: words)
    
    """
    tfidf = TfidfVectorizer(stop_words=stop_words)
    data_tfidf = tfidf.fit_transform(df[column_name])

    tfidf_dtm = pd.DataFrame(data_tfidf.toarray(), columns=tfidf.get_feature_names())
    tfidf_dtm.index = df.index
    
    return tfidf_dtm
doc_word = tfidf_dtm(speech_df,'transcript')
doc_word.iloc[:,:-1]
# Use NMF model, specify number of topics
# (Following Vinny's lecture)
nmf_model = NMF(6, max_iter=800)
doc_topic = nmf_model.fit_transform(doc_word)
doc_topic.shape
# Use components in NMF model to find the top 10 words for a given topic
topics = nmf_model.components_.argsort(axis=1)[:,-1:-11:-1]

# Create topic_worrd df
words = doc_word.columns
topic_words = [[words[index] for index in topic] for topic in topics]
pd.DataFrame(topic_words,index=['Topic #' + '{:02d}'.format(i) for i in range(6)])
len(words)<jupyter_output><empty_output><jupyter_text>K-means (clustering)<jupyter_code>from sklearn.decomposition import TruncatedSVD
from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline
svd = TruncatedSVD(n_components=50)
kmeans = KMeans(n_clusters=6)
pipeline = make_pipeline(svd,kmeans)
pipeline.fit(csr_mat)
labels = pipeline.predict(csr_mat)
speakers = list(speech_df.speaker)
# speakers
speaker_label = pd.DataFrame({'label': labels, 'speaker': speakers})
speaker_label
speaker_label.to_pickle('../dump/speaker_label_nmf')<jupyter_output><empty_output><jupyter_text>NMF (DataCamp)<jupyter_code>from sklearn.decomposition import NMF
model = NMF(n_components=6)
model.fit(csr_mat)
nmf_features = model.transform(csr_mat)
# nmf_features

# Normalize the NMF features
from sklearn.preprocessing import normalize
norm_features = normalize(nmf_features)
print(model.components_)
df = pd.DataFrame(nmf_features,index=speakers)
df<jupyter_output><empty_output><jupyter_text>Recommender: cosine similarity<jupyter_code># Recommender using cosine similarity

df = pd.DataFrame(norm_features,index=speakers)
transcript = df.iloc[0]
similarities = df.dot(transcript)
# Find speeches most similar to that of SIDDHARTHA MUKHERJEE
print(similarities.nlargest())
print(similarities.nsmallest())<jupyter_output><empty_output><jupyter_text>NMF (Vinny's lecture)<jupyter_code>vectorizer = CountVectorizer(stop_words = 'english')
doc_word = vectorizer.fit_transform(speech_df.transcript)
doc_word.shape
doc_word
nmf_model = NMF(6)
doc_topic = nmf_model.fit_transform(doc_word)
doc_topic.shape
words = vectorizer.get_feature_names()
words[27030]
topics = nmf_model.components_.argsort(axis=1)[:,-1:-11:-1]
# topics
topic_words = [[words[index] for index in topic] for topic in topics]
pd.DataFrame(topic_words,index=['Topic #' + '{:02d}'.format(i) for i in range(6)])
speech_df['topic'] = doc_topic.argmax(axis=1)
speech_df.sort_values('topic').head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1363729718628055, 'fraction_numerical': 0.008215239268843705, 'mean_word_length': 6.469325153374233, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '462684', 'license': 'MIT', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1679, 'n_words': 326, 'path': 'notebook/04_topic_modeling_2.ipynb', 'repo_name': 'katiehuang1221/onl_ds5_project_4', 'size': '4869'}