<jupyter_start><jupyter_text>Natural language processing and non-negative matrix factorization (NMF) for topic modeling<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.datasets import fetch_20newsgroups
from sklearn.decomposition import NMF, LatentDirichletAllocation
import TCD19_utils as TCD
TCD.set_plotting_style_2()

# Magic command to enable plotting inside notebook
%matplotlib inline
# Magic command to enable svg format in plots
%config InlineBackend.figure_format = 'svg'
#Set random seed
seed = np.random.seed(42)<jupyter_output><empty_output><jupyter_text>Natural language processing (NLP) vocabulary Text is by itself an unstructured data type.  * corpus: set of documents* stop words: * tokenization: Tokenization breaks unstructured data, in this case text, into chunks of information which can be counted as discrete elements. These counts of token occurrences in a document can be used directly as a vector representing that document. This immediately turns an unstructured string (text document) into a structured, numerical data structure suitable for machine learning. * n-gram :<jupyter_code>dataset = fetch_20newsgroups(shuffle=True, random_state=1,
                             remove=('headers', 'footers', 'quotes'))
documents = dataset.data
type(documents)
documents[-1]<jupyter_output><empty_output><jupyter_text>We can see that each element of the list is a document containing a news article. We will now proceed to transform our document list to a term frequency-inverse document frequency (TF-IDF) matrix. This matrix has a  (`docs`, `words`) shape, where the words will be our features. The values inside the matrix will be TF-IDF that is defined by the following formula. \begin{align}\text{TF-IDF} = \text{TF} \times \log{ \frac{N}{DF}}\end{align}Where: $\text{TF} = \text{word} \text{counts in} \, document_{i}$,  $\text{DF (document frequency) =  counts of the documents that contain} \, word_{i} $ , $\text{N = total number of documents}$. Notice that if a word is ubiquitous in all documents (e.g. stop words), the term in the right turns to zero. In this sense TF-IDF makes a robust assesment of the word frequency in a document, eliminating the bias of highly repeated words. This makes sense as ubiquitous words throughout different documents might not contain any information about the documents themselves. The sci-kit learn package has a great implementation using the `tfidf_vectorizer`. Some important arguments of this function are `max_df` threshold for the proportion of documents that have highly repeated words, (`min_df`is the low limit cutoff, if `int` it is in counts), `stop_words` let's you pick a language to eliminate stop words from, and `max_features` let's you consider the top words term frequency across the corpus.Let's compute the TF-IDF vectorizer on our documents.<jupyter_code>no_features = 1000

# NMF is able to use tf-idf
tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2,
                                   max_features=no_features,
                                   stop_words='english')

tfidf = tfidf_vectorizer.fit_transform(documents)
tfidf_feature_names = tfidf_vectorizer.get_feature_names()

type(tfidf)<jupyter_output><empty_output><jupyter_text>We can see that our TF-IDF matrix is a scipy sparse matrix and that we cannot readily visualize it's components.<jupyter_code>tfidf_feature_names[-1]<jupyter_output><empty_output><jupyter_text>We can see that our features are words.<jupyter_code>df_tfidf = pd.DataFrame(tfidf.toarray(), columns=tfidf_feature_names)

df_tfidf.head()
df_tfidf.shape<jupyter_output><empty_output><jupyter_text>Now we are ready to extract the topics from our documents.  Non-negative matrix factorization (NMF) NMF can be thought of as a clustering technique, as  it finds a decomposition of samples $\matrix{X}$ into two matrices $\matrix{W}$ and $\matrix{H}$ of non-negative elements, by minimizing the distance d between $\matrix{X}$ and the matrix product $\matrix{W}$$\matrix{H}$ . More specifically each matrix represents the following:$\matrix{W}$ (clusters) = the topics (clusters) discovered from the documents.$\matrix{H}$ (coefficient matrix) = the membership weights for the topics in each document.$\matrix{X}$ (Document-word matrix) — input that contains which words appear in which documents.\begin{align}\matrix{X} = \matrix{W}\matrix{H}\end{align}We won't go to any mathematical detail but just know that the current implementation in scikit learn uses the minimization of the Frobenius norm, the matrix analog of the euclidean distance, and that you can use different divergence measurements like the Kullback-leibler divergence by modifying the `beta_loss` parameter.<jupyter_code>import sklearn
sklearn.__version__
n_topics = 10

# Run NMF on the TF-IDF matrix
nmf = NMF(n_components=n_topics).fit(tfidf)
#Transform
nmf_W = nmf.transform(tfidf)
nmf_H = nmf.components_

print('Topic (W) matrix has a', nmf_W.shape , 'shape')
print('Coeffficient (H) matrix has a ', nmf_H.shape, 'shape')<jupyter_output><empty_output><jupyter_text>We can see that the $\matrix{W}$ matrix has a (`n_documents`, `n_topics`) shape.We can readily see that the NMF $\matrix{H}$ matrix has a shape of (`n_topics`, `n_features`) .Therefore if we want to get the topic associations from each document we must get the biggest argument from the topic matrix.<jupyter_code>nmf_topics = []
for i in range(nmf_W.shape[0]):
    nmf_topics.append(nmf_W[i].argmax())
def display_topics(model, feature_names, no_top_words):
    
    for topic_idx, topic in enumerate(model.components_):
        
        #print topic index
        print ("Topic %d:" % (topic_idx))
        
        #print topic 
        print (" ".join([feature_names[i] for i in topic.argsort()[: - no_top_words -1 :-1]]))
        
no_top_words = 15
display_topics(nmf, tfidf_feature_names, no_top_words)<jupyter_output><empty_output><jupyter_text>Voilà. We have our topics and the most important words associated with them. From this we can readily make a list that will serve for visualization purposes.<jupyter_code>classes = ['random','video','catholic church','gamers', 'bike / car selling',
           'email','windows','computer science','cybersecurity',
           'hardware']<jupyter_output><empty_output><jupyter_text>Now, let's see if by using a PCA on the topic matrix, we can visualize the documents in its document space.<jupyter_code>palette = TCD.palette(cmap = True)
from sklearn.decomposition import PCA

pca = PCA(random_state = 42)

doc_pca = pca.fit_transform(nmf_W)
plt.figure(figsize = (8,6))

plt.scatter(doc_pca[:,0], doc_pca[:, 1], alpha = 0.8,
            c = nmf_topics, cmap = palette.reversed())

plt.xlabel('PCA 1')
plt.ylabel('PCA 2')
cbar = plt.colorbar(boundaries=np.arange(11)-0.5)
cbar.set_ticks(np.arange(10))
cbar.set_ticklabels(classes)

#plt.savefig('news_pca.png', dpi = 500, bbox_inches = 'tight')
plt.tight_layout();

from umap import UMAP
reducer = UMAP(random_state = 42)

doc_umap = reducer.fit_transform(nmf_W)
plt.figure(figsize = (8,6))

plt.scatter(doc_umap[:,0], doc_umap[:, 1], alpha = 0.8, 
            c = nmf_topics, cmap = palette.reversed())

plt.xlabel('UMAP 1')
plt.ylabel('UMAP 2')
cbar = plt.colorbar(boundaries=np.arange(11)-0.5)
cbar.set_ticks(np.arange(10))
cbar.set_ticklabels(classes)

#plt.savefig('news_UMAP.png', dpi = 500, bbox_inches = 'tight')
plt.tight_layout();
reducer = UMAP(random_state = 42, y = nmf_topics)

doc_umap = reducer.fit_transform(nmf_W)
plt.figure(figsize = (8,6))

plt.scatter(doc_umap[:,0], doc_umap[:, 1], alpha = 0.8, 
            c = nmf_topics, cmap = palette.reversed())

plt.xlabel('UMAP 1')
plt.ylabel('UMAP 2')
cbar = plt.colorbar(boundaries=np.arange(11)-0.5)
cbar.set_ticks(np.arange(10))
cbar.set_ticklabels(classes)

#plt.savefig('news_UMAP_learn.png', dpi = 500, bbox_inches = 'tight')
plt.tight_layout();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10228818800247372, 'fraction_numerical': 0.01038961038961039, 'mean_word_length': 5.052395209580839, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '597482', 'license': 'MIT', 'n_tokens_mistral': 2673, 'n_tokens_neox': 2530, 'n_words': 835, 'path': 'notebooks/t17-TCD-topic_modeling_NMF.ipynb', 'repo_name': 'eflobau/TCD_19', 'size': '8085'}