<jupyter_start><jupyter_text>***What is NLP?*** - Natural Language Processing- SOTA of NLP achievements: https://github.com/sebastianruder/NLP-progress- Useful libraries: nltk, spaCy, gensim, sklearn, fastai... ***Topic modeling with LDA, SVD, and NMF***<jupyter_code>import pandas as pd
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn import decomposition
from scipy import linalg
import matplotlib.pyplot as plt
%matplotlib inline
np.set_printoptions(suppress = True)<jupyter_output><empty_output><jupyter_text>We play around with Newsgroups dataset, extracted on Usenet, which was popular before the web really took off. This dataset includes 18,000 newsgroups posts with 20 topics.<jupyter_code>categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']
remove = ('headers', 'footers', 'quotes')
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, remove=remove)
newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, remove=remove)
newsgroups_train.filenames.shape, newsgroups_train.target.shape
print("\n".join(newsgroups_train.data[:2]))
num_topics, num_top_words = 6, 8<jupyter_output><empty_output><jupyter_text>Exploratory data analysis (EDA)<jupyter_code>from os import path
from PIL import Image
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
from sklearn.feature_extraction import stop_words
train = pd.DataFrame({'Content': newsgroups_train.data})
label = pd.DataFrame({'Label':newsgroups_train.target})
data = pd.concat([train, label], axis =1)
data.head(5)
# Start with one content:
text = data.Content[0]

# Create and generate a word cloud image:
wordcloud = WordCloud().generate(text)

# Display the generated image:
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()
# lower max_font_size, change the maximum number of word and lighten the background:
wordcloud = WordCloud(max_font_size=50, max_words=100, background_color="white").generate(text)
plt.figure()
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.show()
text = "\n".join(content for content in data.Content)
print ("There are {} words in the combination of all contents.".format(len(text)))

# Generate a word cloud image
wordcloud = WordCloud(stopwords=stop_words.ENGLISH_STOP_WORDS, background_color="white").generate(text)

# Display the generated image:
# the matplotlib way:
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>Stopword, stemming, lemmatization<jupyter_code>import gensim
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
import nltk
nltk.download('wordnet')
from nltk import stem
from nltk.stem import WordNetLemmatizer, SnowballStemmer
from nltk.stem.porter import *
import numpy as np
np.random.seed(2018)
stemmer = SnowballStemmer('english')
def lemmatize_stemming(text):
    return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos='v'))

def preprocess(text):
    result = []
    for token in gensim.utils.simple_preprocess(text):
        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
            result.append(lemmatize_stemming(token))
    return result
#Test a content
sample = data.Content.iloc[0]
print('Original conent: ', sample)
words = []
for word in sample.split(' '):
    words.append(word)
print('Tokenized and lemmatized document: ')
print(preprocess(sample))

processed_data = data.Content.map(preprocess)
processed_data[:5]
#Create BOW containing #times a word appears in training set
dictionary = gensim.corpora.Dictionary(processed_data)

count = 0
for k, v in dictionary.iteritems():
    print(k, v)
    count += 1
    if count > 10:
        break

#Filter out tokens with less than 15 docs (abs),1 docs(based on fraction of corpus),
#Keep first 100k frequent tokens
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)
#Gensim doc2bow - how many words and how many times those words appear per documents
bow_corpus = [dictionary.doc2bow(doc) for doc in processed_data]

#preview
bow_1310 = bow_corpus[1310]
for i in range(len(bow_1310)):
    print("Word {} (\"{}\") appears {} time.".format(bow_1310[i][0], 
                                                     dictionary[bow_1310[i][0]], 
                                                     bow_1310[i][1]))
#LDA using BOW
lda_model = gensim.models.LdaMulticore(bow_corpus, num_topics=4, id2word=dictionary, passes=2, workers=2)
for idx, topic in lda_model.print_topics(-1):
    print('Topic: {} \nWords: {}'.format(idx, topic))<jupyter_output>Topic: 0 
Words: 0.017*"space" + 0.008*"launch" + 0.007*"post" + 0.006*"nasa" + 0.006*"time" + 0.005*"program" + 0.005*"orbit" + 0.005*"satellit" + 0.005*"think" + 0.005*"problem"
Topic: 1 
Words: 0.018*"imag" + 0.009*"know" + 0.009*"space" + 0.009*"program" + 0.008*"file" + 0.008*"graphic" + 0.007*"like" + 0.007*"data" + 0.006*"peopl" + 0.006*"softwar"
Topic: 2 
Words: 0.012*"think" + 0.009*"time" + 0.008*"like" + 0.007*"space" + 0.006*"know" + 0.006*"peopl" + 0.005*"mean" + 0.005*"say" + 0.005*"object" + 0.005*"moral"
Topic: 3 
Words: 0.011*"peopl" + 0.010*"think" + 0.009*"jesus" + 0.008*"say" + 0.008*"believ" + 0.008*"know" + 0.007*"point" + 0.007*"christian" + 0.006*"thing" + 0.006*"good"<jupyter_text>[Topic Frequency-Inverse Document Frequency](http://www.tfidf.com/) (TF-IDF) is a way to normalize term counts by taking into account how often they appear in a document, how long the document is, and how commmon/rare the term is.TF = ( occurrences of term t in document) / ( of words in documents)IDF = log( of documents /  documents with term t in it)<jupyter_code>#LDA using TF-IDF
from gensim import corpora, models
from pprint import pprint

tfidf = models.TfidfModel(bow_corpus)
corpus_tfidf = tfidf[bow_corpus]

for doc in corpus_tfidf:
#     pprint(doc)
    break

lda_model_tfidf = gensim.models.LdaMulticore(corpus_tfidf, num_topics=4, id2word=dictionary, passes=2, workers=4)
for idx, topic in lda_model_tfidf.print_topics(-1):
    print('Topic: {} Word: {}'.format(idx, topic))
for index, score in sorted(lda_model_tfidf[bow_corpus[1310]], key=lambda tup: -1*tup[1]):
    print("\nScore: {}\t \nTopic: {}".format(score, lda_model_tfidf.print_topic(index, 10)))<jupyter_output>Score: 0.9875485897064209	 
Topic: 0.006*"file" + 0.005*"space" + 0.005*"imag" + 0.004*"thank" + 0.004*"format" + 0.004*"know" + 0.004*"like" + 0.004*"think" + 0.004*"work" + 0.004*"help"<jupyter_text>*** Singular Value Decomposition (SVD) *** - Topics are **orthogonal**.- The SVD algorithm factorizes a matrix into one matrix with **orthogonal** columns and one with **orthogonal** rows (along with a diagonal matrix, which contains the relative importance of each factor).<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
vectorizer = CountVectorizer(stop_words='english')
vectors = vectorizer.fit_transform(newsgroups_train.data).todense()
vectors.shape
vocab = np.array(vectorizer.get_feature_names())
vocab[5000:7234]
%time U, s, Vh = linalg.svd(vectors, full_matrices=False)
print(U.shape, s.shape, Vh.shape)
num_top_words=8

def show_topics(a):
    top_words = lambda t: [vocab[i] for i in np.argsort(t)[:-num_top_words-1:-1]]
    topic_words = ([top_words(t) for t in a])
    return [' '.join(t) for t in topic_words]
show_topics(Vh[:5])<jupyter_output><empty_output><jupyter_text>*** Non-negative Matrix Factorization (NMF) *** - A factorization of a non-negative data set 𝑉: 𝑉 = 𝑊𝐻 into non-negative matrices 𝑊,𝐻.-Factors into one skinny positive matrix and one short positive matrix.--> Fast/easy to use BUT took years of research and expertise to create<jupyter_code>m,n=vectors.shape
d=4  # num topics
clf = decomposition.NMF(n_components=d, random_state=1)
W1 = clf.fit_transform(vectors)
H1 = clf.components_

show_topics(H1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1345984667588287, 'fraction_numerical': 0.0422269699635541, 'mean_word_length': 5.433306386418755, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '81499', 'license': 'MIT', 'n_tokens_mistral': 3059, 'n_tokens_neox': 2739, 'n_words': 686, 'path': '01_Topic Modeling.ipynb', 'repo_name': 'honghanhh/nlp19_intro', 'size': '7957'}