<jupyter_start><jupyter_text>Document Clustering and Topic ModelingIn this project, we use unsupervised learning models to cluster unlabeled documents into different groups, visualize the results and identify their latent topics/structures.Contents* [Part 1: Load Data](Part-1:-Load-Data)* [Part 2: Tokenizing and Stemming](Part-2:-Tokenizing-and-Stemming)* [Part 3: TF-IDF](Part-3:-TF-IDF)* [Part 4: K-means clustering](Part-4:-K-means-clustering)* [Part 5: Topic Modeling - Latent Dirichlet Allocation](Part-5:-Topic-Modeling---Latent-Dirichlet-Allocation)<jupyter_code>!pip install lda<jupyter_output>Collecting lda
[?25l  Downloading https://files.pythonhosted.org/packages/fd/27/d62628d914bff7f048e2b433c3adea9e7072fa20028f1d4194999051cd9d/lda-1.1.0-cp36-cp36m-manylinux1_x86_64.whl (348kB)
[K     |████████████████████████████████| 358kB 4.8MB/s 
[?25hRequirement already satisfied: numpy<2.0,>=1.13.0 in /usr/local/lib/python3.6/dist-packages (from lda) (1.17.5)
Collecting pbr<4,>=0.6
[?25l  Downloading https://files.pythonhosted.org/packages/0c/5d/b077dbf309993d52c1d71e6bf6fe443a8029ea215135ebbe0b1b10e7aefc/pbr-3.1.1-py2.py3-none-any.whl (99kB)
[K     |████████████████████████████████| 102kB 14.2MB/s 
[?25hInstalling collected packages: pbr, lda
Successfully installed lda-1.1.0 pbr-3.1.1<jupyter_text>Part 1: Load Data<jupyter_code>import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import nltk
import re
import os

from sklearn import decomposition
from sklearn.feature_extraction.text import TfidfVectorizer
import matplotlib.pyplot as plt
import lda
nltk.download('popular')<jupyter_output>[nltk_data] Downloading collection 'popular'
[nltk_data]    | 
[nltk_data]    | Downloading package cmudict to /root/nltk_data...
[nltk_data]    |   Package cmudict is already up-to-date!
[nltk_data]    | Downloading package gazetteers to /root/nltk_data...
[nltk_data]    |   Package gazetteers is already up-to-date!
[nltk_data]    | Downloading package genesis to /root/nltk_data...
[nltk_data]    |   Package genesis is already up-to-date!
[nltk_data]    | Downloading package gutenberg to /root/nltk_data...
[nltk_data]    |   Package gutenberg is already up-to-date!
[nltk_data]    | Downloading package inaugural to /root/nltk_data...
[nltk_data]    |   Package inaugural is already up-to-date!
[nltk_data]    | Downloading package movie_reviews to
[nltk_data]    |     /root/nltk_data...
[nltk_data]    |   Package movie_reviews is already up-to-date!
[nltk_data]    | Downloading package names to /root/nltk_data...
[nltk_data]    |   Package names is already up-to-date!
[nltk_data]    | Do[...]<jupyter_text>Read data from files. In summary, the data have 100 titles and 100 synoposes (combined from imdb and wiki).<jupyter_code>#import three lists: titles and wikipedia synopses
#titles = open(os.path.join(data_path, 'title_list.txt')).read().split('\n')
titles = open('title_list.txt').read().split('\n')
titles = titles[:100] #ensures that only the first 100 are read in

#The wiki synopses and imdb synopses of each movie is seperated by the keywords "BREAKS HERE". 
#Each synoposes may consist of multiple paragraphs.
#synopses_wiki = open(os.path.join(data_path, 'synopses_list_wiki.txt')).read().split('\n BREAKS HERE')
synopses_wiki = open('synopses_list_wiki.txt').read().split('\n BREAKS HERE')

synopses_wiki = synopses_wiki[:100]

#synopses_imdb = open(os.path.join(data_path, 'synopses_list_imdb.txt')).read().split('\n BREAKS HERE')
synopses_imdb = open('synopses_list_imdb.txt').read().split('\n BREAKS HERE')
synopses_imdb = synopses_imdb[:100]

#Combine imdb and wiki to get full synoposes for the top 100 movies. 
synopses = []
for i in range(len(synopses_wiki)):
    item = synopses_wiki[i] + synopses_imdb[i]
    synopses.append(item)
    
#Because these synopses have already been ordered in popularity order, 
#we just need to generate a list of ordered numbers for future usage.
ranks = range(len(titles))<jupyter_output><empty_output><jupyter_text>Part 2: Tokenizing and StemmingLoad stopwords and stemmer function from NLTK library. Stop words are words like "a", "the", or "in" which don't convey significant meaning. Stemming is the process of breaking a word down into its root.<jupyter_code># Use nltk's English stopwords.
stopwords = nltk.corpus.stopwords.words('english')

print ("We use " + str(len(stopwords)) + " stop-words from nltk library.")
print (stopwords[:10])
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")

def tokenization_and_stemming(text):
    tokens = [word.lower() for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent) if word not in stopwords]
#     tokens=[]
#     for sent in nltk.sent_tokenize(text):
#         for word in nltk.word_tokenize(sent):
#             if word not in stopwords:
#                 tokens.append(word);   
    filtered_tokens = []
    # filter out any tokens not containing letters (e.g., numeric tokens, raw punctuation)
    for token in tokens:
        if re.search('[a-zA-Z]', token):
            filtered_tokens.append(token)
    stems = [stemmer.stem(t) for t in filtered_tokens]
    return stems

def tokenization(text):
    tokens = [word.lower() for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent) if word not in stopwords]
    filtered_tokens = []
    for token in tokens:
        if re.search('[a-zA-Z]', token):
            filtered_tokens.append(token)
    return filtered_tokens
tokenization_and_stemming("she looked at her father's arm.")<jupyter_output><empty_output><jupyter_text>Use the defined functions to analyze (i.e. tokenize, stem) our synoposes.<jupyter_code>docs_stemmed = []
docs_tokenized = []
for i in synopses:
    tokenized_and_stemmed_results = tokenization_and_stemming(i)
    docs_stemmed.extend(tokenized_and_stemmed_results)
    
    tokenized_results = tokenization(i)
    docs_tokenized.extend(tokenized_results)<jupyter_output><empty_output><jupyter_text>Create a mapping from stemmed words to original tokenized words for result interpretation.<jupyter_code>vocab_frame_dict = {docs_stemmed[x]:docs_tokenized[x] for x in range(len(docs_stemmed))}
print (vocab_frame_dict['angel'])<jupyter_output>angeles<jupyter_text>Part 3: TF-IDF<jupyter_code>#define vectorizer parameters
tfidf_model = TfidfVectorizer(max_df=0.8, max_features=200000,
                                 min_df=0.2, stop_words='english',
                                 use_idf=True, tokenizer=tokenization_and_stemming, ngram_range=(1,1))

tfidf_matrix = tfidf_model.fit_transform(synopses) #fit the vectorizer to synopses

print ("In total, there are " + str(tfidf_matrix.shape[0]) + \
      " synoposes and " + str(tfidf_matrix.shape[1]) + " terms.")

tfidf_model.get_params()<jupyter_output><empty_output><jupyter_text>Save the terms identified by TF-IDF.<jupyter_code>tf_selected_words = tfidf_model.get_feature_names()<jupyter_output><empty_output><jupyter_text>Part 4: K-means clustering<jupyter_code>from sklearn.cluster import KMeans

num_clusters = 5
km = KMeans(n_clusters=num_clusters)
km.fit(tfidf_matrix)
clusters = km.labels_.tolist()<jupyter_output><empty_output><jupyter_text>4.1. Analyze K-means Result<jupyter_code># create DataFrame films from all of the input files.
films = { 'title': titles, 'rank': ranks, 'synopsis': synopses, 'cluster': clusters}
frame = pd.DataFrame(films, index = [clusters] , columns = ['rank', 'title', 'cluster'])
frame.head(10)

print ("Number of films included in each cluster:")
frame['cluster'].value_counts().to_frame()
grouped = frame['rank'].groupby(frame['cluster'])
print ("Average rank (1 to 100) per cluster:")
grouped.mean().to_frame()
print ("<Document clustering result by K-means>")

#km.cluster_centers_ denotes the importances of each items in centroid.
#We need to sort it in decreasing-order and get the top k items.
order_centroids = km.cluster_centers_.argsort()[:, ::-1] 

Cluster_keywords_summary = {}
for i in range(num_clusters):
    print ("Cluster " + str(i) + " words:", end='')
    Cluster_keywords_summary[i] = []
    for ind in order_centroids[i, :6]: #replace 6 with n words per cluster
        Cluster_keywords_summary[i].append(vocab_frame_dict[tf_selected_words[ind]])
        print (vocab_frame_dict[tf_selected_words[ind]] + ",", end='')
    print ()
    #Here ix means index, which is the clusterID of each item.
    #Without tolist, the values result from dataframe is <type 'numpy.ndarray'>
    cluster_movies = frame.ix[i]['title'].values.tolist()
    print ("Cluster " + str(i) + " titles (" + str(len(cluster_movies)) + " movies): ") 
    print (", ".join(cluster_movies))
    print()<jupyter_output><Document clustering result by K-means>
Cluster 0 words:mr.,filmed,singing,musical,love,parents,
Cluster 0 titles (14 movies): 
Raging Bull, Citizen Kane, Titanic, 2001: A Space Odyssey, Singin' in the Rain, Amadeus, Gandhi, The Best Years of Our Lives, Mr. Smith Goes to Washington, Annie Hall, Nashville, The Graduate, The African Queen, A Clockwork Orange

Cluster 1 words:car,killing,police,father,n't,meet,
Cluster 1 titles (41 movies): 
The Godfather, The Wizard of Oz, The Godfather: Part II, Psycho, Sunset Blvd., Vertigo, On the Waterfront, West Side Story, The Silence of the Lambs, Chinatown, Some Like It Hot, Unforgiven, Rocky, To Kill a Mockingbird, My Fair Lady, Jaws, The Good, the Bad and the Ugly, Butch Cassidy and the Sundance Kid, The Treasure of the Sierra Madre, The Apartment, High Noon, Goodfellas, The Exorcist, The French Connection, It Happened One Night, Rain Man, Good Will Hunting, Fargo, The Grapes of Wrath, Shane, The Green Mile, Close Encounters of the Third Kind, [...]<jupyter_text>It shows that the main topics of the 4 clusters of movies are: killing, family, wars & love 4.2. Plot K-means Result<jupyter_code>pca = decomposition.PCA(n_components=2)
tfidf_matrix_np=tfidf_matrix.toarray()
pca.fit(tfidf_matrix_np)
X = pca.transform(tfidf_matrix_np)

xs, ys = X[:, 0], X[:, 1]

#set up colors per clusters using a dict
cluster_colors = {0: '#1b9e77', 1: '#d95f02', 2: '#7570b3', 3: '#e7298a', 4: '#66a61e'}
#set up cluster names using a dict
cluster_names = {}
for i in range(num_clusters):
    cluster_names[i] = ", ".join(Cluster_keywords_summary[i])
%matplotlib inline 

#create data frame with PCA cluster results
df = pd.DataFrame(dict(x=xs, y=ys, label=clusters, title=titles)) 
groups = df.groupby(clusters)

# set up plot
fig, ax = plt.subplots(figsize=(16, 9))
#Set color for each cluster/group
for name, group in groups:
    ax.plot(group.x, group.y, marker='o', linestyle='', ms=12, 
            label=cluster_names[name], color=cluster_colors[name], 
            mec='none')

ax.legend(numpoints=1,loc=4)  #show legend with only 1 point, position is right bottom.

plt.show() #show the plot<jupyter_output><empty_output><jupyter_text>Part 5: Topic Modeling - Latent Dirichlet AllocationI will use LDA to group the movies.<jupyter_code>#Now we use synopses to build a 100*551 matrix (terms)
terms_dict = {tf_selected_words[x]:x for x in range(len(tf_selected_words))}

feature_matrix_synopses_tf = []
for i in synopses:
    current_doc_stats = [0] * len(tf_selected_words)
    allwords_stemmed = tokenization_and_stemming(i)
    for get_terms in allwords_stemmed:
        if get_terms in tf_selected_words:
            current_doc_stats[terms_dict[get_terms]] += 1
    current_doc_stats = np.asarray(current_doc_stats)
    feature_matrix_synopses_tf.append(current_doc_stats)
    
feature_matrix_synopses_tf = np.asarray(feature_matrix_synopses_tf)
model = lda.LDA(n_topics=5, n_iter=500, random_state=1)<jupyter_output><empty_output><jupyter_text>* "model.topicword" saves the importance of tf_selected_words in LDA model, i.e. words similarity matrix* The shape of "model.topicword" is (n_topics,num_of_selected_words)* "model.doctopic" saves the document topic results, i.e. document topic matrix.* The shape of "model.doctopic" is (num_of_documents, n_topics)<jupyter_code>model.fit(feature_matrix_synopses_tf)
topic_word = model.topic_word_  # model.components_ also works
n_top_words = 7

topic_keywords_list = []
for i, topic_dist in enumerate(topic_word):
    #Here we select top(n_top_words-1)
    lda_topic_words = np.array(tf_selected_words)[np.argsort(topic_dist)][:-n_top_words:-1] 
    for j in range(len(lda_topic_words)):
        lda_topic_words[j] = vocab_frame_dict[lda_topic_words[j]]
    topic_keywords_list.append(lda_topic_words.tolist())
doc_topic = model.doc_topic_
topic_doc_dict = {}

print ("<Document clustering result by LDA>")
for i in range(len(doc_topic)):
    topicID = doc_topic[i].argmax()
    if topicID not in topic_doc_dict:
        topic_doc_dict[topicID] = [titles[i]]
    else:
        topic_doc_dict[topicID].append(titles[i])
for i in topic_doc_dict:
    print ("Cluster " + str(i) + " words: " + ", ".join(topic_keywords_list[i]))
    print ("Cluster " + str(i) + " titles (" + str(len(topic_doc_dict[i])) + " movies): ")
    print (', '.join(topic_doc_dict[i]))
    print ()<jupyter_output><Document clustering result by LDA>
Cluster 2 words: father, family, new, end, life, years
Cluster 2 titles (28 movies): 
The Godfather, Casablanca, <NAME>, The Godfather: Part II, Sunset Blvd., Forrest Gump, The Sound of Music, Singin' in the Rain, Amadeus, A Streetcar Named Desire, The Best Years of Our Lives, Doctor Zhivago, Braveheart, The Good, the Bad and the Ugly, Goodfellas, City Lights, The King's Speech, A Place in the Sun, Midnight Cowboy, Mr. Smith Goes to Washington, Rain Man, Annie Hall, Out of Africa, Good Will Hunting, Terms of Endearment, Giant, Network, Yankee Doodle Dandy

Cluster 4 words: killing, death, men, use, john, shooting
Cluster 4 titles (23 movies): 
The Shawshank Redemption, On the Waterfront, Star Wars, 2001: A Space Odyssey, Chinatown, The Bridge on the River Kwai, 12 Angry Men, Gladiator, From Here to Eternity, Saving Private Ryan, Unforgiven, Raiders of the Lost Ark, To Kill a Mockingbird, Jaws, Butch Cassidy and the Sundance Kid, The Treasure of[...]<jupyter_text>Appendix: K-means¶<jupyter_code>from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=300, centers=4,
                  random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], s=50);
from sklearn.cluster import KMeans
est = KMeans(4)  # 4 clusters
est.fit(X)
y_kmeans = est.predict(X)
plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11908533645567877, 'fraction_numerical': 0.02086920586817274, 'mean_word_length': 5.004962779156328, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '48601', 'license': 'MIT', 'n_tokens_mistral': 5396, 'n_tokens_neox': 5053, 'n_words': 1352, 'path': 'NLP/clustering_Topic_modeling.ipynb', 'repo_name': 'singhbhupender1/DataSciencePortfolio', 'size': '14525'}