<jupyter_start><jupyter_text>Read the data Dataset: https://archive.ics.uci.edu/ml/datasets/Poker+Hand<jupyter_code>import pandas as pd
pd.set_option('display.max_rows', 10)

# read the data in and add column names
data_train = pd.read_csv("poker.txt", header=None,
                        names=['S1', 'C1', 'S2', 'C2', 'S3', 'C3','S4', 'C4', 'S5', 'C5', 'CLASS'])

data_train
cluster=data_train[['S1', 'C1', 'S2', 'C2', 'S3', 'C3','S4', 'C4', 'S5', 'C5']] # get train data
cluster<jupyter_output><empty_output><jupyter_text>standarize the data (preprocessing)<jupyter_code>from sklearn import preprocessing

# standardize clustering variables to have mean=0 and sd=1 so that card suit and
# rank are on the same scale as to have the variables equally contribute to the analysis
clustervar=cluster.copy() # create a copy -> best practice

clustervar['S1']=preprocessing.scale(clustervar['S1'].astype('float64'))
clustervar['C1']=preprocessing.scale(clustervar['C1'].astype('float64'))
clustervar['S2']=preprocessing.scale(clustervar['S2'].astype('float64'))
clustervar['C2']=preprocessing.scale(clustervar['C2'].astype('float64'))
clustervar['S3']=preprocessing.scale(clustervar['S3'].astype('float64'))
clustervar['C3']=preprocessing.scale(clustervar['C3'].astype('float64'))
clustervar['S4']=preprocessing.scale(clustervar['S4'].astype('float64'))
clustervar['C4']=preprocessing.scale(clustervar['C4'].astype('float64'))
clustervar['S5']=preprocessing.scale(clustervar['S5'].astype('float64'))
clustervar['C5']=preprocessing.scale(clustervar['C5'].astype('float64'))

clus_train = clustervar
clus_train<jupyter_output><empty_output><jupyter_text>K-means<jupyter_code>from scipy.spatial.distance import cdist
from sklearn.cluster import KMeans
model = KMeans(n_clusters=3, init='random', random_state=22) # init = how to select the first cent, Lernverfahren
    
model.fit(clus_train) # learn on training data 
    
clusassign = model.predict(clus_train) # get cluster assignment
    
print(clusassign[:30]) # cluster of first 30 rows
# print(data_train[:30])
model.labels_[:100]
len(clusassign)<jupyter_output><empty_output><jupyter_text>evaluation<jupyter_code>from sklearn import metrics

score = metrics.silhouette_score(clus_train, model.labels_, sample_size=1000)
print(score)<jupyter_output><empty_output><jupyter_text>The best value is 1 and the worst value is -1. Values near 0 indicate overlapping clusters. Negative values generally indicate that a sample has been assigned to the wrong cluster, as a different cluster is more similar.https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html  PCA<jupyter_code>from sklearn.decomposition import PCA # PCA from sklearn package

pca_2 = PCA(2) # return 2 first canonical variables > want to reduce to 2 dimensions (in this example from 10)
plot_columns = pca_2.fit_transform(clus_train) # fit PCA to the train dataset
plot_columns<jupyter_output><empty_output><jupyter_text>plot<jupyter_code>import matplotlib.pylab as plt

plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model.labels_,) # plot 1st canonical variable on x axis, 2nd on y-axis
plt.xlabel('Canonical variable 1') # pc1, canonical = transformed
plt.ylabel('Canonical variable 2') #pc2
plt.title('Scatterplot of Canonical Variables for 2 Clusters')
plt.show() 
# a color per cluster
import matplotlib.pylab as plt

plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model.labels_, alpha=0.25) # plot 1st canonical variable on x axis, 2nd on y-axis
plt.xlabel('Canonical variable 1') # pc1, canonical = transformed
plt.ylabel('Canonical variable 2') #pc2
plt.title('Scatterplot of Canonical Variables for 2 Clusters')
plt.show() 
# a color per cluster<jupyter_output><empty_output><jupyter_text>Topic modelingAssignment is in this area further info: https://towardsdatascience.com/end-to-end-topic-modeling-in-python-latent-dirichlet-allocation-lda-35ce4ed6b3e0<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
tfidf_vectorizer = TfidfVectorizer(max_features=1000,
                                   stop_words='english')
# read data
with open('politics.txt','r') as pfh, open('sports.txt','r') as sfh:
    politics = pfh.read()
    sports = sfh.read()
pfh.close()
sfh.close()
politics
sports
type(sports)
data = [politics,sports]
data
type(data)
data[1]
# tfidf transformation
tfidf = tfidf_vectorizer.fit_transform(data)
tfidf.data[:50]
lda = LatentDirichletAllocation(n_components=10, max_iter=5, # LDA, n_components = number of topics to be found
                                learning_method='online', # 
                                learning_offset=50., 
                                random_state=0) # set seed
                                # doc_topic_prior = alpha = 0.01 per default
                                # topic_word_prior = beta

lda.fit(tfidf)

lda.components_<jupyter_output><empty_output><jupyter_text>every value above is connected to a word below (e. g. 0.9064... to '10')<jupyter_code>tfidf_feature_names = tfidf_vectorizer.get_feature_names()
print(tfidf_feature_names[:20])<jupyter_output><empty_output><jupyter_text>print out top 10 words<jupyter_code>import numpy as np
x =[3,5,7,1,0,9,20,5,13,6]
a = np.argsort(x) # for getting the top words later on
print(a)  # returns index with lowest number first, e. g. index 4 contains number 0
print(a[:-3])
print(a[:-3 : -1]) # last two values, first two descending values
for topic_idx, topic in enumerate(lda.components_):
    top_words = [tfidf_feature_names[i] for i in topic.argsort()[:-10-1:-1]] # -10 = last 10 values & 
    print('Topic:',topic_idx,'--',top_words) # -1 because python starts at 0<jupyter_output><empty_output><jupyter_text>evaluate based on perplexity (Ratlosigkeit, Verwirrung)<jupyter_code>pscores = []
for n_topic in [5,10,15,20,30]:
    lda = LatentDirichletAllocation(n_components=n_topic, max_iter=5,random_state=7)

    lda.fit(tfidf)

    perplexity_score = lda.perplexity(tfidf)
    print(perplexity_score)
    pscores.append(perplexity_score)

# pscores
# perplexity score of 0 is best value
## plot the perplexity score with n_topics
plt.plot([5,10,15,20,30],pscores,'r+--')
plt.xlabel('# of topics')
plt.ylabel('Perplexity score')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12832206321748701, 'fraction_numerical': 0.030193426639408712, 'mean_word_length': 5.489795918367347, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '194381', 'license': 'MIT', 'n_tokens_mistral': 2345, 'n_tokens_neox': 2206, 'n_words': 484, 'path': 'help_files/Lab3.ipynb', 'repo_name': 'ramon-schildknecht/unstructered_data_analysis', 'size': '6359'}