<jupyter_start><jupyter_text>CLUSTERING Comparisons         Clustering is a type of **Unsupervised Machine Learning**, which can determine relationships of unlabeled data. DBSCAN stands for Density-Based Spatial Clustering of Applications with Noise.This notebook will show one approach to prepare data for exploration of DBScan, Agglomerative and KMeans.  Based on [How DBSCAN Clustering Works](https://www.analyticsvidhya.com/blog/2020/09/how-dbscan-clustering-works/) Data informationTest DBScan over features-  DBSCAN suffers from the curse of dimensionality.-  This data has over 60 dimensions, so a few features will be modeled, not all of them at once.-  Want to avoid false correlation like ice cream sales and drowning deaths, but still visualize groups and noise.  Dependencies<jupyter_code>import pandas as pd
import csv
import os 
import numpy as np
import math
import matplotlib.pyplot as plt
import matplotlib
from sklearn.cluster import DBSCAN
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import normalize
from sklearn.decomposition import PCA
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Preparing DataUse pandas to prepare data for machine learning.<jupyter_code>#Reading in summarized feature set

file_name = os.path.join(os.getcwd(), "summary_out_Text2.csv")

df = pd.read_csv(file_name,skipinitialspace=True)<jupyter_output><empty_output><jupyter_text>Look at the structure of the data.<jupyter_code>df.head()
df.columns
df.shape<jupyter_output><empty_output><jupyter_text>Select columns that will be used as features for Machine Learning.<jupyter_code>#load two features
x = df[['dollars_obligated','MaxTxYear']]<jupyter_output><empty_output><jupyter_text>Dealing with missing values can be done by removing rows with missing data....<jupyter_code>#check number of rows
print ("original number of rows:  %d" % (len(x.index)))

#see the nan rows
x[x.isna().any(axis=1)]

#remove rows
x1 = x.dropna()
print ("new number of rows:  %d" % (len(x1.index)))

## confirm that there are no null values
x1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 207714 entries, 0 to 207713
Data columns (total 2 columns):
 #   Column             Non-Null Count   Dtype  
---  ------             --------------   -----  
 0   dollars_obligated  207714 non-null  float64
 1   MaxTxYear          207714 non-null  int64  
dtypes: float64(1), int64(1)
memory usage: 3.2 MB<jupyter_text>To see variance of features: boxplot (from Seaborn) can be used with the MinMaxscaler (from scikit-learn) to visualize this.<jupyter_code>scaler = StandardScaler()
X_scaled = scaler.fit_transform(x1)

fmX = pd.DataFrame(X_scaled)

ax = sns.boxplot(data=fmX)
ax
#look at the box plot of the unscaled data
ax = sns.boxplot(data=x1)
#scatter plot the first 2 columns
plt.figure(figsize=(10,10))
plt.scatter(fmX[0],fmX[1],s=15,color='grey')
plt.title('Dataset',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
#start with KMeans cluster of the 2
k_means=KMeans(n_clusters=4,random_state=42)
k_means.fit(fmX[[0,1]])
fmX['KMeans_labels']=k_means.labels_
# Plotting resulting clusters
colors=['purple','red','blue','green']
plt.figure(figsize=(10,10))
plt.scatter(fmX[0],fmX[1],c=fmX['KMeans_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('K-Means Clustering Scaled Data',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
# Normalizing the data so that 
# the data approximately follows a Gaussian distribution
X_normalized = normalize(X_scaled)
  
# Converting the numpy array into a pandas DataFrame
df_X_normalized = pd.DataFrame(X_normalized)
#start with KMeans cluster of the 2
k_means_norm=KMeans(n_clusters=4,random_state=42)
k_means_norm.fit(df_X_normalized[[0,1]])
df_X_normalized['KMeans_labels']=k_means_norm.labels_
# Plotting resulting clusters
colors=['purple','red','blue','green']
plt.figure(figsize=(10, 10))
plt.scatter(df_X_normalized[0],df_X_normalized[1],c=df_X_normalized['KMeans_labels']
            ,cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('K-Means Clustering Scaled and Normalized Features',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()

#picking a subset of data
df_X_normalized.describe()
df_samp = df_X_normalized.sample(frac = 0.10)
df_samp.describe()
df_samp.shape
#Look at agglomerative - selected a very small sample since memory error
#running this took 80% of memory -any more pings my pc
from sklearn.cluster import AgglomerativeClustering
model = AgglomerativeClustering(n_clusters=4, affinity='euclidean')
model.fit(df_samp[[0,1]])
df_samp['HR_labels']=model.labels_
# Plotting resulting clusters
plt.figure(figsize=(10,10))
plt.scatter(df_samp[0],df_samp[1],c=df_samp['HR_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('Hierarchical Clustering',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
#finally DBScan, though both heirarchical and KMeans did pretty well
from sklearn.cluster import DBSCAN
dbscan=DBSCAN()
dbscan.fit(df_samp[[0,1]])
df_samp['DBSCAN_labels']=dbscan.labels_
df_samp.shape
# Plotting resulting clusters
plt.figure(figsize=(10,10))
plt.scatter(df_samp[0],df_samp[1],c=df_samp['DBSCAN_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('DBSCAN Clustering',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
#try with other data that has been one hot encoded
df_fs=df[['naics_code','level_3_cat_platform']]
#remove nulls
df_fs1 =df_fs.dropna()
#scale the new data and plot
scaler = StandardScaler()
df_fs_scaled = scaler.fit_transform(df_fs1)

df_scaled = pd.DataFrame(df_fs_scaled)

ax = sns.boxplot(data=df_scaled)
ax
#start with KMeans cluster of the 2
k_means=KMeans(n_clusters=4,random_state=42)
k_means.fit(df_scaled[[0,1]])
df_scaled['KMeans_labels']=k_means.labels_
# Plotting resulting clusters
colors=['purple','red','blue','green']
plt.figure(figsize=(10,10))
plt.scatter(df_scaled[0],df_scaled[1],c=df_scaled['KMeans_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('K-Means Clustering Scaled Data',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
# Normalizing the data so that 
# the data approximately follows a Gaussian distribution
df_norm = normalize(df_scaled)
  
# Converting the numpy array into a pandas DataFrame
df_norm = pd.DataFrame(df_norm)
#start with KMeans cluster of the 2
k_means_norm=KMeans(n_clusters=4,random_state=42)
k_means_norm.fit(df_norm[[0,1]])
df_norm['KMeans_labels']=k_means_norm.labels_
# Plotting resulting clusters
colors=['purple','red','blue','green']
plt.figure(figsize=(10, 10))
plt.scatter(df_norm[0],df_norm[1],c=df_norm['KMeans_labels']
            ,cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('K-Means Clustering Scaled and Normalized Features',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
#try dbscan on these two
df_samp2 = df_norm.sample(frac = 0.05)
print(df_samp2.shape, df.shape)
df_samp2.head()
dbscan.fit(df_samp2[[0,1]])
df_samp2['DBSCAN_labels']=dbscan.labels_
# Plotting resulting clusters
plt.figure(figsize=(10,10))
plt.scatter(df_samp2[0],df_samp2[1],c=df_samp2['DBSCAN_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=15)
plt.title('DBSCAN Clustering',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
#conclusion DBScan works much better on the summarized data than the unsummarized<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1254841208365608, 'fraction_numerical': 0.030209140201394268, 'mean_word_length': 7.019668737060042, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '4006', 'license': 'MIT', 'n_tokens_mistral': 2982, 'n_tokens_neox': 2782, 'n_words': 617, 'path': 'ML/Clustering comparison Summarized Data.ipynb', 'repo_name': 'georgetown-analytics/acquisitioners', 'size': '7746'}