<jupyter_start><jupyter_text>Spectral Clustering This Code template is for the Cluster analysis using a Spectral Clustering algorithm and includes 2D and 3D cluster visualization of the Clusters.  Required Packages<jupyter_code>!pip install plotly
import operator
import warnings
import itertools
import numpy as np
import pandas as pd
import seaborn as sns
import plotly.express as px
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
import plotly.graph_objects as go
from sklearn.cluster import SpectralClustering
from scipy.spatial.distance import pdist, squareform
import scipy
from scipy.sparse import csgraph
from numpy import linalg as LA
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>InitializationFilepath of CSV file<jupyter_code>#filepath
file_path= ""<jupyter_output><empty_output><jupyter_text>List of features which are  required for model training .<jupyter_code>#x_values
features=[]<jupyter_output><empty_output><jupyter_text>Data FetchingPandas is an open-source, BSD-licensed library providing high-performance, easy-to-use data manipulation and data analysis tools.We will use panda's library to read the CSV file using its storage path.And we use the head function to display the initial row or entry.<jupyter_code>df=pd.read_csv(file_path)
df.head()<jupyter_output><empty_output><jupyter_text>Feature SelectionsIt is the process of reducing the number of input variables when developing a predictive model. Used to reduce the number of input variables to both reduce the computational cost of modelling and, in some cases, to improve the performance of the model.We will assign all the required input features to X<jupyter_code>X=df[features]<jupyter_output><empty_output><jupyter_text>Data PreprocessingSince the majority of the machine learning models in the Sklearn library doesn't handle string category data and Null value, we have to explicitly remove or replace null values. The below snippet have functions, which removes the null value if any exists. And convert the string classes data in the datasets by encoding them to integer classes.<jupyter_code>def NullClearner(df):
    if(isinstance(df, pd.Series) and (df.dtype in ["float64","int64"])):
        df.fillna(df.mean(),inplace=True)
        return df
    elif(isinstance(df, pd.Series)):
        df.fillna(df.mode()[0],inplace=True)
        return df
    else:return df
def EncodeX(df):
    return pd.get_dummies(df)<jupyter_output><empty_output><jupyter_text>Calling preprocessing functions on the feature and target set.<jupyter_code>x=X.columns.to_list()
for i in x:
    X[i]=NullClearner(X[i])
X=EncodeX(X)
X.head()<jupyter_output><empty_output><jupyter_text>How to select optimal number of cluster in Spectral Clustering:-   In spectral clustering, one way to identify the number of clusters is to plot the eigenvalue spectrum. If the clusters are clearly defined, there should be a “gap” in the smallest eigenvalues at the “optimal” k. This is called eigengap heuristic.Eigengap heuristic suggests the number of clusters k is usually given by the value of k that maximizes the eigengap (difference between consecutive eigenvalues). The larger this eigengap is, the closer the eigenvectors of the ideal case and hence the better spectral clustering works. This method performs the eigen decomposition on a affinity matrix. Steps are:-    1. Construct the normalized affinity matrix: L = D−1/2ADˆ −1/2.    2. Find the eigenvalues and their associated eigen vectors    3. Identify the maximum gap which corresponds to the number of clusters by eigengap heuristic  Affinity matrixCalculate affinity matrix based on input coordinates matrix and the number of nearest neighbours.<jupyter_code>def getAffinityMatrix(coordinates, k = 7):
    dists = squareform(pdist(coordinates)) 
    knn_distances = np.sort(dists, axis=0)[k]
    knn_distances = knn_distances[np.newaxis].T
    local_scale = knn_distances.dot(knn_distances.T)
    affinity_matrix = -pow(dists,2)/ local_scale
    affinity_matrix[np.where(np.isnan(affinity_matrix))] = 0.0
    affinity_matrix = np.exp(affinity_matrix)
    np.fill_diagonal(affinity_matrix, 0)
    return affinity_matrix

def eigenDecomposition(A, plot = True, topK = 10): #A: Affinity matrix
    L = csgraph.laplacian(A, normed=True)
    n_components = A.shape[0]
    eigenvalues, eigenvectors = LA.eig(L)   
    if plot:
        plt.figure(1,figsize=(20,8))
        plt.title('Largest eigen values of input matrix')
        plt.scatter(np.arange(len(eigenvalues)), eigenvalues)
        plt.grid()
    index_largest_gap = np.argsort(np.diff(eigenvalues))[::-1][:topK]
    nb_clusters = index_largest_gap + 1        
    return nb_clusters
affinity_matrix = getAffinityMatrix(X, k = 10)
k = eigenDecomposition(affinity_matrix)
k.sort()
print(f'Top 10 Optimal number of clusters {k}')<jupyter_output>Top 10 Optimal number of clusters [ 2  3  5  6  9 11 28 41 48 54]<jupyter_text>ModelSpectral Clustering is very useful when the structure of the individual clusters is highly non-convex, or more generally when a measure of the center and spread of the cluster is not a suitable description of the complete cluster, such as when clusters are nested circles on the 2D plane. Model Tuning Parameters   > - n_clusters -> The dimension of the projection subspace.   > - eigen_solver -> The eigenvalue decomposition strategy to use. AMG requires pyamg to be installed. It can be faster on very large, sparse problems, but may also lead to instabilities. If None, then 'arpack' is used.      > - n_components -> Number of eigenvectors to use for the spectral embedding.   > - gamma -> Kernel coefficient for rbf, poly, sigmoid, laplacian and chi2 kernels. Ignored for affinity='nearest_neighbors'.[More information](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html)<jupyter_code>model = SpectralClustering(n_clusters=5, affinity='nearest_neighbors' ,random_state=101)
ClusterDF = X.copy()
ClusterDF['ClusterID'] = model.fit_predict(X)
ClusterDF.head()<jupyter_output><empty_output><jupyter_text>Cluster RecordsThe below bar graphs show the number of data points in each available cluster.<jupyter_code>ClusterDF['ClusterID'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Cluster PlotsBelow written functions get utilized to plot 2-Dimensional and 3-Dimensional cluster plots on the available set of features in the dataset. Plots include different available clusters along with cluster centroid.<jupyter_code>def Plot2DCluster(X_Cols,df):
    for i in list(itertools.combinations(X_Cols, 2)):
        plt.rcParams["figure.figsize"] = (8,6)
        xi,yi=df.columns.get_loc(i[0]),df.columns.get_loc(i[1])
        for j in df['ClusterID'].unique():
            DFC=df[df.ClusterID==j]
            plt.scatter(DFC[i[0]],DFC[i[1]],cmap=plt.cm.Accent,label=j)
        plt.xlabel(i[0])
        plt.ylabel(i[1])
        plt.legend()
        plt.show()

def Plot3DCluster(X_Cols,df):
    for i in list(itertools.combinations(X_Cols, 3)):
        xi,yi,zi=df.columns.get_loc(i[0]),df.columns.get_loc(i[1]),df.columns.get_loc(i[2])
        fig,ax = plt.figure(figsize = (16, 10)),plt.axes(projection ="3d")
        ax.grid(b = True, color ='grey',linestyle ='-.',linewidth = 0.3,alpha = 0.2)
        for j in df['ClusterID'].unique():
            DFC=df[df.ClusterID==j]
            ax.scatter3D(DFC[i[0]],DFC[i[1]],DFC[i[2]],alpha = 0.8,cmap=plt.cm.Accent,label=j)
        ax.set_xlabel(i[0])
        ax.set_ylabel(i[1])
        ax.set_zlabel(i[2])
        plt.legend()
        plt.show()
    
def Plotly3D(X_Cols,df):
    for i in list(itertools.combinations(X_Cols,3)):
        xi,yi,zi=df.columns.get_loc(i[0]),df.columns.get_loc(i[1]),df.columns.get_loc(i[2])
        fig2=px.scatter_3d(df, x=i[0], y=i[1],z=i[2],color=df['ClusterID'])
        fig2.show()
Plot2DCluster(X.columns,ClusterDF)
Plot3DCluster(X.columns,ClusterDF)
Plotly3D(X.columns,ClusterDF)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09557632398753894, 'fraction_numerical': 0.013208722741433022, 'mean_word_length': 4.749283667621777, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '100289', 'license': 'Apache-2.0', 'n_tokens_mistral': 2654, 'n_tokens_neox': 2509, 'n_words': 761, 'path': 'Clustering/Spectral Clustering/SpectralClustering.ipynb', 'repo_name': 'shreepad-nade/ds-seed', 'size': '8025'}