<jupyter_start><jupyter_text>Spectral Clustering [sklearn.cluster.SpectralClustering](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html)Spectral Clustering does a low-dimension embedding of the affinity matrix between samples, followed by a KMeans in the low dimensional space. It is especially efficient if the affinity matrix is sparse and the pyamg module is installed. SpectralClustering requires the number of clusters to be specified. It works well for a small number of clusters but is not advised when using many clusters.For two clusters, it solves a convex relaxation of the normalised cuts problem on the similarity graph: cutting the graph in two so that the weight of the edges cut is small compared to the weights of the edges inside each cluster. This criteria is especially interesting when working on images: graph vertices are pixels, and edges of the similarity graph are a function of the gradient of the image.Usable examples: [Segmenting the picture of a raccoon face in regions](http://scikit-learn.org/stable/auto_examples/cluster/plot_face_segmentation.html)<jupyter_code>%matplotlib inline
%load_ext autoreload
%autoreload 2
import os, sys
import numpy as np
from skimage import io
import matplotlib.pylab as plt
sys.path += [os.path.abspath('.'), os.path.abspath('..')]  # Add path to root
import notebooks.notebook_utils as uts<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>load datset<jupyter_code>p_dataset = os.path.join(uts.DEFAULT_PATH, uts.SYNTH_DATASETS_FUZZY[0])
print ('loading dataset: ({}) exists -> {}'.format(os.path.exists(p_dataset), p_dataset))

p_atlas = os.path.join(uts.DEFAULT_PATH, 'dictionary/atlas.png')
atlas_gt = io.imread(p_atlas)
nb_patterns = len(np.unique(atlas_gt))
print ('loading ({}) <- {}'.format(os.path.exists(p_atlas), p_atlas))
plt.imshow(atlas_gt, interpolation='nearest')
_ = plt.title('Atlas; unique %i' % nb_patterns)
list_imgs = uts.load_dataset(p_dataset)
print ('loaded # images: ', len(list_imgs))
img_shape = list_imgs[0].shape
print ('image shape:', img_shape)<jupyter_output>('loaded # images: ', 800)
('image shape:', (64, 64))<jupyter_text>Pre-Processing<jupyter_code>X = np.array([im.ravel() for im in list_imgs])  # - 0.5
print ('input data shape:', X.shape)

plt.figure(figsize=(7, 3))
_= plt.imshow(X, aspect='auto'), plt.xlabel('features'), plt.ylabel('samples'), plt.colorbar()
uts.show_sample_data_as_imgs(X, img_shape, nb_rows=1, nb_cols=5)<jupyter_output><empty_output><jupyter_text>SparsePCA<jupyter_code>from sklearn.cluster import SpectralClustering
sc = SpectralClustering(n_clusters=nb_patterns, n_jobs=-1) # , assign_labels='discretize'

sc.fit(X[:1200, :].T)
atlas = sc.labels_.reshape(atlas_gt.shape)
plt.imshow(atlas)<jupyter_output>('labels:', (4096,))
{'fraction_non_alphanumeric': 0.11577474641483036, 'fraction_numerical': 0.008744316194473592, 'mean_word_length': 6.486910994764398, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '229516', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 987, 'n_tokens_neox': 944, 'n_words': 265, 'path': 'notebooks/method_SpectralClust.ipynb', 'repo_name': 'Borda/pyBPDL', 'size': '2859'}