<jupyter_start><jupyter_text>Clustering  **In this notebook:*** Setting up a Spark cluster* Loading MNIST dataset* Clustering data with K-means    * On full dimension    * On reduced dimension    * On 2 dimensons  Imports<jupyter_code>!pip install pyspark
import os
import csv
import math
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from pyspark.sql import SQLContext
from pyspark.sql.functions import *
from pyspark import SparkContext, SparkConf
from pyspark.ml.clustering import *
from pyspark.ml.feature import *
from pyspark.ml.linalg import Vectors
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import BinaryClassificationEvaluator

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Evironment Connecting to Spark cluster.<jupyter_code>conf = SparkConf().setAppName("HW2").setMaster("local[4]")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)<jupyter_output><empty_output><jupyter_text>Data Loading Loading MNIST data into Spark data frame and transforming to a feature vector.<jupyter_code>data_path = "../../../data/HW2/mnist_test.csv"
mnist = (
    sqlContext.read.format("csv")
    .options(
        header="false",
        inferSchema="True",
        delimiter=",",
    )
    .load(os.path.abspath(data_path))
)
mnist = mnist.cache()
print(f"Number of samples: {mnist.count()}")
print(f"Number of elements per sample: {len(mnist.columns)}")
print(f"Width and length of the images: {math.sqrt(len(mnist.columns))}")
plt.imshow(np.array(mnist.collect()[2]).reshape(28, 28),  cmap='Greys')
# Plotting Example
rows = mnist.collect()
# Plotting Examples
plt.figure(figsize=(40, 10))
for count, row in enumerate(rows[:40]):

    digits = plt.subplot(4, 10, count + 1)
    digits.imshow(np.array(row).reshape(28, 28), interpolation="nearest")
    digits.set_xticks(())
    digits.set_yticks(())
    digits.set_xlabel("Row %i" % count)
plt.show()
# Vector sampler
assembler = VectorAssembler(inputCols=mnist.columns, outputCol="features")

output = assembler.transform(mnist)<jupyter_output><empty_output><jupyter_text>Clustering with Full Dimension I have tested different clustering algorithms (Kmeans and BisectingKMeansModel) and parameters (max iteration and seed). Kmeans performed better. Also, I tried to set custom cluster centroids, but this is not possible in ML API.  Training K-Means<jupyter_code># Trains a k-means model.
kmeans = KMeans(
    featuresCol="features",
    k=10,
    seed=17,
    maxIter=0,   # Change Iteration to 15 for training
    initMode="k-means||",
    initSteps=6,
)
model = kmeans.fit(output)<jupyter_output><empty_output><jupyter_text>Vizualize the Centroids Each cluster has a center. For each sample, the distance to all cluster centers is calculated and assigned to the cluster where the distance is smallest. Since we have 10 classes with images of numbers from 0-9 in the dataset, the cluster centers should cover every number at best. However, the handwritten numbers 9 and 4, and 7 and 1 are very close.<jupyter_code># Save cluster centers
centers = model.clusterCenters()

with open("output.csv", "w") as f:
    csvwriter = csv.writer(f)
    csvwriter.writerows(model.clusterCenters())
# Visualize cluster centers
plt.figure(figsize=(10, 5))
scale = np.abs(centers).max()
for i in range(10):
    l1_plot = plt.subplot(2, 5, i + 1)
    l1_plot.imshow(
        centers[i].reshape(28, 28),
        interpolation="nearest",#  cmap=plt.cm.RdBu,
        vmin=-scale,
        vmax=scale,
    )
    l1_plot.set_xticks(())
    l1_plot.set_yticks(())
    l1_plot.set_xlabel("Class %i" % i)
plt.suptitle("Centroids vectors")
# Save visualization
plt.savefig("cluster_centers")<jupyter_output><empty_output><jupyter_text>^ after 0 iterations.  max iterations: * 1500 is quite good, same as 3000, same as 5000 (with seed 17 and init step 10)* with init 5,2,3  its worse -> 4 is better * max iter 20 is better than 1500 with seed 17 and init step 4  Clustering on Lower Dimension In addition to the clustering on full dimension, I also clustered the dataset on lower dimesion. Therefore, I used PCA and visualized the clusteres by reconstructing them to the original size.   Lowering Dimension<jupyter_code>pca = PCA(k=100, inputCol="features", outputCol="pcaFeatures")
pca_model = pca.fit(output)
pca_features = pca_model.transform(output).select("pcaFeatures")<jupyter_output><empty_output><jupyter_text>Training K-Means<jupyter_code># Trains a k-means model.
kmeans = KMeans(
    featuresCol="pcaFeatures",
    k=10,
    seed=17,
    maxIter=20,
    initMode="k-means||",
    initSteps=6,
)
model = kmeans.fit(pca_features)<jupyter_output><empty_output><jupyter_text>Visualize the Clusters<jupyter_code>predictions = model.transform(pca_features)
centers = model.clusterCenters()
# Visualize cluster centers
plt.figure(figsize=(10, 5))
scale = np.abs(centers).max()
for i in range(10):
    l1_plot = plt.subplot(2, 5, i + 1)
    l1_plot.imshow(
        np.dot(centers[i], pca_model.pc.toArray().T).reshape(28, 28),
        interpolation="nearest",
        cmap='Greys',
    )
    l1_plot.set_xticks(())
    l1_plot.set_yticks(())
    l1_plot.set_xlabel("Class %i" % i)
plt.suptitle("Centroids vectors")
# Save visualization
plt.savefig("cluster_centers")<jupyter_output><empty_output><jupyter_text>Clustering on Two Dimensions In other to vizualize the cluster centers in 2d I also reduced the dimension to 2. Here we can see that too much information compared to the original data.  Lowering Dimension<jupyter_code>pca = PCA(k=2, inputCol="features", outputCol="pcaFeatures")
pca_model = pca.fit(output)
pca_features = pca_model.transform(output).select("pcaFeatures")<jupyter_output><empty_output><jupyter_text>Training K-Means<jupyter_code># Trains a k-means model.
kmeans = KMeans(
    featuresCol="pcaFeatures",
    k=10,
    seed=17,
    maxIter=15,
    initMode="k-means||",
    initSteps=6,
)
model = kmeans.fit(pca_features)<jupyter_output><empty_output><jupyter_text>Visualize the Clusters<jupyter_code>predictions = model.transform(pca_features)
predictions.show(4)
centers = model.clusterCenters()
for i in range(10):
    x = predictions[predictions.prediction == i].toPandas()["pcaFeatures"].apply(lambda x: x[0])
    y = predictions[predictions.prediction == i].toPandas()["pcaFeatures"].apply(lambda x: x[1])
    plt.scatter(
    x,y,
    label=f'cluster {i}'
    )
    plt.scatter(
        centers[i][0], centers[i][1], 
        color = 'black'
    )
plt.ylabel("First Principle Component")
plt.xlabel("Second Principle Component")
plt.savefig("cluster_2d")
plt.show()
# Visualize cluster centers
plt.figure(figsize=(10, 5))
scale = np.abs(centers).max()
for i in range(10):
    l1_plot = plt.subplot(2, 5, i + 1)
    l1_plot.imshow(
        np.dot(centers[i], pca_model.pc.toArray().T).reshape(28, 28),
        interpolation="nearest",
        cmap='Greys',
    )
    l1_plot.set_xticks(())
    l1_plot.set_yticks(())
    l1_plot.set_xlabel("Class %i" % i)
plt.suptitle("Centroids vectors")
plt.show()
# Save visualization
#plt.savefig("cluster_centers")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12240605720695458, 'fraction_numerical': 0.02075154234436343, 'mean_word_length': 4.914593698175787, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '280650', 'license': 'MIT', 'n_tokens_mistral': 2607, 'n_tokens_neox': 2444, 'n_words': 602, 'path': '2B/clustering.ipynb', 'repo_name': 'RBendias/scalable-data-science', 'size': '7132'}