<jupyter_start><jupyter_text>SIT742: Modern Data Science **(Week 10: Data Analytics (III))**---- Materials in this module include resources collected from various open-source online repositories.- You are free to use, change and distribute this package.- If you found any issue/bug for this document, please submit an issue at [tulip-lab/sit742](https://github.com/tulip-lab/sit742/issues)Prepared by **SIT742 Teaching Team**--- Session 10B - Spark MLlib (4): Unsupervised LearningSpark has many libraries, namely under MLlib (Machine Learning Library)! Spark allows for quick and easy scalability of practical machine learning!In this lab exercise, you will learn how to create two different clustering algorithms with Spark MLlib! These clustering algorithms include K-Means clustering and Gaussian Mixture Clustering. We will also look at some of the information that the functions can provide about the models. Content Part 1 [K-Means Clustering](kmc) Part 2 [Gaussian Mixture Clustering](gmc) --- 1. K-Means ClusteringImport the following libraries:    KMeans, KMeansModel from pyspark.mllib.clustering    make_blobs from sklearn.datasets.samples_generator    array from numpy    sqrt from math    numpy as np<jupyter_code>!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://www-us.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
!tar xf spark-2.4.0-bin-hadoop2.7.tgz
!pip install -q findspark

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.4.0-bin-hadoop2.7"

import findspark
findspark.init() 

from pyspark import SparkContext
from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
from pyspark.mllib.clustering import KMeans, KMeansModel
from sklearn.datasets.samples_generator import make_blobs
from numpy import array
from math import sqrt
import numpy as np<jupyter_output><empty_output><jupyter_text>We will be generating our own data using the make_blobs class, so we will need to create a random seed to initialize a random number generator. Using numpy's random.seed() function, pass it the value 0<jupyter_code>np.random.seed(0)<jupyter_output><empty_output><jupyter_text>Next, we will create the dataset. Using the make_blobs function will output two numpy.ndarray values, which we will store as X and y. The make_blobs function accepts three inputs:    1st: n_samples: The number of samples you want to generate. (We will use 6500 samples)    2nd: centers: A list of lists containing the centers that we want the clusters to center around. The centers look like: [x_value, y_value]. (We will be creating three centers at the following coordinates: [1,2], [3,-1], and [-4,-2])    3rd: cluster_std: The standard deviation of the clusters, which determines how spread out the clusters are. (We will use 1.1)<jupyter_code>X, y = make_blobs(n_samples=6500, centers=[[1, 2], [3, -1], [-4, -2]], cluster_std=1.1)<jupyter_output><empty_output><jupyter_text>So the first ndarray, X contains the actual samples themselves and the second ndarray, y contains the labels for the samples. We will be using X as the data to train the KMeans model. This means we will need to convert the data from a ndarray to a RDD. Do this using sc.parallelize and passing in X, calling the output kmeans_data.<jupyter_code>kmeans_data = sc.parallelize(X)<jupyter_output><empty_output><jupyter_text>Now we will create the KMeans model called kmeans_model using KMeans.train and passing in the following parameters:    1st: The data (use kmeans_data)    2nd: The number of desired clusters (use 3)    3rd: The max number of iterations (use maxIterations=10)    4th: The number of runs (use runs=10)    5th: The initialization mode (use initializationMode="k-means||")    6th: The initialization steps (use initializationSteps=5)    7th: The value for epsilon (use epsilon=0.003)    8th: The initial model (use initialModel=None) Note: You may get a deprecation warning, please ignore it.<jupyter_code>kmeans_model = KMeans.train(kmeans_data, 3, maxIterations=10, runs=10, initializationMode="k-means||", 
                        initializationSteps=5, epsilon=0.003, initialModel=None)<jupyter_output><empty_output><jupyter_text>Now we will define a function that will evaluate the clustering by computing the Within Set Sum of Squared Error. Within Set Sum of Squared errors will find take the difference between all of the points to its cluster, square each difference, sum all of those results together, then square root it. You can see that in the function below.<jupyter_code>def error(model, point):
    center = model.centers[model.predict(point)]
    return sqrt(sum([x**2 for x in (point - center)]))<jupyter_output><empty_output><jupyter_text>Now we need to take the data (kmeans_data) and map it to the error function (.map(lambda p: error(p))). After, we add up the result of each cluster together using a .reduce function (.reduce(lambda x, y: x + y)). The Within Set Sum of Squared Error result will be stored in a variable called WSSSE. Then you can print out the WSSSE.<jupyter_code>WSSSE = kmeans_data.map(lambda p: error(kmeans_model, p)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))<jupyter_output><empty_output><jupyter_text>Now that we have built the K-Means model, let's look at some of the functions that it comes with. Starting off, let's take a look at the number of clusters in the model. This can be done by calling .k on kmeans_model.<jupyter_code>kmeans_model.k<jupyter_output><empty_output><jupyter_text>We get 3 clusters, which is expected since that's how many clusters we initialized when training the model. Now, we can take a look at the coordinates of where the cluster centers actually are. Call .clusterCenters on kmeans_model.<jupyter_code>kmeans_model.clusterCenters<jupyter_output><empty_output><jupyter_text>So here, we get a list of 3 arrays, each containing the x and y coordinates of each cluster center! Now we will try predicting one of the points from the dataset we created, X. First, let's check the shape of X with .shape. The shape should be (6500, 2), or 6500 data points, each with a x and y value.<jupyter_code>X.shape<jupyter_output><empty_output><jupyter_text>Now take we will use .predict to predict one of the data points from X, so we can see what cluster that point belongs to. We can index X to get a single data point (ex. X[7]). So try something like kmeans_model.predict(X[7])<jupyter_code>kmeans_model.predict(X[7])<jupyter_output><empty_output><jupyter_text>Look back at the make_blobs function we used. There are two inputs, which are X that has the coordinates of each data point and y that has the label for each data point. The label is the cluster where the data point was generated from! So knowing this, we can see if the model clustered the data point under the same cluster as it was made from. We can run kmeans_model.predict(X[7]) == y[7].<jupyter_code>kmeans_model.predict(X[7]) == y[7]<jupyter_output><empty_output><jupyter_text>Let's print out the coordinates of the data point (ex. print(X[7]))<jupyter_code>print(X[7])<jupyter_output><empty_output><jupyter_text>Now we have the x and y coordinates of the data points. Since k-means uses euclidean distance, you can check the coordinates of this point with each cluster center to make sure it is correct if you are curious!   Also note that you can insert a list containing a x distance and y distance and the model will determine which cluster that data point belongs to. (ex. kmeans_model.predict([5, 2]))<jupyter_code>kmeans_model.predict([5, 2])<jupyter_output><empty_output><jupyter_text>Finally, let's see the computation cost of the entire kmeans_data dataset by calling .computeCost on kmeans_model, passing in kmeans_data<jupyter_code>kmeans_model.computeCost(kmeans_data)<jupyter_output><empty_output><jupyter_text>--- 2. Gaussian Mixture ClusteringWe will now move into creating a Gaussian Mixture Clustering Model and looking at its functions as well.Import the following libraries:    GaussianMixture from pyspark.mllib.clustering    array from numpy<jupyter_code>from pyspark.mllib.clustering import GaussianMixture
from numpy import array<jupyter_output><empty_output><jupyter_text>For the dataset, we will be creating a small dataset, as it will be easier to visually see the predictions of each point. First, we will create a numpy array called data, which will contain the data. We can create a numpy array by passing in a list. The list will contain the following values: 4,5, -3,1, 9,5, 1,-2, 7,2, -10,-3, 4,7. Outside of array(), we will use .reshape(7, 2) which will give us 7 data points, where the list values correspond to the x and y coordinates. (ex. for 1 data point: array([1,2]).reshape(1, 2))<jupyter_code>data = array([4,5, -3,1, 9,5, 1,-2, 7,2, -10,-3, 4,7]).reshape(7, 2)<jupyter_output><empty_output><jupyter_text>In order to train our model, we need to turn the data into a RDD. We can run SparkContext's .parallelize, passing in the numpy array or data as input. Call the output gm_data.<jupyter_code>gm_data = sc.parallelize(data)<jupyter_output><empty_output><jupyter_text>Now we can train the Gaussian Mixture Model called gm_model, using GaussianMixture.train, passing in the following parameters:    1st: The training data (use gm_data)    2nd: The number of desired clusters (used 3)    3rd: convergenceTol (use convergenceTol=0.0006)    4th: maxIterations (use maxIterations=50)    5th: initialModel (use initialModel=None)<jupyter_code>gm_model = GaussianMixture.train(gm_data, 3, convergenceTol=0.0006, maxIterations=50, initialModel=None)<jupyter_output><empty_output><jupyter_text>Now that we have built the Gaussian Mixture Model, let's try out some functions on it! Let's start off by seeing how much clusters it has, with .k on gm_model<jupyter_code>gm_model.k<jupyter_output><empty_output><jupyter_text>There should be 3 clusters as the output, which is the value we used to train the model. Now let's take a look at what each Gaussian looks like with gm_model.gaussians<jupyter_code>gm_model.gaussians<jupyter_output><empty_output><jupyter_text>With this output, you can get a feel for the size of each of these gaussians by looking at the mean (mu) and the standard deviation (sigma). We can also look at the weight of each gaussian in the mixture in order to see which gaussians make up most of the mixture. Run  gm_model.weights<jupyter_code>gm_model.weights<jupyter_output><empty_output><jupyter_text>Now it's time to use .predict to see what each of the data points that were used to train the model are classified as. Use gm_data as input into .predict. Outside of .predict, add on .collect() so we can see the output in list form.<jupyter_code>gm_model.predict(gm_data).collect()<jupyter_output><empty_output><jupyter_text>Now let's try using .predictSoft on gm_data and see what that output looks like. Remember that predictSoft provides each gaussians membership to each datapoint, showing which gaussian has the most precedence. Don't forget a .collect() afterwards to display the output.<jupyter_code>gm_model.predictSoft(gm_data).collect()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07911166607206564, 'fraction_numerical': 0.016678558687120943, 'mean_word_length': 5.327878103837472, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 98, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '82030', 'license': 'MIT', 'n_tokens_mistral': 3514, 'n_tokens_neox': 3311, 'n_words': 1402, 'path': 'Jupyter/SIT742P10B-MLlib-Unsupervised.ipynb', 'repo_name': 'mkrahiman/sit742', 'size': '11212'}