<jupyter_start><jupyter_text>Lab 04 - "Unsupervised Machine Learning"Chartered Financial Data Scientist (CFDS), Spring Term 2019 In the third lab you learned about how to utilize two **supervised** learning classification techniques namely (1) the Gaussian Naive-Bayes (Gaussian NB) classifier and (2) the k Nearest-Neighbor (kNN) classifier. In this fourth lab we will learn about an **unsupervised** machine learning technique referred to as **k-Means Clustering**. We will use this technique to classify un-labeled data (i.e., data without defined categories or groups). In general, clustering based techniques are widely used in unsupervised machine learning.The **k-Means Clustering** algorithm is one of the most popular clustering algorithms used in machine learning. The goal of k-Means Clustering is to find groups (clusters) in a given dataset. It can be used (1) to **confirm business assumptions** about what types of groups exist or (2) to **identify unknown groups** in complex data sets. Some examples of business related use cases are:>- Segment customers by purchase history;>- Segment users by activities on an application or a website;>- Group inventory by sales activity; or,>- Group inventory by manufacturing metrics.(Source: https://www.datascience.com/blog/k-means-clustering)Once the algorithm has been run and the groups are defined, any new data can be easily assigned to the correct group. Pls. don't hesitate to contact me (via <EMAIL>) in case of any difficulties with the lab content or any questions.  Lab Objectives: After today's lab you should be able to:> 1. Understand how a **k-Means Clustering** algorithm can be trained and evaluated.> 2. Know how to Python's **sklearn library** to train and evaluate arbitrary classifiers.> 3. Understand how to evaluate the **classification results** of the k-Means algorithm.> 4. Know how to select an **optimal number of clusters** or cluster means.  Setup of the Jupyter Notebook Environment Similar to the previous labs, we need to import a couple of Python libraries that allow for data analysis and data visualization. We will be using pandas, numpy, sklearn, matplotlib and seaborn library to conduct this:<jupyter_code># import the pandas data science library
import pandas as pd
import numpy as np

# import sklearn data and data pre-processing libraries
from sklearn import datasets

# import sklearn k-means classifier library
from sklearn.cluster import KMeans

# import matplotlib data visualization library
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Create nice looking plots using the **seaborn** plotting theme:<jupyter_code>plt.style.use('seaborn')<jupyter_output><empty_output><jupyter_text>Enable inline Jupyter notebook plotting:<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Suppress potential warnings:<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Unsupervised Machine Learning: k-Means Clustering  Step 1.0: Dataset Download and Data Assessment The iris dataset is a classic and very easy multi-class classification dataset. This data sets consists of 3 different types of irises’ (classes),  namely Setosa, Versicolour, and Virginica) as well as their respective petal and sepal length (features). In total, the dataset consists of **150 samples** (50 samples per class) as well as their corresponding **4 different measurements** taken for each sample.  (Source: http://www.lac.inpe.br/~rafael.santos/Docs/R/CAP394/WholeStory-Iris.html) Please, find below the list of the individual measurements (features):>- `Sepal length (cm)`>- `Sepal width (cm)`>- `Petal length (cm)`>- `Petal width (cm)`Further details on the dataset can be obtained from the following publication: *<NAME>. "The use of multiple measurements in taxonomic problems" Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to Mathematical Statistics" (John Wiley, NY, 1950)."*Let's load the dataset and conduct a preliminary data assessment:<jupyter_code>iris = datasets.load_iris()<jupyter_output><empty_output><jupyter_text>Print and inspect feature names of the dataset:<jupyter_code>iris.feature_names<jupyter_output><empty_output><jupyter_text>Print and inspect the class names of the dataset:<jupyter_code>iris.target_names<jupyter_output><empty_output><jupyter_text>Print and inspect the top 5 feature rows of the dataset:<jupyter_code>pd.DataFrame(iris.data).head(5)<jupyter_output><empty_output><jupyter_text>Print and inspect the top 5 labels of the dataset:<jupyter_code>pd.DataFrame(iris.target).head(5)<jupyter_output><empty_output><jupyter_text>Determine and print the feature dimensionality of the dataset:<jupyter_code>iris.data.shape<jupyter_output><empty_output><jupyter_text>Determine and print the label dimensionality of the dataset:<jupyter_code>iris.target.shape<jupyter_output><empty_output><jupyter_text>Let's briefly envision how the feature data is collected and recorded in the data:  Plot the data distributions of the distinct features:<jupyter_code>plt.figure(figsize=(10, 10))
iris_plot = sns.load_dataset("iris")

# supervised scenario
#sns.pairplot(iris_plot, diag_kind='hist', hue='species');

# unsupervised scenario
sns.pairplot(iris_plot, diag_kind='hist');<jupyter_output><empty_output><jupyter_text>Step 1.1. Introduction into k-Means Clustering As mentioned in the introduction **k-Means Clustering** is one of the most popular "first choice" clustering algorithms to find groups (clusters) in a given dataset $X$. Let's now briefly revisit the distinct step of the algorithm before applying it to the iris dataset. But before that let's assume:- We have dataset $X$ consisting records $x_1, x_2, x_3, ..., x_n \in \mathcal{R}^d$; - That samples are clustered around $k$ centers (the "$k$ means") denoted by $\mu_1, \mu_{2}, ..., \mu_{k} \in \mathcal{R}^d$; and,- Each sample $x_{i}$ belongs to its closest mean $\mu_{i}$.We can then iteratively perform the following steps that comprise the **k-Means Clustering** algorithm:>- **Step 1** - Pick $k$ random points $\mu_{i}$ as cluster centers called 'means'.>- **Step 2** - Assign each $x_i$ to its to nearest cluster mean by calculating its distance to each mean.>- **Step 3** - Determine the new cluster centers by calculating the average of the assigned points in each cluster.>- **Step 4** - Repeat Step 2 and 3 until none of the cluster assignments change.Note, that a single execution of all the four steps outlined above is usually referred to as 'iteration'.  Step 1.1.1. k-Means clustering in a 2-Dimensional Feature Space Now, let's see how we can apply it to the iris dataset. We will start with an introductory example of detecting the classes of the iris dataset based on two of its features namely the (1) `Petal length (cm)` and  `Petal width (cm)`. Let's first gain an intuition of those two features as well as their distribution by visualizing them accordingly:<jupyter_code># prepare plot
fig = plt.figure()
ax = fig.add_subplot(111)

# add grid
ax.grid(linestyle='dotted')

# plot petal length (3rd feature in the dataset) vs. petal width (4th feature in the dataset)
ax.scatter(iris.data[:,2], iris.data[:,3])

# add axis legends
ax.set_xlabel("[petal_length]", fontsize=10)
ax.set_ylabel("[petal_width]", fontsize=10)

# add plot title
plt.title('Petal Length vs. Petal Width', fontsize=10);<jupyter_output><empty_output><jupyter_text>Let's now define the parameters of the k-Means Clustering. We will start be specifying the number of clusters we aim to detect in the dataset:<jupyter_code>no_clusters = 3<jupyter_output><empty_output><jupyter_text>Next, we need to define a corresponding number (in our case 3) random cluster centers or 'means' that will be used in the first iteration of the algoritm:<jupyter_code>random_means = np.array([[1.0, 3.0], [2.0, 6.0], [1.0, 7.0]])<jupyter_output><empty_output><jupyter_text>Finally, we will define a maximum number of iterations that we want to run the **k-Means Clustering** algorithm. Please, note that the clustering terminates once there will be no further changes in the cluster assignments. However, it's good practice to define an upper bound:<jupyter_code>max_iterations = 10<jupyter_output><empty_output><jupyter_text>Now, we ready to init the **k-Means Clustering**  using Python's `sklearn` libary of data science algorithms:<jupyter_code>kmeans = KMeans(n_clusters=no_clusters, init=random_means, max_iter=max_iterations)<jupyter_output><empty_output><jupyter_text>Train the k-Mains classifier and learn a model of the `Petal length (cm)` and  `Petal width (cm)` features in the iris dataset:<jupyter_code>kmeans.fit(iris.data[:,2:4])<jupyter_output><empty_output><jupyter_text>Now that we have learned a model of the data, let's inspect the distinct cluster labels that have been assigned to the records of the iris dataset:<jupyter_code>labels = kmeans.labels_
print(labels)<jupyter_output>[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 2 1 1 1 1
 1 1 1 1 1 1 1 1 2 1 1 1 2 1 1 2 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1
 1 1]<jupyter_text>Furthermore, we want to inspect the coordinates of the cluster controids assigned by the algorithm:<jupyter_code>centers = kmeans.cluster_centers_
print(centers)<jupyter_output>[[1.462      0.246     ]
 [5.63777778 2.06      ]
 [4.30727273 1.36181818]]<jupyter_text>Let's plot the iris dataset records now using the two features `Petal length (cm)`, `Petal width (cm)` and `Petal length (cm)`, their corresponding learned labels as well as the controids determined by the **k-Means Clustering** algorithm:<jupyter_code># prepare plot
fig = plt.figure()
ax = fig.add_subplot(111)

# add grid
ax.grid(linestyle='dotted')

# plot petal length vs. petal width and corresponding classes
ax.scatter(iris.data[:,2], iris.data[:,3], c=labels.astype(np.float), cmap=plt.cm.Set1)

# plot cluster centers
ax.scatter(centers[:,0], centers[:,1], marker='x', c='black', s=100)

# add axis legends
ax.set_xlabel("[petal_length]", fontsize=10)
ax.set_ylabel("[petal_width]", fontsize=10)

# add plot title
plt.title('Petal Length vs. Petal Width', fontsize=10);<jupyter_output><empty_output><jupyter_text>In addition, let's inspect the distance of all dataset records $X$ to their nearest means $\mu_{i}$. Let's recall that k-Means corresponds to a local optimization of the sum of "squared erros", as expressed by:$$E(\mu_{1}, \mu_{2}, ..., \mu_{k}) = \sum_{i=1}^{n}(x_{i}-\mu_{k(i)})^{2},$$were $x_{i}$ denots a single observation in the dataset and $\mu_{k(i)}$ its closest mean in the feature space $\mathcal{R}^{d}$.We can obtain the sum of those squared distances $E(\mu_{1}, \mu_{2}, ..., \mu_{k})$ by calling the `inertia_` attribute of the kmeans clustering object. It will return the sum of squared distances of each sample to its closest cluster center:<jupyter_code>distances = kmeans.inertia_
print(distances)<jupyter_output>31.437981063156826<jupyter_text>Exercises: We recommend you to try the following exercises as part of the lab:**1. Train and evaluate the k-Means Clustering results and squared distances $E(\mu_{1}, \mu_{2}, ..., \mu_{k})$ for distinct max iterations.**> Continuously increase the number of training iterations $i$ of the k-Means Clustering starting with 1 and up to 5 iterations ($i=1,...,5$) and repeat the clustering accordingly. What can be observed in terms of the cluster means as well as the sum of squared cluster distances with increasing $i$. **2. Determine if the k-Mean algorithm always converges to the same results.**> Carefully review the k-Means algorithm and answer the question and answer to following question: Does the k-Means algorithm always converge to the same result? Please, explain your reasoning. **3. Applicability k-Means clustering to distinct data distributions.**> Consider the following data distributions. Determine which are suitable for a k-Means clustering and what $k$ value should be applied in the clustering. Please, explain your reasoning.  Step 1.1.2. k-Means Clustering in a 3-Dimensional Feature Space Enable interactive 3D plotting:<jupyter_code>%matplotlib notebook
from importlib import reload
reload(plt)
from mpl_toolkits.mplot3d import Axes3D
import seaborn as sns<jupyter_output><empty_output><jupyter_text>We will now extend the k-Means Clustering that you learned about to a 3-dimensional features space by clustering the first three features, namely `Sepal length (cm)`, `Sepal width (cm)` and `Petal length (cm)`, of the Iris dataset. Let's start by visually inspecting the 3-dimensional feature space:<jupyter_code># prepare plot
plt.rcParams['figure.figsize'] = [8, 6]
fig = plt.figure()
ax = fig.add_subplot(111)

# init 3D plotting
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=30, azim=120)

# add grid
ax.grid(linestyle='dotted')

# plot petal length (3rd feature in the dataset) vs. petal width (4th feature in the dataset)
ax.scatter(iris.data[:,0], iris.data[:,1], iris.data[:,2], s=40)

# add axis legends
ax.set_xlabel("[sepal_length]", fontsize=10)
ax.set_ylabel("[sepal_width]", fontsize=10)
ax.set_zlabel("[petal_length]", fontsize=10)

# add plot title
plt.title('Sepal Length vs. Sepal Width vs. Petal Length', fontsize=10)

# show the three dimensional plot
plt.show();<jupyter_output><empty_output><jupyter_text>We will start again by defining the max. number of iterations we aim to run the k-Means algorithm:<jupyter_code>max_iterations = 10<jupyter_output><empty_output><jupyter_text>Let's also initialize our initial means randomly:<jupyter_code>random_means = np.array([[1.0, 3.0, 3.0], [2.0, 6.0, 5.0], [1.0, 7.0, 2.0]])<jupyter_output><empty_output><jupyter_text>As well as the k-Means clustering algorithm:<jupyter_code>kmeans = KMeans(n_clusters=no_clusters, init=random_means, max_iter=max_iterations)<jupyter_output><empty_output><jupyter_text>Let's now fit the 3-dimensional data (instead of the two-dimensional data that we fitted above):<jupyter_code>kmeans.fit(iris.data[:,0:3])<jupyter_output><empty_output><jupyter_text>Let's further inspect the labels:<jupyter_code>labels = kmeans.labels_
print(labels)<jupyter_output>[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 2 2 2 1 2 2 2 2
 2 2 1 1 2 2 2 2 1 2 1 2 1 2 2 1 1 2 2 2 2 2 1 2 2 2 2 1 2 2 2 1 2 2 2 1 2
 2 1]<jupyter_text>And the learned cluster centers:<jupyter_code>centers = kmeans.cluster_centers_
print(centers)<jupyter_output>[[5.006      3.428      1.462     ]
 [5.86833333 2.74       4.38166667]
 [6.8525     3.07       5.6925    ]]<jupyter_text>Let's plot the iris dataset records now using the three features `Sepal length (cm)`, `Sepal width (cm)` and `Petal length (cm)`, their corresponding learned labels as well as the controids determined by the **k-Means Clustering** algorithm:<jupyter_code># prepare plot
plt.rcParams['figure.figsize'] = [8, 6]
fig = plt.figure()
ax = fig.add_subplot(111)

# init 3D plotting
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=30, azim=120)

# add grid
ax.grid(linestyle='dotted')

# plot petal length vs. petal width and corresponding classes
ax.scatter(iris.data[:,0], iris.data[:,1], iris.data[:,2], c=labels.astype(np.float), cmap=plt.cm.Set1, s=40)

# plot cluster centers
ax.scatter(centers[:,0], centers[:,1], centers[:,2], marker='x', c='black', s=100)

# add axis legends
ax.set_xlabel("[sepal_length]", fontsize=10)
ax.set_ylabel("[sepal_width]", fontsize=10)
ax.set_zlabel("[petal_length]", fontsize=10)

# add plot title
plt.title('Sepal Length vs. Sepal Width vs. Petal Length', fontsize=10);

# show the three dimensional plot
plt.show();<jupyter_output><empty_output><jupyter_text>Finally, let's again inspect the distance of each dataset record to its nearest means calling the `inertia_` attribute of the kmeans clustering object:<jupyter_code>distances = kmeans.inertia_
print(distances)<jupyter_output>69.44196666666667<jupyter_text>Exercises: We recommend you to try the following exercises as part of the lab:**1. Apply k-Means clustering to all features of the Iris dataset.**> Use the k-Means classifier to learn a model of all four features contained in the Iris dataset, namely `Sepal length (cm)`, `Sepal width (cm)`, `Petal length (cm)` and `Petal width (cm)`.  Step 1.2. Detecting the Optimal Number of Clusters Enable inline Jupyter notebook plotting:<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Recall that, one of the basic ideas behind unsupervised machine learning methods, such as **k-Means clustering**, is to define clusters such that the total intra-cluster variation (usually measured by the total sum of squared distances) is minimized: $$k^{*} =\underset{k}{\arg \min} \sum_{i=1}^{n}(x_{i}-\mu_{k(i)})^{2},$$ were $x_{i}$ denots a single observation in the dataset and $\mu_{k(i)}$ its closest mean in the feature space $\mathcal{R}^{d}$. Challange: What is the optimal number of clusters $k$ for a given dataset? Selection of the right $k$ may result in the following issues:- if $k$ too small (undersegmentation), then the clusters are too diverse; and;- if $k$ too high (oversegmentation), then the clusters are too fine-grain.Examples:   Solution: We can then use the sum of "squared erros" $E(\mu_{1}, \mu_{2}, ..., \mu_{k})$ metric to find an optimal number of clusters $k$! This can be achieved by the execution of the so-called **'elbow'** technique defined by the following algorithm:>- **Step 1** - Compute the k-Means clustering algorithm for different number of clusters $k$.>- **Step 2** - For each $k$ calculate the sum of the within-cluster sum of squared distances $E(\mu_{1}, \mu_{2}, ..., \mu_{k})$.>- **Step 3** - For each $k$ plot the $k$ value vs. its corresponding sum of within-cluster sum of squared distances $E$. >- **Step 4** - Inspect the plot and determine the location of a bend (appropriate number of clusters). Let's utilize the **'elbow'** technique by first defining the max. number of iterations that we aim to apply to each k-Means clustering run:<jupyter_code>max_iterations = 100<jupyter_output><empty_output><jupyter_text>Now we can implement the and run the 'elbow' technique:<jupyter_code># init the list of squared distances
sum_of_squared_distances = []

# define the range of k-values to investigate
K = range(1,30)

# iterate over all k-values
for k in K:
    
    # init the k-Means clustering algorithm of the current k-value
    kmeans = KMeans(n_clusters=k, init='random', max_iter=max_iterations)
    
    # run the k-Means clustering of sepal-length and sepal-width features
    kmeans = kmeans.fit(iris.data[:,0:2])
    
    # collect the sum of within squared distances of the current k-value
    sum_of_squared_distances.append(kmeans.inertia_)<jupyter_output><empty_output><jupyter_text>Upon completion of the loop above let's inspect the distinct within-cluster sum of squared distances $E$:<jupyter_code># print the collected sum of squared distances of each k
sum_of_squared_distances<jupyter_output><empty_output><jupyter_text>Furthermore, let's plot the cluster number $k$ vs. the within-cluster sum of squared distances $E$:<jupyter_code># prepare plot
fig = plt.figure()
ax = fig.add_subplot(111)

# add grid
ax.grid(linestyle='dotted')

# plot petal length (3rd feature in the dataset) vs. petal width (4th feature in the dataset)
ax.plot(K, sum_of_squared_distances)

# add axis legends
ax.set_xlabel("$k$", fontsize=10)
ax.set_ylabel("Sum Within-Cluster Distance $E$", fontsize=10)

# add plot title
plt.title('Cluster number $k$ vs. Within-Cluster $E$', fontsize=10);<jupyter_output><empty_output><jupyter_text>**1. Determine the optimal number of cluster values $k$ of all four features contained in the iris dataset.**> Determinethe optimal number of clusters $k$ needed to cluster the observations of all four features contained in the iris dataset using the **'elbow'** technique outlined above.  Lab Summary: In this fourth lab, a step by step introduction into the unsupervised **k-Means clustering** machine learning algorithm was presented. The code and exercises presented in this lab may serves as a starting point for more complex and tailored programs. You may want to execute the content of your lab outside of the Jupyter notebook environment e.g. on compute node or server. The cell below converts the lab notebook into a standalone and executable python script.<jupyter_code>!jupyter nbconvert --to script cfds_lab_04.ipynb<jupyter_output>[NbConvertApp] Converting notebook cfds_lab_04.ipynb to script
[NbConvertApp] Writing 19893 bytes to cfds_lab_04.py
{'fraction_non_alphanumeric': 0.09898246787369226, 'fraction_numerical': 0.033153394162327425, 'mean_word_length': 5.280828082808281, 'pattern_counts': {'":': 0, '<': 148, '<?xml version=': 0, '>': 173, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 49, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '102814', 'license': 'MIT', 'n_tokens_mistral': 7183, 'n_tokens_neox': 6421, 'n_words': 2674, 'path': 'Rihad-Variawa-CFDS/lab_04/cfds_colab_04.ipynb', 'repo_name': '2series/self_analysis', 'size': '20954'}