<jupyter_start><jupyter_text>Since K-Means calculates the distance between the data points and cluster centroid, it is not scale invariant and the **data must be scaled to have mean 0 and standard deviation 1**. (Specifically, K-Means is initiated with $K$ random centroids. It calculates the distance between each of the centroids and the data points, so that the data points close to the centroids form a cluster. The centroid of the clusters is re-calculated and kept updated as more and more data points fall into the cluster. The algorithm stops when the pre-defined $K$ clusters are formed.)<jupyter_code>scaler = StandardScaler()
data.iloc[:, :-1] = scaler.fit_transform(data.iloc[:, :-1])
data.describe()
# Visualize original Iris type by petal length/width
sns.lmplot(x="petal length (cm)", 
           y="petal width (cm)", 
           hue="target", 
           data=data, 
           fit_reg=False,
           palette=sns.color_palette("muted"));
model = KMeans(n_clusters=3)  # 3 clusters since there are 3 iris types
model.fit(data.iloc[:, 2:4])  # cluster only by petal length/width<jupyter_output><empty_output><jupyter_text>Since K-Means is unsupervised, it doesn't know which label (0, 1 or 2) to assign to each cluster. As shown below, the model correctly identified clusters 0 and 1 but switched the labels, so let's switch them back without changing the clusters.<jupyter_code>model.labels_
np.choose(model.labels_, [1, 0, 2]).astype(np.int64)
data["pred"] = np.choose(model.labels_, [1, 0, 2]).astype(np.int64)
sns.lmplot(x="petal length (cm)", 
           y="petal width (cm)", 
           hue="pred", 
           data=data, 
           fit_reg=False,
           palette=sns.color_palette("muted"));
print(classification_report(data["target"], data["pred"], target_names=iris.target_names))
model.cluster_centers_<jupyter_output><empty_output><jupyter_text>**Disadvantages of K-Means:*** Need to pre-determine $K$. This is not a problem if you have domain knowledge of many $K$ the data should have. Most often, the point of clustering is to identify different subgroups of the data. [Silhouette analysis](http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html) is used to help choose $K$.* Since K-Means is initiated with random cluster centroids, running the algorithm multiple times gives different values for the final cluster centroids and membership of data points to clusters thereof. Therefore, K-Means must be run many times.* K-Means can only cluster data points with linear boundaries. (For example, it may not be able to cluster data points arranged in two concentric circles.) Spectral clustering helps form clusters with non-linear boundaries.* Clusters formed by K-Means clustering may be distorted in the presense of outliers. Gaussian mixed model (GMM) clustering accommodates the presence of outliers.As shown below, K-Means returns different results during different iterations. The centroids during each of the 5 iterations are different, which in-turn affects the cluster membership of data points. The accuracy of correctly identifying each of the three iris types varies from 3%, 37%, 96%.<jupyter_code>for k in range(1, 6):
    model = KMeans(n_clusters=3)
    model.fit(data.iloc[:, 2:4])
    data["pred"] = np.choose(model.labels_, [1, 0, 2]).astype(np.int64)
    print(classification_report(data["target"], data["pred"], target_names=iris.target_names))
    print model.cluster_centers_
silhouette_score(data.iloc[:, 2:4], model.labels_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09546606589693044, 'fraction_numerical': 0.011827654181920586, 'mean_word_length': 4.656050955414012, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '476760', 'license': 'MIT', 'n_tokens_mistral': 1071, 'n_tokens_neox': 1009, 'n_words': 400, 'path': 'labs/chapter10/kMeans.ipynb', 'repo_name': 'dsnair/ISLR', 'size': '3551'}