<jupyter_start><jupyter_text>clustering* Goals    * unsupervised learning - no target var or labeled corpus - only raw features        * model needs to learn the underlying structure of the data        * find patterns    * corpus - data used to train the model (Features)* algorithms    * clustering - find patterns         * group based on common attributes         * large complexity - too many attributes for too many items         * feature vector - the combination of N features plotted on N-demensional hypercube             * distance between points define how similar they are         * convert string data to numbers    * autoencoding - find latent factors e.g. PCA|use case|example|model||---|---|---||self-sufficient unlabelled data| identify photos of a person | autoencoder||latent factor analysis | common drivers for stocks | autoencoder ||clustering | find relevant documents in a corpus | cluster ||anomaly detection | detect credit card fraud | autoencoder | |quantization | compress true color (24-bit) image to 8-bit; what are the most representative colors needed for a particular image | clustering ||pre-training  | for supervised learning | clustering or autoencoder |* Kmeans    * maximize intra-cluster similarity    * minimize inter-cluster similarity    * steps        * K centroids - pick at random or use algorithms (aka refernce vectors) or use prior iteration        * assign each point to cluster - mapped to new centroids in each step        * recalculate the centroid means        * iterate until centroids no longer move = convergence    * hyperparmaeters        * K = number of centroids        * distance measure - how to determine nearest (usu euclidian distance)        * how to update the cluster center (usu mean)    * evaluating cluster - mostly required labeled data -except silhouette        * related group 1 (usu reported together): range 0-1; similar to precision and recall; higher values are better            * homgeneity score - each cluster should contain members of the same class            * completeness - all members of a class should lie in the same cluster (inversely related to homogeneity)            * v-measure - balances homegeneity and completeness (harmonic mean)                * closer to lower value                * favors even weight for metrics                }           * ARI (adjusted Rand Index)            * measure between original labels and the assigned clusters            * adjusted for the probability of correct labeling by chance            * W<NAME>            * range -1 to 1                * 1 = agree perfect                * -1 = bad clustering - labels and cluster are independent        * adjusted mutual info            * measure of mutual information (overlap) between clusters            * range 1 = best clustering 0 or negative is bad (labels and clusters independent)        * silhouette score ***(doesn't require labels)***            * silhouette coefficient - how similar the object is to others in same cluster and different from objects in other clusters            * score is the average of coefficients for all data points       other clustering models hyperparameter tuning clustering on images<jupyter_code>!conda list
import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import random
#import warnings #should only be on integer conversions to float
#warnings.filterwarnings("ignore")
import sklearn
from sklearn import metrics
from sklearn.cluster import KMeans

#random 'toy' dataset
data_1 = np.array([[random.randint(1,400) for i in range(2)] for j in range(50)], dtype=np.float64)
data_2 = np.array([[random.randint(300,700) for i in range(2)] for j in range(50)], dtype=np.float64)
data_3 = np.array([[random.randint(600,900) for i in range(2)] for j in range(50)], dtype=np.float64)

data = np.append(np.append(data_1, data_2, axis=0), data_3, axis=0)
data.shape
fig, ax = plt.subplots(figsize=(12,8))
plt.scatter(data[:,0], data[:,1], s=200)
labels_1 = np.array([0 for i in range(50)])
labels_2 = np.array([1 for i in range(50)])
labels_3 = np.array([2 for i in range(50)])
labels = np.append(np.append(labels_1, labels_2, axis=0), labels_3, axis=0)
labels
df = pd.DataFrame({'data_x': data[:,0], 'data_y': data[:,1], 'labels': labels})

df.sample(10)
colors = ['green', 'blue','purple']
plt.figure(figsize=(12,8))
plt.scatter(df['data_x'], df['data_y'], c=df['labels'], s=200, cmap=matplotlib.colors.ListedColormap(colors))
kmeans_model = KMeans(n_clusters=3, max_iter=10000).fit(data)
kmeans_model.labels_
centroids = kmeans_model.cluster_centers_ # labels could be different than original data
centroids
fig, ax = plt.subplots(figsize=(12,8))

plt.scatter(centroids[:,0], centroids[:,1], c='r', s=250, marker='s')

for i in range(len(centroids)):
    plt.annotate(i, (centroids[i][0] + 7, centroids[i][1] + 7), fontsize=20)
print("homogeneity_score", metrics.homogeneity_score(labels, kmeans_model.labels_))
#scores are not effected by differences between the real labels and the cluster output designations
print("completeness_score", metrics.completeness_score(labels, kmeans_model.labels_))
#will probably be close to homogeneity but slightly different
print("v_measure_score", metrics.v_measure_score(labels, kmeans_model.labels_))
#fyi: harmonic mean typically used to find the average of 2 rates
print("adjusted_rand_score", metrics.adjusted_rand_score(labels, kmeans_model.labels_))
#adjusted for chance
print("adjusted_mutual_info_score", metrics.adjusted_mutual_info_score(labels, kmeans_model.labels_))
#also adjusted for chance # 0 = randomly laeled
print("silhouette_score", metrics.silhouette_score(data, kmeans_model.labels_))
#using distance measure - similar to own cluster + different from other clusters
colors = ['green', 'blue','purple']
plt.figure(figsize=(12,8))
plt.scatter(df['data_x'], df['data_y'], c=df['labels'], s=200, cmap=matplotlib.colors.ListedColormap(colors), alpha=.5)
plt.scatter(centroids[:,0], centroids[:,1], c='r', s=250, marker='s')
for i in range(len(centroids)):
    plt.annotate(i, (centroids[i][0] + 7, centroids[i][1] + 7), fontsize=20)
data_test = np.array([[442.,621.],
                     [50.,153.],
                     [333., 373.],
                     [835., 816.]])
label_pred = kmeans_model.predict(data_test)
label_pred
colors = ['green', 'blue','purple']
plt.figure(figsize=(12,8))
plt.scatter(df['data_x'], df['data_y'], c=df['labels'], s=200, cmap=matplotlib.colors.ListedColormap(colors), alpha=.5)

plt.scatter(data_test[:,0], data_test[:,1], c='orange', s=300, marker='^')
for i in range(len(label_pred)):
    plt.annotate(label_pred[i], (data_test[i][0]+7, data_test[i][1] -7), fontsize=30)

plt.scatter(centroids[:,0], centroids[:,1], c='r', s=250, marker='s')
for i in range(len(centroids)):
    plt.annotate(i, (centroids[i][0] + 7, centroids[i][1] + 7), fontsize=20)<jupyter_output><empty_output><jupyter_text>iris dataset<jupyter_code>iris_df = pd.read_csv('datasets/iris.csv', skiprows=1, 
                      names=['sepal-length','sepal-width','petal-length','petal-width','class'])
iris_df.head()
#shuffle
iris_df =iris_df.sample(frac=1).reset_index(drop=True)
iris_df.head()
iris_df['class'].value_counts()
iris_df.shape
from sklearn import preprocessing
label_encoding = preprocessing.LabelEncoder()
iris_df['class'] = label_encoding.fit_transform(iris_df['class'].astype(str))
iris_df.head()
iris_df.plot.scatter('sepal-length', 'sepal-width', figsize=(12,8), s=250)
iris_df.plot.scatter('petal-width', 'petal-length', figsize=(12,8), s=250)
iris_df.plot.scatter('sepal-length', 'petal-length', figsize=(12,8), s=250)
iris_2D = iris_df[['sepal-length','petal-length']]
iris_2D.sample(5)
iris_2D.shape
iris_2D = np.array(iris_2D)
kmeans_model_2D = KMeans(n_clusters=3, max_iter=1000).fit(iris_2D)
kmeans_model_2D.labels_ #may not match original class labels: 0 here may not map to 0 in df
centroids_2D = kmeans_model_2D.cluster_centers_

centroids_2D
fig, ax = plt.subplots(figsize=(12,8))

plt.scatter(centroids_2D[:,0], centroids_2D[:,1], c='r', s=250, marker='s')

for i in range(len(centroids_2D)):
    plt.annotate(i,(centroids_2D[i][0]  , centroids_2D[i][1]) , fontsize=25)
iris_labels = iris_df['class']
print("homogeneity_score", metrics.homogeneity_score(iris_labels, kmeans_model_2D.labels_))
#scores are not effected by differences between the real labels and the cluster output designations
print("completeness_score", metrics.completeness_score(iris_labels, kmeans_model_2D.labels_))
#will probably be close to homogeneity but slightly different
print("v_measure_score", metrics.v_measure_score(iris_labels, kmeans_model_2D.labels_))
#fyi: harmonic mean typically used to find the average of 2 rates
print("adjusted_rand_score", metrics.adjusted_rand_score(iris_labels, kmeans_model_2D.labels_))
#adjusted for chance
print("adjusted_mutual_info_score", metrics.adjusted_mutual_info_score(iris_labels, kmeans_model_2D.labels_))
#also adjusted for chance # 0 = randomly laeled
print("silhouette_score", metrics.silhouette_score(iris_2D, kmeans_model_2D.labels_))
#using distance measure - similar to own cluster + different from other clusters
colors = ['green', 'blue','purple']
plt.figure(figsize=(12,8))
plt.scatter(iris_df['sepal-length'], iris_df['petal-length'], c=iris_df['class'], s=200,
            cmap=matplotlib.colors.ListedColormap(colors), alpha=.5)

plt.scatter(centroids_2D[:,0], centroids_2D[:,1], c='r', s=250, marker='s')
for i in range(len(centroids_2D)):
    plt.annotate(i, (centroids_2D[i][0] , centroids_2D[i][1] ), fontsize=20)
iris_features = iris_df.drop('class',axis=1)
iris_features.head()
iris_labels = iris_df['class']
iris_labels.sample(5)
kmeans_model = KMeans(n_clusters=3).fit(iris_features)
kmeans_model.labels_
kmeans_model.cluster_centers_
print("homogeneity_score", metrics.homogeneity_score(iris_labels, kmeans_model.labels_))
print("completeness_score", metrics.completeness_score(iris_labels, kmeans_model.labels_))
print("v_measure_score", metrics.v_measure_score(iris_labels, kmeans_model.labels_))
print("adjusted_rand_score", metrics.adjusted_rand_score(iris_labels, kmeans_model.labels_))
print("adjusted_mutual_info_score", metrics.adjusted_mutual_info_score(iris_labels, kmeans_model.labels_))
print("silhouette_score", metrics.silhouette_score(iris_features, kmeans_model.labels_))<jupyter_output>homogeneity_score 0.7514854021988338
completeness_score 0.7649861514489815
v_measure_score 0.7581756800057784
adjusted_rand_score 0.7302382722834697
adjusted_mutual_info_score 0.7551191675800484
silhouette_score 0.5525919445499757<jupyter_text>other algorithms categories* centroid-based (e.g. kmeans) - central reference vector (may not be part of original data) distances measueres* hierarchical (e.g. agglomerative and BIRCH) - connectivity based - no central mean - points are compared with one another    * max distance needed to connect diff parts of the cluster    * tree of points where leaves are merged to create cluster    * start with each point in its own cluster and merge closest clusters iteratively    * dendrogram     * choose a different branch to get diff  of clusters    * types        * agglomerative = combine points        * divisive = start with 1 big cluster and divide (not in scikit-learn ?)    * vs kmeans: does not require a means to aggregate points, or repesent data in hypercube, easier data representation, not as scalable* distribution-based (e.g. Gaussian mixture) - based on statistica distributions - clusters are likely to belong to the same distribution    * sample random objects from distribution     * prone to over-fitting and complex* desnity-based (e.g. DBSCAN and mean shift) - create cluster from areas that have a higher density of points    * objects in sparse areas are considered noise or border points meanshift , db scan, mean shift, agglomerative, BIRCH, affinity, spectral, mini-batch<jupyter_code>from sklearn.cluster import AgglomerativeClustering, DBSCAN, MeanShift, Birch, AffinityPropagation, MiniBatchKMeans
#data reload
iris_df = pd.read_csv('datasets/iris.csv', skiprows=1, 
                      names=['sepal-length','sepal-width','petal-length','petal-width','class'])
iris_df.head()
#shuffle
iris_df = iris_df.sample(frac=1).reset_index(drop=True)
label_encoding = preprocessing.LabelEncoder()
iris_df['class'] = label_encoding.fit_transform(iris_df['class'].astype(str))
iris_df.head()
iris_features = iris_df.drop('class',axis=1)
iris_features.head()
iris_labels = iris_df['class']
iris_labels.sample(5)
def build_model(clustering_model, data, labels):
    
    model = clustering_model(data)

    res = { 'homogeneity_score': metrics.homogeneity_score(labels, model.labels_),
            'completeness_score': metrics.completeness_score(labels, model.labels_),
            'v_measure_score': metrics.v_measure_score(labels, model.labels_),
            'adjusted_rand_score': metrics.adjusted_rand_score(labels, model.labels_),
            'adjusted_mutual_info_score': metrics.adjusted_mutual_info_score(labels, model.labels_),
            'silhouette_score': metrics.silhouette_score(data, model.labels_)}
    
    print(res)
    return model, pd.Series(res).to_frame().T

            
def k_means(data, n_clusters=3, max_iter=1000):
    model = KMeans(n_clusters=n_clusters, max_iter=max_iter).fit(data)
    
    return model
(m,s) = build_model(k_means, iris_features, iris_labels)
s<jupyter_output><empty_output><jupyter_text>choosing the algorithm choose based on number of clusters and size of datasets* Birch/Agglomerative    * Birch removes outliers (noise)    * incrementally process (recalc as new data is added)    * does scale well to high dimensional data (large  features)    * agglomerative works when Euclidian distances is not present    * Mean shift/ affinity    * small datasets - large  clusters    * uneven cluster sizes and manifold shapes    * mean = pariwise distances between points - computationally intensive    * affinity uses graph distance - doesn't need  clusters up front    * kmeans/dbscan    * even cluster sizes and flat surface data    * better performance with mini-batch (faster and almost as good)    * dbscan = manifolds and uneven cluster sizes*spectral    * smaller datasets    * graph distance    * intuitive results    * even cluster sizes    * works for manifold    * reslies on distance  agglomerative clustering* bottom-up* nearness = hyperparameter    * metric: Euclidian(l2), (l1) manhattan, cosine of angles of 2 feature vectors, pre-computed matrix    * linkage criteria:         * single - min of distance between all points in 2 clusters        * complete(maximum) - max diance of all points in 2 clustesrs         * average - average distance of each points in 2 clustesrs        * ward - default - minimizes the variances of points in the 2 clusters<jupyter_code>def agglomerative_fn(data, n_clusters=3):
    model = AgglomerativeClustering(n_clusters=n_clusters).fit(data)
    return model
m,s = build_model(agglomerative_fn, iris_features, iris_labels)
s<jupyter_output>{'homogeneity_score': 0.7608008469718723, 'completeness_score': 0.7795958005591144, 'v_measure_score': 0.7700836616487869, 'adjusted_rand_score': 0.7311985567707745, 'adjusted_mutual_info_score': 0.7671669615713111, 'silhouette_score': 0.5540972908150553}<jupyter_text>dbscan clustering* Density-Based Spatial Clustering of Applications with Noise* density based* uneven cluster sizes - works on manifolds* not as good as BIRCH in outlier handling* parameters    * eps - minimum distance - points within are considered neighbors    * min_samples = min  of points needed to be dense        * greater than number of features        * noisy data should have larger value<jupyter_code>def dbscan_fn(data, eps=.45, min_samples=4):
    model=DBSCAN(eps=eps, min_samples=min_samples).fit(data)
    print(type(model))
    return model
m,s = build_model(dbscan_fn, iris_features, iris_labels)
s<jupyter_output><class 'sklearn.cluster._dbscan.DBSCAN'>
{'homogeneity_score': 0.5773205947971476, 'completeness_score': 0.6093983666695363, 'v_measure_score': 0.5929259393972258, 'adjusted_rand_score': 0.5084974632998323, 'adjusted_mutual_info_score': 0.5842965531192635, 'silhouette_score': 0.37208250029643425}<jupyter_text>mean shift clustering* define a neighborhood for all points - apply a function (kernel) to all points in the neighberhood    * flat kernel = sum of all points    * Guassian (RBF - radial basis function) kernel - probablility weighted sum of points        * Guassian distribution = normal mean &mu; and stdev &sigma;        * mean &mu; is center point and stdev &sigma; is 'bandwidth' (a hyperparameter)    * shift points toward those with higher RBF    * stop when points stop moving    * Bandwidth        * low = tall skinny - ignore pionts far from mean        * high = broad flat    * vs kmeans        * no need to specify k up-front        * can handle more complex data (e.g. image pixels)        * more hyperparameter tuning        * computationally intensive o(n2)        * better outlier handling<jupyter_code>def mean_shift_fn(data, bandwidth=.85):
    model = MeanShift(bandwidth=bandwidth).fit(data)
    return model
m,s = build_model(mean_shift_fn, iris_features, iris_labels)
s<jupyter_output>{'homogeneity_score': 0.7603645798041669, 'completeness_score': 0.7717917344958113, 'v_measure_score': 0.7660355440487252, 'adjusted_rand_score': 0.7436826319432357, 'adjusted_mutual_info_score': 0.763083127524581, 'silhouette_score': 0.5509296349732906}<jupyter_text>BIRCH clustering* Balanced Iterative Reducing and Clustering using Hierarchies    * tries to merge using trees* large dataset required - many clusters output* detects/removes outliers* incrementally processes incoming data (online learning/clustering)* very time/memory efficient (can work on parts of the data)* doesn't work well with large  features<jupyter_code>def birch_fn(data, n_clusters=3):
    model = Birch(n_clusters=n_clusters).fit(data)
    return model
m,s = build_model(birch_fn, iris_features, iris_labels)
s<jupyter_output>{'homogeneity_score': 0.7001154370964626, 'completeness_score': 0.7454382753016934, 'v_measure_score': 0.7220663465703595, 'adjusted_rand_score': 0.64225125183629, 'adjusted_mutual_info_score': 0.7184641371994781, 'silhouette_score': 0.5107111996874596}<jupyter_text>affinity propagation* small datasets into many clusters* uneven clusters sizes and manifold data* does not require K up-front* no assumptions about internal data* graph distances - nearest neighbors* exemplars - most representative of other data in the dataset* data points are nodes that pass messages to one another - messages = willingness of point to be exemplar<jupyter_code>def affinity_propagation_fn(data, damping=.6, max_iter=1000):
    """
    damping = extent to which the current value is maintained relative to incoming values = learning rate
    """
    model=AffinityPropagation(damping=damping, max_iter=max_iter).fit(data)
    return model
m,s = build_model(affinity_propagation_fn, iris_features, iris_labels)
s<jupyter_output>{'homogeneity_score': 0.8512533506223854, 'completeness_score': 0.49170090756246776, 'v_measure_score': 0.6233451996084364, 'adjusted_rand_score': 0.4373692389986675, 'adjusted_mutual_info_score': 0.6124927560312433, 'silhouette_score': 0.348833613127065}<jupyter_text>mini-batch k-means* perform kmeans on randomly sampled subsets of the data* signifcantly faster + slightly worse (trade-off generally acceptable)<jupyter_code>def mini_batch_kmeans_fn(data, n_clusters=3, max_iter=1000):
    model = MiniBatchKMeans(n_clusters=n_clusters, max_iter=max_iter, batch_size=20).fit(data)
    return model
    
m,s = build_model(mini_batch_kmeans_fn, iris_features, iris_labels)
s<jupyter_output>{'homogeneity_score': 0.7364192881252849, 'completeness_score': 0.7474865805095324, 'v_measure_score': 0.7419116631817836, 'adjusted_rand_score': 0.7163421126838475, 'adjusted_mutual_info_score': 0.7386548254402863, 'silhouette_score': 0.5509643746707443}<jupyter_text>spectral* pre-computed similarity matrix* simple/intuitive* even cluster size* works with manifold data* affinity matrix using nearest neighbors - can caclulate or pass-in as arg    * how similar points are* Eigenvalue (spectrum) decomposition behidn the scenes    * map vectors to lower dimensionality space    * apply kmeans to simplified data    * similar to applying kernel trick prior to running kmeans* dbscan is a special case of spectral<jupyter_code>from sklearn.cluster import SpectralClustering
SS=1000 #self-simliarity
IS = 10 # intra cluster self-simlarity
LS = .01 # low similarity - diff for points in separate clusters

similarity_mat = [[SS, IS, IS, LS, LS, LS, LS, LS, LS],
                  [IS, SS, IS, LS, LS, LS, LS, LS, LS],
                  [IS, IS, SS, LS, LS, LS, LS, LS, LS],
                  [LS, LS, LS, SS, IS, IS, LS, LS, LS],
                  [LS, LS, LS, IS, SS, IS, LS, LS, LS],
                  [LS, LS, LS, IS, IS, SS, LS, LS, LS],
                  [LS, LS, LS, LS, LS, LS, SS, IS, IS],
                  [LS, LS, LS, LS, LS, LS, IS, SS, IS],
                  [LS, LS, LS, LS, LS, LS, IS, IS, SS]] 
spectral_model = SpectralClustering(n_clusters=3, affinity='precomputed').fit(similarity_mat)
spectral_model.labels_<jupyter_output><empty_output><jupyter_text>hyperparameter tuning* ParamterGrid* score to evaluate models - Silhouette (doesn't require labels)    * derived from silhouette coeffecients - simlarity to same cluster, diff form other clusters    * overall score is average of point scores        * i is a given point        * bi  = mean distance of point i from all points in the neaest cluster        * ai = mean distances of point i from all other points in the same cluster        * typically between -1 to 1        * 0 scores indiciate overlap* kmeans hyperparamters    * number of clusters (most important) - sometimes obvious        * standar method to find best value            * elbow method                * pick a range of values and implement kmeans for all                * plot average distance from centroid for each value                * find elbow in plot                *             * silhouette method                * pick range of value                * plot silhouettes for each                * pick the K with the highest silhouette (closest to 1 - largest positive values (excluding outliers?))                    * silhouettes coef < 1 are outlier points                * closeness to same cluster should be much much lower than closeness to other cluster    * seeds (initial values)        * random is fine for small data        * big data techniques            * PCA - normalized - divide into bins    * distances measures - typically euclidian        * mahalanobis distance - normalize each dimensiont to have equal variance        * cosine distnace - cosine of angle between point and centroid<jupyter_code>from sklearn.model_selection import ParameterGrid
#cross-validation does not make sense for clustering
drivers_df = pd.read_csv('datasets/driver_details.csv')
drivers_df.head()
#distance - how far do they drive
#speeding - how fast do they go
drivers_df.shape
drivers_df[drivers_df.isnull().any(axis=1)]#no null values
drivers_df.describe()
drivers_features = drivers_df.drop('Driver_ID', axis=1)
parameters = {'n_clusters': [2,3,4,5,10,20,30]}

paramater_grid = ParameterGrid(parameters)

list(paramater_grid)
best_score=-1
model=KMeans()
for g in paramater_grid:
    model.set_params(**g)
    model.fit(drivers_features)
    
    ss = metrics.silhouette_score(drivers_features, model.labels_)
    print('Parameter', g, 'Score: ', ss)
    if ss > best_score:
        best_score=ss
        best_grid=g
best_grid
drivers_features.plot.scatter('Distance_Feature', 'Speeding_Feature', s=250, figsize=(12,8))<jupyter_output><empty_output><jupyter_text>dbscan HPT<jupyter_code>parameters = {'eps': [.9,1.,5.,10.,12.,14.,20.],'min_samples':[5,7,10,12]}
paramater_grid = ParameterGrid(parameters)

list(paramater_grid)
len(paramater_grid)#models
%%time
model = DBSCAN()
best_score = -1
for g in paramater_grid:
    model.set_params(**g)
    model.fit(drivers_features)
    ss = metrics.silhouette_score(drivers_features, model.labels_)
    print('Parameter: ', g, 'Score: ', ss)
    if ss> best_score:
        best_score = ss
        best_grid = g
best_grid
model.set_params(**best_grid)
model.fit(drivers_features)
len(model.labels_)# noise data will have label -1
n_clusters = len(set(model.labels_)) - (1 if -1 in model.labels_ else 0)
n_clusters
n_noise = list(model.labels_).count(-1)
n_noise
another_grid = {'eps':5., 'min_samples':5} #bad model params yield more noisy points

model.set_params(**another_grid)
model.fit(drivers_features)

n_clusters = len(set(model.labels_)) - (1 if -1 in model.labels_ else 0)
print('number of clusters: ', n_clusters)

n_noise = list(model.labels_).count(-1)
print('noisy points: ', n_noise)
another_grid = {'eps':5., 'min_samples':7} #bad model params yield more noisy points
# with higher min samples - fewer clusters are formed

model.set_params(**another_grid)
model.fit(drivers_features)

n_clusters = len(set(model.labels_)) - (1 if -1 in model.labels_ else 0)
print('number of clusters: ', n_clusters)

n_noise = list(model.labels_).count(-1)
print('noisy points: ', n_noise)<jupyter_output>number of clusters:  3
noisy points:  157<jupyter_text>mean shift HPT<jupyter_code>%%time
from sklearn.cluster import estimate_bandwidth
estimate_bandwidth(drivers_features) #time taken is quadratic in the number of samples - computationally expensive
model = MeanShift(bandwidth=estimate_bandwidth(drivers_features)).fit(drivers_features)
metrics.silhouette_score(drivers_features, model.labels_)<jupyter_output><empty_output><jupyter_text>clustering with image data* vs tensorflow which works with a list of images as a single 4-D tensor<jupyter_code># using mnist handwritten dataset
mnist_data = pd.read_csv('datasets/train.csv')
mnist_data.head()
#28x28 = 784 pixels (single channel)
mnist_data.shape
sorted(mnist_data['label'].unique())
mnist_features = mnist_data.drop('label', axis=1)
mnist_features.head()
mnist_labels = mnist_data['label']
mnist_labels.head()
#helper function
def display_image(index):
    print("Digit: ", mnist_labels[index])
    plt.imshow(mnist_features.loc[index].values.reshape(28,28), cmap='Greys')
display_image(5)
display_image(500)
%%time
kmeans_model = KMeans(n_clusters=10, max_iter=1000).fit(mnist_features)
kmeans_centroids = kmeans_model.cluster_centers_
kmeans_centroids
kmeans_centroids.shape
fig, ax = plt.subplots(figsize=(12,8))
for centroid in range(len(kmeans_centroids)):
    plt.subplot(2,5,centroid+1)
    plt.imshow(kmeans_centroids[centroid].reshape(28,28), cmap='Greys')
np.unique(kmeans_model.labels_)
mnist_test = mnist_data.sample(10, replace=False)

mnist_test_features = mnist_test.drop('label', axis=1)
mnist_test_labels = mnist_test['label']
mnist_test_labels
mnist_test_labels = np.array(mnist_test_labels)
mnist_test_labels
pred_clusters = kmeans_model.predict(mnist_test_features)
pred_results = pd.DataFrame({'actual_digit': mnist_test_labels, 'pred_cluster': pred_clusters})
pred_results.head(10)
#need to remap pred_clusters to the digits above from imshow
%%time
minibatch_kmeans_model = MiniBatchKMeans(n_clusters=10, max_iter=10000, batch_size=100).fit(mnist_features)
minibatch_kmeans_centroids = mini_batch_kmeans_model.cluster_centers_
minibatch_kmeans_centroids.shape
fig, ax = plt.subplots(figsize=(12,8))
for centroid in range(len(minibatch_kmeans_centroids)):
    plt.subplot(2,5,centroid+1)
    plt.imshow(minibatch_kmeans_centroids[centroid].reshape(28,28), cmap='Greys')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10331678538024318, 'fraction_numerical': 0.04122598748111375, 'mean_word_length': 4.87220109843684, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '263498', 'license': 'MIT', 'n_tokens_mistral': 9748, 'n_tokens_neox': 8723, 'n_words': 2428, 'path': 'scikit-learn_ps/ClusteringModels1.ipynb', 'repo_name': 'rsunderscore/learning', 'size': '27803'}