<jupyter_start><jupyter_text>Hierarchical Clustering - Mean ShiftK-Means is called "flat clustering" and required us to specify how many clusters we think existsMean shift estimates how many clusters exists and then finds the centers of those clustersMean Shift (MS) starts by placing a cluster center at the "1st" or "1st random" feature point.- It then  assigns a "bandwidth" or "radius" around each initial cluster center and determines how many features are within the "bandwidth" or "radius".    - This is done for every single "cluster center".After taking the mean of all of the data points in the cluster center + radius (bandwidth), it assigns this average as the new cluster center.    - MS again assigned a bandwidth around that new cluster center and iterates until convergence        - convergence means difference between old and new cluster center is below epsilonAs the cluster center moves around looking for convergence, it finds new feature points.  These new features points change the cluster center (maybe they should change the bandwidth too(?).The method allows such that if you start from any point any given cluster, MS will find the same set of feature points within the same cluster (cluster center, radius).  Once MS finds no new feature points and stops moving the cluster center, it has reached convergence.But there are known feature points outside of this cluster.  This must start again using the feature points that are outside the given cluster.- This creates a new cluster (i.e. MS determines the number of clusters && their centers)    This new cluster iterates to convergence again.  If there are more more feature points outside BOTH clusters, then MS will make a third cluster.If all feature points are accounted for by known clusters with centers and bandwidths, then MS has completed optimization---What if the clustering is not obvious?---MS can make sets of concentric bandwidths around the same clustering center, but with weights assigned to them.Such that the set of feature sets has 1 given center, but the points furthest away from that center are weighted less than the feature points that are closest to the center.- for a 'cluster' with 3 radii, the weights could be (3,2,1) for (nearest, middlest, furthest), respectively.- could also penalize in some sort of squared error fashion too ---Simple Example of Hierarchical Clustering with Mean Shift---<jupyter_code>%matplotlib inline
import numpy as np
from sklearn.cluster import MeanShift
from sklearn.datasets.samples_generator import make_blobs

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import style
style.use('fivethirtyeight')

nsamples= 100
centers = [[1,1,1], [5,5,5], [3,10,10]]
X, _    = make_blobs(n_samples=nsamples, centers=centers, cluster_std=1)

ms  = MeanShift()
ms.fit(X)

labels          = ms.labels_
cluster_centers = ms.cluster_centers_

print(cluster_centers)

n_clusters_ = len(np.unique(labels))

print("Number of estimated clusters:", n_clusters_)

colors = 10*plt.rcParams['axes.color_cycle']

fig = plt.figure()
ax  = fig.add_subplot(111, projection='3d')

for i in range(len(X)):
    ax.scatter(X[i][0], X[i][1], X[i][2], c=colors[labels[i]], marker='o')

ax.scatter(cluster_centers[:,0], cluster_centers[:,1], cluster_centers[:,2], 
               marker='x', color='k', s=150, lw=5, zorder=10)
print(n_clusters_)
print(centers)
print(cluster_centers)<jupyter_output><empty_output><jupyter_text>---Titanic Data Set---We assumed that the titanic data set would split into 2 groups because either survived or died; and we assumed that the more wealthy a passenger was, the more likely they were to have survivedUsing MeanShift instead of K-Means lets us remove that assumption and ask not only "where" are the cluster centers, but how many are there.<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd

from sklearn import preprocessing, cross_validation
from sklearn.cluster import MeanShift

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import style
style.use('fivethirtyeight')
'''
Pclass Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)
survival Survival (0 = No; 1 = Yes)
name Name
sex Sex
age Age
sibsp Number of Siblings/Spouses Aboard
parch Number of Parents/Children Aboard
ticket Ticket Number
fare Passenger Fare (British pound)
cabin Cabin
embarked Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)
boat Lifeboat
body Body Identification Number
home.dest Home/Destination
'''


# https://pythonprogramming.net/static/downloads/machine-learning-data/titanic.xls
df = pd.read_excel('titanic.xls')
original_df = pd.DataFrame.copy(df)
df.drop(['body','name'], 1, inplace=True)
df.fillna(0,inplace=True)
def handle_non_numerical_data(df):
    
    # handling non-numerical data: must convert.
    columns = df.columns.values

    for column in columns:
        text_digit_vals = {}
        def convert_to_int(val):
            return text_digit_vals[val]

        #print(column,df[column].dtype)
        if df[column].dtype != np.int64 and df[column].dtype != np.float64:
            
            column_contents = df[column].values.tolist()
            #finding just the uniques
            unique_elements = set(column_contents)
            # great, found them. 
            x = 0
            for unique in unique_elements:
                if unique not in text_digit_vals:
                    # creating dict that contains new
                    # id per unique string
                    text_digit_vals[unique] = x
                    x+=1
            # now we map the new "id" vlaue
            # to replace the string. 
            df[column] = list(map(convert_to_int,df[column]))

    return df
df = handle_non_numerical_data(df)
df.drop(['ticket','home.dest'], 1, inplace=True)
X = np.array(df.drop(['survived'], 1).astype(float))
X = preprocessing.scale(X)
y = np.array(df['survived'])
ms = MeanShift()
ms.fit(X)
labels          = ms.labels_
cluster_centers = ms.cluster_centers_
n_clusters      = len(cluster_centers)
n_clusters
original_df['cluster_group'] = np.nan

for i,label in enumerate(labels):
    original_df['cluster_group'].iloc[i] = label

n_clusters_ = len(np.unique(labels))
n_clusters_
survival_rates = {}
for clusterNow in range(n_clusters_):
    temp_df = original_df[(original_df['cluster_group'] == float(clusterNow))]
    
    # Define Survival Clusters
    survival_cluster = temp_df[(temp_df['survived'] == 1)]
    survival_rate    = len(survival_cluster) / len(temp_df)
    survival_rates[clusterNow] = survival_rate

survival_rates
for clusterNow in range(n_clusters_):
    # print(clusterNow, original_df[original_df['cluster_group'] == clusterNow]['pclass'].drop_duplicates().values, survival_rates[clusterNow]*100, end='\n\n')
    print('Cluster: ', clusterNow)
    print(clusterNow, original_df.describe(), end='\n\n')
colors = 10*plt.rcParams['axes.color_cycle']

fig = plt.figure()
ax  = fig.add_subplot(111, projection='3d')

for i in range(len(X)):
    ax.scatter(X[i][0], X[i][1], X[i][6], c=colors[labels[i]], marker='o')

ax.scatter(cluster_centers[:,0], cluster_centers[:,1], cluster_centers[:,3], 
               marker='x', color='k', s=150, lw=5, zorder=10)
for k in np.unique(labels):
    print(np.where(labels == k)[0].sum())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09567105441715294, 'fraction_numerical': 0.011670511602659792, 'mean_word_length': 4.16468114926419, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '629821', 'license': 'MIT', 'n_tokens_mistral': 2261, 'n_tokens_neox': 2169, 'n_words': 779, 'path': 'Clustering/Hierarchical Clusters - Mean Shift Algorithm.ipynb', 'repo_name': 'exowanderer/PythonProgramming_PracticalMachineLearning', 'size': '7369'}