<jupyter_start><jupyter_text>6.	Run a hierarchical clustering method and see how your result differs from above one.https://www.analyticsvidhya.com/blog/2019/05/beginners-guide-hierarchical-clustering/https://www.kaggle.com/binovi/wholesale-customers-data-set The aim of this problem is to segment the clients of a wholesale distributor based on their annual spending on diverse product categories, like milk, grocery, region, etc.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load the data and look at the first few rows:<jupyter_code>data = pd.read_csv('Wholesale customers data.csv')
data.head()<jupyter_output><empty_output><jupyter_text>There are multiple product categories – Fresh, Milk, Grocery, etc. The values represent the number of units purchased by each client for each product. Our aim is to make clusters from this data that can segment similar clients together. We will, of course, use Hierarchical Clustering for this problem.But before applying Hierarchical Clustering, we have to normalize the data so that the scale of each variable is the same. Why is this important? Well, if the scale of the variables is not the same, the model might become biased towards the variables with a higher magnitude like Fresh or Milk et’s first normalize the data and bring all the variables to the same scale:<jupyter_code>from sklearn.preprocessing import normalize
data_scaled = normalize(data)
data_scaled = pd.DataFrame(data_scaled, columns=data.columns)
data_scaled.head()<jupyter_output><empty_output><jupyter_text>we can see that the scale of all the variables is almost similar. Now, we are good to go. Let’s first draw the dendrogram to help us decide the number of clusters for this particular problem:<jupyter_code>import scipy.cluster.hierarchy as shc
plt.figure(figsize=(10, 7))  
plt.title("Dendrograms")  
dend = shc.dendrogram(shc.linkage(data_scaled, method='ward'))<jupyter_output><empty_output><jupyter_text>The x-axis contains the samples and y-axis represents the distance between these samples. The vertical line with maximum distance is the blue line and hence we can decide a threshold of 6 and cut the dendrogram<jupyter_code>plt.figure(figsize=(10, 7))  
plt.title("Dendrograms")  
dend = shc.dendrogram(shc.linkage(data_scaled, method='ward'))
plt.axhline(y=6, color='r', linestyle='--')<jupyter_output><empty_output><jupyter_text>We have two clusters as this line cuts the dendrogram at two points. Let’s now apply hierarchical clustering for 2 clusters<jupyter_code>from sklearn.cluster import AgglomerativeClustering
cluster = AgglomerativeClustering(n_clusters=2, affinity='euclidean', linkage='ward')  
cluster.fit_predict(data_scaled)<jupyter_output><empty_output><jupyter_text>We can see the values of 0s and 1s in the output since we defined 2 clusters. 0 represents the points that belong to the first cluster and 1 represents points in the second cluster. Let’s now visualize the two clusters:<jupyter_code>plt.figure(figsize=(10, 7))  
plt.scatter(data_scaled['Milk'], data_scaled['Grocery'], c=cluster.labels_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07723704866562009, 'fraction_numerical': 0.007849293563579277, 'mean_word_length': 6.392111368909513, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '43196', 'license': 'MIT', 'n_tokens_mistral': 993, 'n_tokens_neox': 930, 'n_words': 350, 'path': '6_hierarchical_clustering.ipynb', 'repo_name': 'rucha04/example22', 'size': '3185'}