<jupyter_start><jupyter_text>Customer Analysis and SegmentationIn this notebook we perform clustering based on some anonymous customer data. We use a K-means clustering algorithm, and subsequently analyse the results. Here, we're analysing customer spending on an unidentified set of products, using the background of our customers to cluster by. EDA We'll start by importing the usual packages, as well as our cluster algo from Scikit-Learn. This section focuses on gaining an understanding of our data.<jupyter_code>import pandas as pd
import numpy as np 
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = [12, 7]

import seaborn as sns 

from sklearn.preprocessing import RobustScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

import datetime
import warnings
warnings.filterwarnings("ignore")
import os
data = pd.read_csv('input/cust_data.csv')
print(data.shape)
data.head()<jupyter_output>(60856, 5)<jupyter_text>There are a few categorical vars that we'll need to sort. Also we can drop the id feature right away.<jupyter_code>data.drop('id', axis=1, inplace=True)
data.info()
data.describe()<jupyter_output><empty_output><jupyter_text>Data PreproccessingHere we transform and scale our features and look for anomalies or outliers. These steps are necessary to a high performing algorithm, and should often be revisited even after the algorithm has been fitted. Let's plot our new processed data to look for any interesting first signs<jupyter_code>pd.plotting.scatter_matrix(data, alpha = 0.3, figsize = (20,16), diagonal = 'kde');<jupyter_output><empty_output><jupyter_text>Our K-Nearest Neighbours algorithm has to take in scaled data, or it won't perform particularly well. (Send me your thoughts if you disagree). Our data could be skewed by those zero values in features such as credit_score and credit_amount. We can't normalise by logarithm (as the zero values will return NaNs) but we also don't want to get rid of these data because they might be indicitive of some specific clusters. We will Robust Scaling instead (transform the feature vector by subtracting the median and then dividing by the interquartile range). This method is chosen because (as the name would suggest) it is more robust to outliers, due to its use of the median.<jupyter_code>robust = RobustScaler()
df_scaled = pd.DataFrame(robust.fit_transform(data), columns=data.columns)
df_scaled.head()
df_scaled.describe()<jupyter_output><empty_output><jupyter_text>Principal Component AnalysisHere we discover what dimensions about the data maximize the variance of features involved. PCA will also tell us the explained variance ratio of each dimension. This is how much variance within the data is explained by that dimension alone.<jupyter_code># Apply PCA by fitting the good data with only two dimensions
pca = PCA(n_components = 2, random_state = 0)
pca = pca.fit(df_scaled)

# Transform the good data using the PCA fit above
df_reduced = pca.transform(df_scaled)

# Create a DataFrame for the reduced data
df_reduced = pd.DataFrame(df_reduced, columns = ['Dimension 1', 'Dimension 2'])
df_reduced.head()
sns.scatterplot(df_reduced['Dimension 1'], df_reduced['Dimension 2'], alpha=0.4)
plt.show()<jupyter_output><empty_output><jupyter_text>Modelling<jupyter_code># Use silhouette score to find optimal number of clusters to segment the data
num_clusters = np.arange(2,10)
kmeans_results = {}
for size in num_clusters:
    kmeans = KMeans(n_clusters = size).fit(df_reduced)
    preds = kmeans.predict(df_reduced)
    kmeans_results[size] = silhouette_score(df_reduced, preds, sample_size=10000, metric='euclidean')
best_size = max(kmeans_results, key=kmeans_results.get)

# Build a model with the optimal number of clusters
optimized_kmeans = KMeans(n_clusters = best_size).fit(df_reduced)
kmeans_preds = optimized_kmeans.predict(df_reduced)
# Reports the centers of each cluster
kmeans_centers = optimized_kmeans.cluster_centers_
# Report the best silhouette score 
kmeans_score = silhouette_score(df_reduced, preds, sample_size=10000, metric='euclidean')

kmeans_assignment = optimized_kmeans.predict(df_reduced)
kmeans_results<jupyter_output><empty_output><jupyter_text>According to our clustering algorithm, we should segment our customers into 4 segments<jupyter_code>df_pred = pd.concat([df_reduced, pd.DataFrame(kmeans_preds, columns=['pred'])], axis=1)
df_pred.head()
sns.scatterplot(x=df_pred['Dimension 1'], y=df_pred['Dimension 2'], alpha=0.6, hue=df_pred['pred'])
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07843977711101587, 'fraction_numerical': 0.00964423489069867, 'mean_word_length': 6.443381180223286, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '414789', 'license': 'Apache-2.0', 'n_tokens_mistral': 1489, 'n_tokens_neox': 1365, 'n_words': 502, 'path': 'analysis-and-customer-segmentation.ipynb', 'repo_name': 'TheNerdyCat/customer-segmentation', 'size': '4666'}