<jupyter_start><jupyter_text>_Data pre-processing for k-mean clustering_<jupyter_code>import pandas as pd
from datetime import timedelta
import os
import seaborn as sns
import numpy as np
from matplotlib import pyplot as plt
print(os.getcwd())<jupyter_output>/home/hans/python_codes/DataCamp/data_pre-processing_fo_k-mean_clustering<jupyter_text>Advantiges of k-mean clustering* one of the most popular unsupervised learning method* Simple and fast* Worsk well (with certain assumptions about the data)  Key k-mean assumptions* Symmetric distribution of variables (not skewed)* Variables with same average values* Variables with same variance  Variables on the same scale* K-means assumes equal mean* And equal variance* It's not the case with RFM Data  Dataset<jupyter_code>datamart_rfm = pd.read_excel('../data/datamart.xlsx')
datamart_rfm.describe()<jupyter_output><empty_output><jupyter_text>Identifying skewness* Visual analysis of the distribution* If it has a tail - it's skewed  Exploring distribution of Rencency<jupyter_code>sns.distplot(datamart_rfm['Recency'])
plt.show()<jupyter_output><empty_output><jupyter_text>Exploring distribution of Frequency<jupyter_code>sns.distplot(datamart_rfm['Frequency'])
plt.show()<jupyter_output><empty_output><jupyter_text>Data transfomations to manage skewness * Logarithmic transformations (positive values only)<jupyter_code>frequency_log = np.log(datamart_rfm['Frequency'])
sns.distplot(frequency_log)
plt.show()<jupyter_output><empty_output><jupyter_text>Dealing with negative values- Adding a constant before log transformation- Cube root transformation  Centering and scaling variables  Identifying an issue- Analyze key statistics of the dataset- Compare mean and standard deviation<jupyter_code>datamart_rfm.describe().round(1)<jupyter_output><empty_output><jupyter_text>Centering variables with different means- K-means works well on variables with the same mean- Centering variables is done by subtracting average value from each observation<jupyter_code>datamart_centered = datamart_rfm - datamart_rfm.mean()
datamart_centered.describe().round(2)<jupyter_output><empty_output><jupyter_text>Scaling variables with different variance- K-means works better on variables with the same variance / standard deviation- Scaling variablkes is done by dividing them by standard deviation of each<jupyter_code>datamart_scaled = datamart_rfm / datamart_rfm.std()
datamart_scaled.describe().round(2)<jupyter_output><empty_output><jupyter_text>Combining centering and scaling- Subtract mean and divide by standard deviation manually- Or use a scaler from 'scikit-learn' library (returns numpy.ndarray object)<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(datamart_rfm)
datamart_normalized = scaler.transform(datamart_rfm)
print ('mean: ', datamart_normalized.mean(axis=0).round(2))
print ('std: ', datamart_normalized.std(axis=0).round(2))<jupyter_output>mean:  [-0. -0. -0.  0.  0.  0.  0. -0.  0.]
std:  [1. 1. 1. 1. 1. 1. 1. 1. 1.]<jupyter_text>_Sequence of structuring pre-processing steps_  Why the sequence matters ?-  Log transformation only works with positive values-  Normalization forces data to have negative values and **log** wil not work  Sequence1. Unskew the data - log transformation2. Standardize the same average values3. Scale to the same standard diviation4. Store as separate array to be used for clustering  Coding the sequence- Unskew the data with log transformation<jupyter_code>import numpy as np
#datamart_rfm = pd.read_excel('../data/datamart.xlsx')
datamart_log = np.log(datamart_rfm[['Frequency','Recency']])<jupyter_output><empty_output><jupyter_text>- Normalize the variables with **StandardScaler**<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(datamart_log)<jupyter_output><empty_output><jupyter_text>- Store it seperately for clustering<jupyter_code>datamart_normalized = scaler.transform(datamart_log)
np.save('../data/datamart_normalized',datamart_normalized)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0877666094631037, 'fraction_numerical': 0.007109585682765384, 'mean_word_length': 7.192771084337349, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '317552', 'license': 'MIT', 'n_tokens_mistral': 1347, 'n_tokens_neox': 1279, 'n_words': 353, 'path': 'data_pre-processing_for_k-mean_clustering/data pre-processing for k-mean clustering.ipynb', 'repo_name': 'hanshof/ml_tutorials', 'size': '4079'}