<jupyter_start><jupyter_text>Check for missing values:<jupyter_code>cc_general.isna().sum()<jupyter_output><empty_output><jupyter_text>Check for outliers:Using IQR, we can follow the below approach to find outliers:- Calculate the first and third quartile (Q1 and Q3).- Further, evaluate the interquartile range, IQR = Q3-Q1.- Estimate the lower bound, the lower bound = Q1*1.5- Estimate the upper bound, upper bound = Q3*1.5- The data points that lie outside of the lower and the upper bound are outliers.<jupyter_code>def outlier_percent(data):
    Q1 = data.quantile(0.25)
    Q3 = data.quantile(0.75)
    IQR = Q3 - Q1
    minimum = Q1 - (1.5 * IQR)
    maximum = Q3 + (1.5 * IQR)
    num_outliers =  np.sum((data < minimum) |(data > maximum))
    num_total = data.count()
    return (num_outliers/num_total)*100
non_categorical_data = cc_general.drop(['CUST_ID'], axis=1)
for column in non_categorical_data.columns:
    data = non_categorical_data[column]
    percent = str(round(outlier_percent(data), 2))
    print(f'Outliers in "{column}": {percent}%')<jupyter_output>Outliers in "BALANCE": 7.77%
Outliers in "BALANCE_FREQUENCY": 16.68%
Outliers in "PURCHASES": 9.03%
Outliers in "ONEOFF_PURCHASES": 11.32%
Outliers in "INSTALLMENTS_PURCHASES": 9.69%
Outliers in "CASH_ADVANCE": 11.51%
Outliers in "PURCHASES_FREQUENCY": 0.0%
Outliers in "ONEOFF_PURCHASES_FREQUENCY": 8.74%
Outliers in "PURCHASES_INSTALLMENTS_FREQUENCY": 0.0%
Outliers in "CASH_ADVANCE_FREQUENCY": 5.87%
Outliers in "CASH_ADVANCE_TRX": 8.98%
Outliers in "PURCHASES_TRX": 8.56%
Outliers in "CREDIT_LIMIT": 2.77%
Outliers in "PAYMENTS": 9.03%
Outliers in "MINIMUM_PAYMENTS": 9.74%
Outliers in "PRC_FULL_PAYMENT": 16.47%
Outliers in "TENURE": 15.26%<jupyter_text>Preprocessingin this part, I'm going to:1. remove the outliers2. impute missing data3. scale the data4. Reduce dimentions using PCA  Removing the outliersfirst, let's get rid of the noise. we're going to first set all outliers as `NaN`, so it will be taken care of in the next stage, where we impute the missing values.<jupyter_code>for column in non_categorical_data.columns:
    data = non_categorical_data[column]
    
    Q1 = data.quantile(0.25)
    Q3 = data.quantile(0.75)
    IQR = Q3 - Q1
    minimum = Q1 - (1.5 * IQR)
    maximum = Q3 + (1.5 * IQR)
 
    outliers = ((data < minimum) |(data > maximum))
    non_categorical_data[column].loc[outliers] = np.nan
    
non_categorical_data.isna().sum()<jupyter_output><empty_output><jupyter_text>Imputing the missing dataI use `KNN imputer`: Each sample’s missing values are imputed using the mean value from n_neighbors nearest neighbors found in the training set.<jupyter_code>imputer = KNNImputer()
imp_data = pd.DataFrame(imputer.fit_transform(non_categorical_data), columns=non_categorical_data.columns)
imp_data.isna().sum()<jupyter_output><empty_output><jupyter_text>Scale the DataI use `StandardScaler`<jupyter_code>std_imp_data = pd.DataFrame(StandardScaler().fit_transform(imp_data), columns=imp_data.columns)
std_imp_data.describe()<jupyter_output><empty_output><jupyter_text>Dimention Reduction using PCAK-means, DBSCAN and agglomerative clustering, all use the Euclidean distance, which starts to lose its meaning when the number of dimensions starts increasing. so, before using these methods, we have to reduce the number of dimensions. I'm going to use PCA, which is by far the most popular dimensionality reduction algorithm. ***⚠ If you are not familiar with PCA or need to learn more about it, I highly recommend you read [my Notes](https://cookie-baboon-7d6.notion.site/PCA-0879e6e386b747c6881506bfdf0e9c61) on this dimentionality reduction method, In which I cover almost anything you need to know about this algorithm.***here I set parameter `n_components` equals to 0.9, which means that the PCA will automatically produce enough PCs that will preserve 90% of variance in the dataset.<jupyter_code>pca = PCA(n_components=0.9, random_state=42)
pca.fit(std_imp_data)
PC_names = ['PC'+str(x) for x in range(1,len(pca.components_)+1)]
pca_data = pd.DataFrame(pca.transform(std_imp_data), columns=PC_names)
fig, ax = plt.subplots(figsize=(24, 16))
plt.imshow(pca.components_.T,
           cmap="Spectral",
           vmin=-1,
           vmax=1,
          )
plt.yticks(range(len(std_imp_data.columns)), std_imp_data.columns)
plt.xticks(range(len(pca_data.columns)), pca_data.columns)
plt.xlabel("Principal Component")
plt.ylabel("Contribution")
plt.title("Contribution of Features to Components")
plt.colorbar()<jupyter_output><empty_output><jupyter_text>Train the Modelnow that we have done the preprocessing step, we can perform K-means clustering on our data. first, we have to find good parameters for our model.  Find the `n_clusters` parameter using the elbow method<jupyter_code>model = KMeans(random_state=42)
distortion_visualizer = KElbowVisualizer(model, k=(2,10))

distortion_visualizer.fit(pca_data)       
distortion_visualizer.show()       
km_model = KMeans(distortion_visualizer.elbow_value_, random_state=42)
labels = km_model.fit_predict(pca_data)<jupyter_output><empty_output><jupyter_text>Evaluate the ModelLet's see how good/bad is our model.we start by caculating two metrics: - **The Davis-Bouldin Index** is the average similarity between each cluster and the closest cluster. Scores range from 0 and up. 0 indicates better clustering.- **The Silhouette Coefficient** is a value between -1 and 1. The higher the score, the better. 1 indicates tight clusters, and 0 means overlapping clusters.<jupyter_code>print(f'Davies-Bouldin index = {davies_bouldin_score(pca_data, labels)}')
print(f'Silhouette Score = {silhouette_score(pca_data, labels)}')<jupyter_output>Davies-Bouldin index = 1.553675096074755
Silhouette Score = 0.1989293328933083
{'fraction_non_alphanumeric': 0.10221760221760222, 'fraction_numerical': 0.03309078309078309, 'mean_word_length': 5.479236812570146, 'pattern_counts': {'":': 18, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '621860', 'license': 'MIT', 'n_tokens_mistral': 2177, 'n_tokens_neox': 1975, 'n_words': 560, 'path': 'Credit-Card-Classification-KMeans.ipynb', 'repo_name': 'farkoo/K-means-Cluster-Credit-Card', 'size': '5772'}