<jupyter_start><jupyter_text>Read data, build model, and plot clusters Fill nulls:1. numerical - mean2. categorical - median Cleaning data...The critera for cleaning is:1. remove columns have more than 20% nulls2. remove row have more than 10% nulls3. remove kba columns -- bias the dataset with minimum information4. change the OST_WEST_KZ to dummy data5. extract decade data from PRAEGENDE_JUGENDJAHRE6. remove columns with too much dummy values (CAMEO_DEU_2015)7. reengineer the mix datatype rows<jupyter_code>azdias_clean = pd.read_csv('azdias_clean.csv')
customers_clean = pd.read_csv('customers_clean.csv')
azdias_clean.drop('Unnamed: 0', axis=1, inplace=True)
customers_clean.drop('Unnamed: 0', axis=1, inplace=True)
# Build model
pca_n = 60
n_clusters = 8

model = build_model(pca_n, n_clusters)

print('Fitting model...')
model.fit(azdias_clean)

print('Saving model...')
f = open('model' + str(n_clusters) + '.pkl', 'wb')
pickle.dump(model, f)
# predict the model
az_prediction = model.predict(azdias_clean)
cus_prediction = model.predict(customers_clean)
# plot the clusters
plot_clusters(az_prediction, cus_prediction)<jupyter_output><empty_output><jupyter_text>We can see that in cluster **3 and 7**, the customers are **overrepresented**, and in cluster **4 and 5**, customers are **underrepresented**.  Check princeple components  Check the cluster center<jupyter_code>(model.named_steps['kmeans'].cluster_centers_[4,:]).argmax(),(model.named_steps['kmeans'].cluster_centers_[4,:]).argmin()
(model.named_steps['kmeans'].cluster_centers_[3,:]).argmax(),(model.named_steps['kmeans'].cluster_centers_[3,:]).argmin()
for i in range(1, 4):
    print('PCA', i)
    print_interesting_features(azdias_clean.columns.values, model.named_steps['pca'].components_, i)
    print()<jupyter_output>PCA 1
Highest:
	FINANZ_SPARER        0.244
	SEMIO_REL            0.212
	SEMIO_PFLICHT        0.211
Lowest:
	ALTERSKATEGORIE_GROB -0.222
	FINANZ_VORSORGER     -0.221
	FINANZ_MINIMALIST    -0.191

PCA 2
Highest:
	SEMIO_KAEM           0.199
	SEMIO_ERL            0.190
	ANREDE_KZ            0.172
Lowest:
	SEMIO_KULT           -0.203
	SEMIO_FAM            -0.185
	SEMIO_REL            -0.179

PCA 3
Highest:
	SEMIO_VERT           0.310
	SEMIO_SOZ            0.241
	SEMIO_FAM            0.212
Lowest:
	ANREDE_KZ            -0.330
	SEMIO_KAEM           -0.286
	SEMIO_DOM            -0.268<jupyter_text>We can see that in the underrepresented clusters, the most positive compnents are PC1, and the most negative components are pc2 and pc3. We noticed that:- PC1 is associated with finantial habbit and peronality (traditional minded etc.)- PC2 and PC3 is associated with personality traits, community/family size.This may indicate that finantial habbit (saving or not etc.) and personality can used to differentiate the customers.<jupyter_code>(model.named_steps['kmeans'].cluster_centers_[2,:]).argmax(),(model.named_steps['kmeans'].cluster_centers_[2,:]).argmin()
(model.named_steps['kmeans'].cluster_centers_[6,:]).argmax(),(model.named_steps['kmeans'].cluster_centers_[6,:]).argmin()
for i in [0, 6, 7]:
    print('PCA', i)
    print_interesting_features(azdias_clean.columns.values, model.named_steps['pca'].components_, i)
    print()<jupyter_output>PCA 0
Highest:
	D19_GESAMT_ANZ_24    0.198
	D19_GESAMT_ANZ_12    0.190
	D19_VERSAND_ANZ_24   0.182
Lowest:
	D19_GESAMT_DATUM     -0.194
	D19_VERSAND_DATUM    -0.186
	D19_GESAMT_ONLINE_DATUM -0.184

PCA 6
Highest:
	D19_BANKEN_ANZ_12    0.246
	D19_BANKEN_ANZ_24    0.236
	D19_TELKO_ANZ_12     0.216
Lowest:
	HEALTH_TYP           -0.287
	VERS_TYP             -0.261
	D19_BANKEN_DATUM     -0.211

PCA 7
Highest:
	HEALTH_TYP           0.373
	VERS_TYP             0.357
	NATIONALITAET_KZ     0.253
Lowest:
	D19_BANKEN_DATUM     -0.190
	D19_BANKEN_ONLINE_DATUM -0.187
	D19_VERSAND_ANZ_12   -0.172<jupyter_text>We noticed that:- PC0 is associated with transaction activities and online activities.- PC6 is associated with online activities and health condition.- PC7 is associated with health, insuarance, online transactions.This indicated that transaction activities, insuarance and health may used to differentiate customers.  Check original dataframe<jupyter_code>azdias_clean['preds'] = az_prediction
customers_clean['preds'] = cus_prediction
# select columns to analyze
cols = ['TITEL_KZ', 'ANZ_TITEL', 'ANZ_HH_TITEL', 'D19_VERSAND_ANZ_12', 'D19_VERSAND_ANZ_24', 'D19_GESAMT_ANZ_12', 
 'D19_GESAMT_ANZ_24', 'GREEN_AVANTGARDE', 'D19_BANKEN_ANZ_12', 'AGER_TYP', 'D19_TELKO_ANZ_12', 'D19_VERSI_ANZ_12',
'FINANZTYP', 'D19_VERSI_ANZ_24', 'GEBURTSJAHR', 'LP_FAMILIE_FEIN', 'D19_TELKO_ANZ_24', 'ALTER_HH','LP_FAMILIE_GROB',
'MOBI_REGIO', 'preds']
azdias_ana = azdias_clean.loc[:, cols]
customers_ana = customers_clean.loc[:, cols]
azdias_ana.groupby(['preds']).mean().T.loc[:,[3,4]].mean(axis=1)
customers_ana.groupby(['preds']).mean().T.loc[:,[2,6]].mean(axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11394971613949716, 'fraction_numerical': 0.057177615571776155, 'mean_word_length': 4.586636466591166, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '96754', 'license': 'CNRI-Python', 'n_tokens_mistral': 2078, 'n_tokens_neox': 1892, 'n_words': 399, 'path': 'segementation/Arvato Segementation Report.ipynb', 'repo_name': 'XiyueWang/arvato_segement_prediction', 'size': '4932'}