<jupyter_start><jupyter_text>WIP  Customer SegmentationK-Means Clustering---Through customer segmentation (for example RFM score), we hope to better identify how different subsections of our customer base is purchasing. Doing this allows us to serve our customers better by hopefully providing insight into the products that they want; we can also inform them of promotions and products which they might be interested in, which will hopefully be more profitable for us -- a win/win. We can perform this segmentation in multiple ways (for example the component columns of RFM score), and provide different insights -- with multiple segmentations, we can compare within groups based on other factors.2. By using the demographic information from hh_demographic, can we train a model to 'predict' the demographic information for the rest of our households? xgboost<jupyter_code>#### K-MEANS NOTES
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['figure.figsize']=(16,6) 
plt.style.use('seaborn')

import my_funcs
merged = my_funcs.load_merged()
hh_agg = my_funcs.load_hh_agg(merged)
demo = my_funcs.load_demo()
# inverting these columns for correlation visualization?
hh_agg['total_loyalty'] = hh_agg['total_loyalty'] * -1
hh_agg['total_coupon'] = hh_agg['total_coupon'] * -1

plt.figure(figsize=(12,6))
sns.heatmap(hh_agg.corr(), vmin=-1, vmax=1, cmap='coolwarm')

# inverting these columns
hh_agg['total_loyalty'] = hh_agg['total_loyalty'] * -1
hh_agg['total_coupon'] = hh_agg['total_coupon'] * -1
np.corrcoef(hh_agg['total_spend'], hh_agg['total_loyalty'])
# I wanted this correlation to appear as a positive one in the heatmap.<jupyter_output><empty_output><jupyter_text>The total_spend category and the total_loyalty category are highly negatively correlated -- indicating that customers with the highest loyalty discounts are also the ones with the highest spend values in our dataset.---  Unsupervised learning; K-Means ClusteringK-Means clustering involves calculating the euclidean distance in a (multi-dimensional) feature space. It's a computationally intensive process where:    - centroids are placed randomly in the (X-dimensional, ie 3-D) space.    - the centroids try to move towards places of lower 'inertia', where they are in the middle of more data pointsThe parameters for the sklearn KMeans class are listed below:    'n_clusters': 8,    'init': 'k-means++',    'max_iter': 300,    'tol': 0.0001,    'precompute_distances': 'deprecated',    'n_init': 10,    'verbose': 0,    'random_state': None,    'copy_x': True,    'n_jobs': 'deprecated',    'algorithm': 'auto'}notice the `n_clusters`, `n_init` and `max_iter` methods; `n_clusters` refers to the number of centroids (and therefore cluster labels) which will be produced.from the docs...`n_init` : int, default=10    Number of time the k-means algorithm will be run with different    centroid seeds. The final results will be the best output of    n_init consecutive runs in terms of inertia.`max_iter` : int, default=300    Maximum number of iterations of the k-means algorithm for a    single run.  Questions Before Preprocessing I know this is `unsupervised learning`, but we need to think about two things here:    - what proximity between data-points in a multi-dimensional euclidean space really means    - what columns of data we're feeding our model; don't we already have 'cluster' labels?    To this point, I think an appropriate course of action would be to run the algorithm on several subsets of information that we've accumulated and pipe-lined to this point. The first thing I'd be interested to know about would be the similarity between the products that customers are purchasing.  - the first labels could be based on our derived section labels; the total spend in each category. - a second thought is to take the normalized per-basket category spend; one column for each category. - a third option would be to take some sort of timeseries or quarterly spend, using total or per-basket.<jupyter_code>hh_agg.columns<jupyter_output><empty_output><jupyter_text>Creating subsections of hh_agg<jupyter_code>hh_summary = hh_agg[['household_key', 'total_spend', 'total_loyalty', 'total_coupon',
       'total_quantity', 'num_baskets', 'unique_products', 'first_purchase',
       'last_purchase', 'avg_basket_spend', 'avg_item_cost','days_between_purchases']].copy()
hh_summary.set_index('household_key', inplace=True)
hh_products = hh_agg[['household_key', 'alcohol',
       'beverages', 'concessions', 'dairy', 'drug', 'garden', 'grain_goods',
       'grocery', 'home_family', 'junk_food', 'kitchen', 'meat', 'misc',
       'produce', 'seasonal']].copy()
hh_products.set_index('household_key', inplace=True)
# sns.heatmap(hh_products)
# this gives me an idea for timeseries analysis...<jupyter_output><empty_output><jupyter_text>Scaling the dataScaling our data appropriately does a lot to improve the results and performance of this model. By normalizing our numeric columns between either -1 and 1; or 0 and 1; we give our model a better chance to interpret meaningful variance by eliminating the impact of 'scale' between columns with different units of measure.  Scaling `hh_products`<jupyter_code>prod_ss = StandardScaler().fit_transform(hh_products)<jupyter_output><empty_output><jupyter_text>The fit_transform method of the StandardScaler object returns a **2-D numpy array**. Since we didn't alter hh_products in place (beyond the index), we can simply call its index and columns to recreate a DataFrame;<jupyter_code># pd.DataFrame(prod_ss, index=hh_products.index, columns=hh_products.columns)<jupyter_output><empty_output><jupyter_text>Optimizing CentroidsI love this job. Where else do you get to say stuff like that?<jupyter_code>### Iterating to find optimal KMeans value
def plot_k_means_test(df, n_clusters=range(1,55)):
    inertias = []
    smallest = []
    for i in n_clusters:
        # instantiate

        KMM = KMeans(n_clusters=i)

        # fit
        KMM.fit(df)

        # evaluate
        inertias.append(KMM.inertia_)

        # note the smallest 'leaf'
        smallest.append(pd.Series(KMM.labels_).value_counts().min())
        
    inertias.insert(0,0)
    smallest.insert(0,0)

    # interpreting inertia and smallest_leaf_size for Kmeans Model
    plt.subplots(1,2, figsize=(16,6))

    plt.subplot(2,1,1)
    plt.title('inertias of K-means clusters')
    plt.plot(inertias)

    plt.xticks(n_clusters)
    plt.axvline(5, c='r', label='5')
    plt.legend()

    plt.subplot(2,1,2)
    plt.title('smallest leaf')
    plt.plot(smallest)
    plt.axvline(smallest[0], c='r')

    plt.xticks(n_clusters)
    plt.show();
plot_k_means_test(hh_products)<jupyter_output><empty_output><jupyter_text>Hard to tell on the inertia plot, but the smallest leaf graph tells us what we want to know. When we have a cluster with 0 members, we know we've gone too far. 5 labels would give us symmetry with our RFM scores; as well as the range for 'number of household members' in the demographic data. Call the function again on the other subset --  Scaling and Plotting `hh_summary`<jupyter_code>summary_ss = StandardScaler().fit_transform(hh_summary)
plot_k_means_test(summary_ss)<jupyter_output><empty_output><jupyter_text>Running final model and comparing with RFM score;<jupyter_code>mymodel = KMeans(n_clusters=5, n_init=25, max_iter=500)

end_ss = mymodel.fit(hh_agg)
results = end_ss.labels_
len(results)
hh_agg['KM5_Labels'] = results
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
plt.subplots(3,5, figsize=(16,6), sharex='row')
for idx, label in enumerate(range(5)):
    df = hh_agg[hh_agg['KM5_Labels']==label]
    plt.subplot(3,5,idx+1)
    plt.title(f'{label}')
    sns.boxplot(df['RFM'])
    
    if idx+6>10:
        plt.title(f'avg. basket spend')
    plt.subplot(3, 5, idx+6)
    sns.boxplot(df['avg_basket_spend'])
    if idx+11>15:
        plt.title('total spend')
    plt.subplot(3, 5, idx+11)
    sns.boxplot(df['total_spend'])

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the cluster labels seem to have distinct `total_spend` distributions. KM5_Labels 1 and 2 appear to be the most similar across these three categories, although groups 4 and 0 are also very similar.<jupyter_code>hh_agg['KM5_Labels'].value_counts()<jupyter_output><empty_output><jupyter_text>Let's look at the difference between group 0 and group 4<jupyter_code>hh_agg_4 = hh_agg[hh_agg['KM5_Labels'] == 4]
hh_agg_0 = hh_agg[hh_agg['KM5_Labels'] == 0]

pd.DataFrame([hh_agg_0.mean(), hh_agg_4.mean()])<jupyter_output><empty_output><jupyter_text>We can see that the groups diverge along `total_spend` mean, as well as `num_baskets`, `unique_products`, `meat`, `misc`, `produce` spending; as well as the `F` and `M` RFM columns. And for labels 2 and 3:<jupyter_code>hh_agg_3 = hh_agg[hh_agg['KM5_Labels'] == 3]
hh_agg_2 = hh_agg[hh_agg['KM5_Labels'] == 2]

pd.DataFrame([hh_agg_3.mean(), hh_agg_2.mean()])<jupyter_output><empty_output><jupyter_text>These groups are again distinguishable along the `total_spend` mean, among other spending categories. It might be worth considering whether 'average section sales per basket' might be a better column. We note that KM5 Label 3 has a mean `M` score of 5 -- a perfect score -- with an `RFM` above 10. The KM5 Label 2 group has an `RFM` score approaching 10 -- still very good. Finally we should look at KM5_Label == 1<jupyter_code>pd.DataFrame(hh_agg[hh_agg['KM5_Labels']==1].mean()).T<jupyter_output><empty_output><jupyter_text>Very low `F` score; but again we have a `M` mean of 5 -- among 170 households! This is suspicious. One thing that stands out to me about these 5 groups is the variance along the `days_between_purchases` mean value for each group; they have values close to: [3, 4, 5.75, 9, 31] across the various labels (these are not in order). To me this indicates that the model is recognizing the inherent value of a frequent purchaser -- it drives up the total spending differences!<jupyter_code># from itertools import cycle

# # setup plot details
# colors = cycle(["navy", "turquoise", "darkorange", "cornflowerblue", "teal"])
hh_agg.sort_values('KM5_Labels', ascending=False)['KM5_Labels'].plot(figsize=(16,6), kind='barh')
newresults = dict()
for col in ['R', 'F', 'M', 'RFM']:
    newresults[col] = dict()
    for label in hh_agg[col].unique():
        data = hh_agg[hh_agg[col]==label]
        newresults[col][label] = int(data[col].value_counts())
        
for col in newresults:
    print(newresults[col].items())<jupyter_output>dict_items([(4, 638), (1, 501), (3, 518), (2, 537), (5, 291)])
dict_items([(3, 499), (4, 495), (1, 497), (2, 497), (5, 497)])
dict_items([(4, 497), (3, 497), (2, 497), (5, 497), (1, 497)])
dict_items([(11, 341), (8, 435), (7, 440), (9, 515), (10, 554), (6, 119), (12, 54), (5, 17), (13, 8), (3, 1), (4, 1)])<jupyter_text>TODO:- which categories my best customers purchase in -- how sales are being affected over time or by campaigns- Compare demographic_hh categories by sales- Generate labels based on cleaned transactions data    - sum of sales by section, divided by total spend? number of transactions?- Compare RFM score with Cluster Labels CHECKING CORRELATION BETWEEN SECTION SALES OVER TIME - PACF?- PCT CHANGE?- REMOVE TREND/SEASONALITY THEN COMPARE PCT.CHANGE?<jupyter_code># import seaborn as sns

# mycontainer = dict()
# for dept in merged['Section Label'].unique():
#     mycontainer[dept] = merged[merged['Section Label']==dept].groupby('WEEK_NO')['SALES_VALUE'].sum()

# a = pd.DataFrame(mycontainer)
# a.pct_change().drop(1)

# mask = np.triu(a.pct_change().drop(1).corr())
# sns.heatmap(a.pct_change().drop(1).corr(), mask=mask, annot=True, vmin=-1, vmax=1, cmap='coolwarm')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10992937266306606, 'fraction_numerical': 0.020938928126298297, 'mean_word_length': 5.143950995405819, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '516399', 'license': 'MIT', 'n_tokens_mistral': 4109, 'n_tokens_neox': 3831, 'n_words': 1291, 'path': '2.3 -- KMeans Clustering using sklearn.ipynb', 'repo_name': 'np1919/DTCJ', 'size': '12035'}