<jupyter_start><jupyter_text>Customer Segmentation Workflow 1- Data Exploration and Feature Engineering ** These notebooks use an anonymised excerpt of a retailer dataset. The goals is to segment customers by both aggregate sales patterns and specific items purchased and to provide more tailored services and improve marketing to customers based on segments.**This is part 1 of a 2 part series presenting a customer segmentation workcycle. The dataset is a log of item purchase transactions with an associated customer id.The goal is to build a workflow for performing customer segmentation through clustering.<jupyter_code># NumPy for numerical computing
import numpy as np

# Pandas for DataFrames
import pandas as pd

# Matplotlib for visualization
from matplotlib import pyplot as plt

# display plots in the notebook
%matplotlib inline 

# Seaborn for easier visualization
import seaborn as sns

def display_all(df):
    with pd.option_context("display.max_rows", 1000, "display.max_columns", 1000): 
        display(df)
df = pd.read_csv('tx_log.csv')<jupyter_output><empty_output><jupyter_text>This dataset is clean and requires very little pre-processing. The dataset is small containing c.35,000 individual transactions with 8 features per transaction.<jupyter_code>df.shape
# First 10 rows of data
display_all(df.T)
df.dtypes
# transactions by country
plt.figure(figsize=(12,13))
# Bar plot by country
sns.countplot(y='Country', data=df)
# Missing data by feature
display_all(df.isnull().sum()/df.shape[0]*100)<jupyter_output><empty_output><jupyter_text>The intention is to cluster by customer segments and therefore we can do nothing with 4% of rows that are missing customer ids.<jupyter_code>df = df.dropna(subset=['CustomerID'])<jupyter_output><empty_output><jupyter_text>Equality operations between float values are sometimes unpredictable therefore the customerid is converted to an int type.<jupyter_code>df['CustomerID'] = df.CustomerID.astype(int)
df["CustomerID"].nunique()
#Average number of transactions per customer
df["CustomerID"].value_counts().mean()
plt.figure(figsize=(12,13))
sns.boxplot(x=df["CustomerID"].value_counts()).set_title("Number of purchses per customer")<jupyter_output><empty_output><jupyter_text>**Create a sales feature to capture the value of each transaction**<jupyter_code># Create 'Sales' interaction feature
df['Sales'] = df.Quantity * df.UnitPrice<jupyter_output><empty_output><jupyter_text>** Feature engineering and data aggregation**The data is still at the individual transaction level.It will have to be rolled up into a number of customer-level and invoice level features to allow customer-level and item-level segmentation.**First sales data is aggregated by customer and by invoice.<jupyter_code>def invoice_level_agg(x):
    names = {"Cart_Value": x["Sales"].sum()}
    return pd.Series(names, index=["Cart_Value"])
cart_data =  df.groupby(["CustomerID", "InvoiceNo"]).apply(invoice_level_agg)
cart_data.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>**The following customer-level features will be engineered:*** 'total_transactions' - the total number of unique transactions for each customer.* 'total_products'- the total number of products purchased by each customer * 'total_unique_products' - the number of unique products purchased by each customer.* 'total_sales' - the total sales for each customer.* 'avg_product_value'- the average value of the products purchased by the customer.<jupyter_code>def customer_level_agg(x):
    names = {"Total_Transactions":x["InvoiceNo"].nunique(),
            "Total_Products": x["StockCode"].count(),
            "Total_Unique_Products": x["StockCode"].nunique(),
            "Total_Sales": x["Sales"].sum(),
            "Average_Product_Value": x["Sales"].mean()}
    
    
    
    return pd.Series(names, index=["Total_Transactions", "Total_Products","Total_Unique_Products", \
                                   "Total_Sales", "Average_Product_Value"])
customer_data = df.groupby(["CustomerID"]).apply(customer_level_agg)<jupyter_output><empty_output><jupyter_text>**Finally cart/invoice-level data is aggregated**<jupyter_code>def cart_level_agg(x):
    names = {  
        "Avg_Cart_Value": x["Cart_Value"].mean(),
        "Min_Cart_Value": x["Cart_Value"].min(),
        "Max_Cart_Value": x["Cart_Value"].max()}
    
    return pd.Series(names, index=["Avg_Cart_Value", "Min_Cart_Value", "Max_Cart_Value"])   
    
agg_cart_data = cart_data.groupby("CustomerID").apply(cart_level_agg)
# Join together customer-level and cart-level features
customer_df = customer_data.join(agg_cart_data)

customer_df.dtypes<jupyter_output><empty_output><jupyter_text>Customer_df will be used in a segmentation analysis in the second notebook<jupyter_code># Save analytical base table
customer_df.to_csv('analytical_base_table.csv')<jupyter_output><empty_output><jupyter_text>Item-Level Segmentation<jupyter_code># Get item_dummies
item_dummies = pd.get_dummies(df.StockCode )

# Add CustomerID to item_dummies.Note this uses the original tx-level dataframe
item_dummies['CustomerID'] = df.CustomerID
# Create item_data by aggregating at customer level
item_data = item_dummies.groupby('CustomerID').sum()
# Display first 5 rows of item_data
item_data.head(5)
# Save item_data.csv
item_data.to_csv('item_data.csv')
# Get list of StockCodes for the 20 most popular items
top_20_items = item_data.sum().sort_values().tail(20).index
# Keep only features for top 20 items
top_20_item_data = item_data[top_20_items]

# Shape of remaining dataframe
top_20_item_data.shape
# Display first 5 rows of top_20_item_data
top_20_item_data.head()
# Save threshold_item_data.csv
top_20_item_data.to_csv('threshold_item_data.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09909753557792433, 'fraction_numerical': 0.009545296771954183, 'mean_word_length': 6.002430133657351, 'pattern_counts': {'":': 9, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '42886', 'license': 'MIT', 'n_tokens_mistral': 1790, 'n_tokens_neox': 1694, 'n_words': 493, 'path': 'Customer_Segmentation_Workflow1.ipynb', 'repo_name': 'harrinac/Customer-Segmentation', 'size': '5762'}