<jupyter_start><jupyter_text>**Retail Customer Analysis**   Purpose This report is an analysis of the [Online Retail II Dataset found on the University of California Irvine Machine Learning Repository](http://archive.ics.uci.edu/ml/datasets/Online+Retail+II) in order analyze the current state of the business, how we are retaining our customers and clustering our customers into High-Value, Mid-Value, and Low-Value categories in order to improve marketing efforts. We will be creating a **classification model** in order to predict in the future if a customer will be High-Value, Mid-Value, or Low-Value. The **hypothesis** that we will be testing will be to analyze if certain products are being purchased by certain customer groups. For example, High-Value customers are purchasing certain products that can be then marketed to Mid-Value and Low-Value customers in order to increase the revenue.**Null Hypothesis**: Product purchased (description) has no an effect on customer groups. **Alternate Hypothesis**: Product purchased (description) does have an effect on customer groups. **Interest In The Dataset**: Personally, I am interested retail businesses and in customer behavior. I believe marketing efforts can be improved in many organizations by marketing to the right customers and in order to provide value to them based on their needs. **Context On The Data**: The dataset consists of transactional data with customers in 41 different countries who make purchases from an online retail company based in the United Kingdom (UK) that sells unique all-occasion gifts1) Data Collection & Cleaning2) Exploratory Analysis4) Customer Segmentation - RFM Analysis 5) Modeling: Classification6) Product Description & Customer Segment Analysis  7) Conclusion  1 - Data Collection & Cleaning  Steps taken here were to load the dataset and applicable packages into the notebook and perform some cleaning in order to conduct a smooth analysis. Findings in this section would be: - Our dataset contains 1,067,371 rows. - Key information that is included would be Date of Purchase, CustomerIDs, Country, Price, Quantity, and the product that was purchased. Key Cleaning completed: - Removing **negative values** due to cancelled purchases. - Limiting the data to 2010 & 2011 as it also included a single month of 2009.- Data for 2011 is up to December 12th, 2011- Removing **duplicate and null** values. - Creating a Revenue column. - Converting date values into datetimeindex objects and creating columns with year and months.<jupyter_code># Import Standard packages

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from datetime import datetime, timedelta,date

# Load the available data and overview
import pandas as pd
df = pd.read_csv("online_retail_II.csv")<jupyter_output><empty_output><jupyter_text>Intial Dataset:<jupyter_code>df.head()
# Quick descriptive statistics on the data set
df.describe()

df.columns<jupyter_output><empty_output><jupyter_text>Column Descriptions**InvoiceNo**: Invoice number. Nominal. A 6-digit integral number uniquely assigned to each transaction. If this code starts with the letter 'c', it indicates a cancellation. **StockCode**: Product (item) code. Nominal. A 5-digit integral number uniquely assigned to each distinct product.**Description**: Product (item) name. Nominal.**Quantity**: The quantities of each product (item) per transaction. Numeric.**InvoiceDate**: Invice date and time. Numeric. The day and time when a transaction was generated.**UnitPrice**: Unit price. Numeric. Product price per unit in sterling (Â£).**CustomerID**: Customer number. Nominal. A 5-digit integral number uniquely assigned to each customer.**Country**: Country name. Nominal. The name of the country where a customer resides. **Findings*** There are cancelled items recognized by the invoice number.* Revenue may be determined via Quantity * Unit Price* Countries may be used to determine operations outside of UK. * Remove space in 'Customer ID' column<jupyter_code>df.head()
df.dtypes
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>There are 4382 missing descriptions, but that won't have an impactThere are 243007 missing Customer Id values, but we assume that each missing CustomerID represents a new customer. This would mean that the missing Customer ID values are unique to the dataset. InvoiceData is an object and needs to be set to a date. Can also create invoice Month and Invoice Year.  Column Name Creation<jupyter_code>df['Revenue'] = df['Price'] * df['Quantity']
df['InvoiceDate'] = pd.DatetimeIndex(df['InvoiceDate']).date
df['InvoiceMonth'] = pd.DatetimeIndex(df['InvoiceDate']).month
df['InvoiceYear'] = pd.DatetimeIndex(df['InvoiceDate']).year
df['InvoiceYearMonth']  = df['InvoiceDate'].map(lambda date: 100*date.year + date.month)
df.rename(columns={'Customer ID':'CustomerID'}, inplace=True)
df['Quantity'] = df['Quantity'].astype('float')
df.head()
#re-oder columns for readability

df = df[['Invoice','StockCode','InvoiceDate','InvoiceMonth','InvoiceYear','InvoiceYearMonth','CustomerID','Country','Description','Quantity','Price','Revenue']]
df.head()<jupyter_output><empty_output><jupyter_text>Duplicate Check<jupyter_code>#check for duplicates 
df_duplicates = df[df.duplicated(subset = ['Invoice', 'StockCode','Description','Quantity','InvoiceDate','Price','CustomerID','Country'],keep = False)]
df_duplicates.sort_values(by = ['InvoiceDate','Description','StockCode']).head(20)
df_duplicates.count() # lots of duplicate values
#dropping duplicates 
df = df.drop_duplicates(subset = ['Invoice', 'StockCode','Description','Quantity','InvoiceDate','Price','CustomerID','Country'])
df.shape[0]<jupyter_output><empty_output><jupyter_text>Dropped a total of 34,337 rows of duplicate values based on the subset of Invoice, StockCode, Description, Quantity, InvoiceDate, Price, CustomerID, and Country.<jupyter_code>#deleting missing values

df = df.dropna(subset=['Invoice','CustomerID'],how='any')

df.shape[0]<jupyter_output><empty_output><jupyter_text>Date Range Limitation<jupyter_code>df.sort_values(by = 'InvoiceDate').head()
df.tail()<jupyter_output><empty_output><jupyter_text>Data starts : December 1st, 2009 Data ends: December 9th, 2011Limit data to 2010 and 2011 only<jupyter_code>start_date = '2009'
start_date=np.int64(start_date)
type(start_date)
df = df[(df['InvoiceYear']>start_date)]
df.sort_values(by = 'InvoiceDate').head()<jupyter_output><empty_output><jupyter_text>Removing Negative Values<jupyter_code>df= df[df.Quantity>0]
len(df)<jupyter_output><empty_output><jupyter_text>Removing Test Values<jupyter_code>df=df[~df.StockCode.str.contains('TEST')]<jupyter_output><empty_output><jupyter_text>Dataset after cleaning:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>2 - Data Exploration  Now that we have cleaned the data we will explore the current state of the business focusing on Revenue, Items Sold, and Total Number of Customers in 2010 and 2011.<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Now that we have completed our cleaning we are left with just under 750,000 rows and 12 columns.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Some importanat metrics to highlight are **Quantity, Price, and Revenue**. Mean of Quantity = 13.52Mean of Price = \\$3.21Std. Dev of Quanity = 148.90Std. Dev of Price = \\$30.21  Yearly Revenue  How much money are we making each year? For this analysis we will be focusing on Revenue as it is assumed that the goal of the business is Revenue.<jupyter_code>## Yearly Revenue
df_revenue_yearly = df.groupby(['InvoiceYear',])['Revenue'].sum().reset_index()
df_revenue_yearly = df_revenue_yearly.sort_values(by = 'Revenue', ascending = False)
df_revenue_yearly = round(df_revenue_yearly,2)
df_revenue_yearly
#plotting Yearly Revenue

plt.figure(figsize = (10,10))
sns.barplot(x = 'InvoiceYear',y = 'Revenue', hue = 'InvoiceYear', data = df_revenue_yearly)
plt.title('Yearly Revenue', fontsize = 16)
plt.xlabel('Year', fontsize = 12)
plt.ylabel('Revenue', fontsize = 12)
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)


plt.show()<jupyter_output><empty_output><jupyter_text>**Findings**: - Revenue is greater in 2010 than in 2011. - This can be attributed to the incomplete data of December 2011.From this we can conclude that the store is performing relatively the same to the year prior.   Monthly Revenue  Are the months different in revenue numbers?  Is there any **seasonality**? Now that we understand how much revenue we received in 2010 and 2011, we are curious over each year if the months are different or consistent.<jupyter_code>#Monthly Revenue Overview
df_monthlyrevenue = df.groupby(['InvoiceMonth','InvoiceYear','InvoiceYearMonth'])['Revenue'].sum().reset_index() # creating a monthly revenue dataframe
df_monthlyrevenue = df_monthlyrevenue.sort_values(['InvoiceYearMonth'],ascending = True)
#plotting Monthly Revenue with a barchart

plt.figure(figsize = (15,10))
sns.barplot(x = 'InvoiceMonth',y = 'Revenue', hue = 'InvoiceYear', data = df_monthlyrevenue)
plt.title('Monthly Revenue by Year', fontsize = 16)
plt.xlabel('Month', fontsize = 12)
plt.ylabel('Revenue', fontsize = 12)
# Monthly Revenue Lineplot
palette = sns.color_palette("mako_r", 2)
plt.figure(figsize =(10,10))
sns.lineplot(x = 'InvoiceMonth', y = 'Revenue', hue = 'InvoiceYear',   palette=palette,data = df_monthlyrevenue)
plt.title('Monthly Revenue by Year', fontsize = 16)
plt.xlabel('Month', fontsize = 12)
plt.ylabel('Revenue', fontsize = 12)<jupyter_output><empty_output><jupyter_text>**Seasonality Check**Analyzing the graph above we can observe seasonality: - Shallow peaks in March, May, and June. - Dips in January, February, April, and December. - Peaks in September to November. - Max revenue sold: \\$1166460.02- Min revenue sold: \\$446084.92- Average revenue sold: \\$695465.99<jupyter_code>print('Max revenue sold: {}'.format(round(df_monthlyrevenue['Revenue'].max(),2)))
print('Min revenue sold: {}'.format(round(df_monthlyrevenue['Revenue'].min(),2)))
print('Average revenue sold: {}'.format(round(df_monthlyrevenue['Revenue'].mean(),2)))<jupyter_output>Max revenue sold: 1166460.02
Min revenue sold: 446084.92
Average revenue sold: 695465.99<jupyter_text>Monthly Items Sold Overview  How much product are we selling each month? Is there seasonality present here as well? Below we've graphed our Monthly Items Sold.<jupyter_code>df_quantity = df.groupby(['InvoiceYear'])['Quantity'].sum().reset_index()#creating a data frame to group quantites
df_quantity
df_quantity = df.groupby(['InvoiceYearMonth', 'InvoiceYear','InvoiceMonth'])['Quantity'].sum().reset_index()#modifying the dataframe to include months

#Monthly Items Sold Overview
plt.figure(figsize = (10,10))
sns.barplot(x = 'InvoiceMonth', y = 'Quantity',hue = 'InvoiceYear' ,data = df_quantity)
plt.title('Monthly Items Sold', fontsize = 16)
plt.xlabel('Month')
plt.ylabel('Items Sold')
plt.xticks(rotation = 90)<jupyter_output><empty_output><jupyter_text>**Findings**- Quantity of items sold seem to correlate with our revenue.  - Strong showing in March 2010. - Seasonality similiar to Revenue. - Max items sold: 678760.0- Min items sold: 265043.0- Average items sold: 422081.67<jupyter_code>print('Max items sold: {}'.format(df_quantity['Quantity'].max()))
print('Min items sold: {}'.format(df_quantity['Quantity'].min()))
print('Average items sold: {}'.format(round(df_quantity['Quantity'].mean(),2)))<jupyter_output>Max items sold: 678760.0
Min items sold: 265043.0
Average items sold: 422081.67<jupyter_text>Monthly Active Customers  In 2010 and 2011 we had 5790 different customers!  Can we identify any seasonality in our cusomters visiting our stores?<jupyter_code>total_active_customers=df['CustomerID'].nunique()
print('In 2010 and 2011 we had {} customers'.format(total_active_customers))
df_active = df.groupby(['InvoiceMonth','InvoiceYear'])['CustomerID'].nunique().reset_index()
plt.figure(figsize = (10,10))
sns.barplot(x = 'InvoiceMonth', y = 'CustomerID', hue = 'InvoiceYear',data = df_active)
plt.title('Monthly Active Customers', fontsize = 16)
plt.xlabel('Month')
plt.ylabel('Total Customers')<jupyter_output><empty_output><jupyter_text>Findings: - The graph seems to resemble our trends viewed in revenue and items sold.- Maximum number of customers seen in one month 1665- Minimum number of customers seen in one month 615- Average number of customers seen in one month 1026.67<jupyter_code>print('Maximum number of customers seen in one month {}'.format(df_active['CustomerID'].max()))
print('Minimum number of customers seen in one month {}'.format(df_active['CustomerID'].min()))
print('Average number of customers seen in one month {}'.format(round(df_active['CustomerID'].mean(),2)))<jupyter_output>Maximum number of customers seen in one month 1665
Minimum number of customers seen in one month 615
Average number of customers seen in one month 1026.67<jupyter_text>Average Revenue per Order  How much is our customer buying on average on each of their visits?<jupyter_code>df_monthly_avg = df.groupby(['InvoiceYear'])['Revenue'].mean().reset_index()
df_monthly_avg
print('The average revenue in 2010 and 2011 was {}'.format(round(df['Revenue'].mean(),2)))
df_monthly_avg = df.groupby(['InvoiceYear'])['Revenue'].std().reset_index()
df_monthly_avg
df_monthly_avg = df.groupby(['InvoiceYearMonth','InvoiceYear','InvoiceMonth'])['Revenue'].mean().reset_index()
df_monthly_avg_pivot = round(pd.pivot_table(df_monthly_avg, values = 'Revenue', columns = 'InvoiceYear', index = 'InvoiceMonth'),2)
df_monthly_avg_pivot
plt.figure(figsize = (10,10))
sns.barplot(x = 'InvoiceMonth', y = 'Revenue',hue = 'InvoiceYear', data = df_monthly_avg)
plt.title('Monthly Average Revenue', fontsize = 16)
plt.xlabel('Month')
plt.ylabel('Revenue')

print('Standard Deviation per Order Overall is {}'.format(round(df_monthly_avg_pivot.std(),2)))<jupyter_output>Standard Deviation per Order Overall is InvoiceYear
2010    1.63
2011    3.08
dtype: float64<jupyter_text>Findings:- Data remains relatively consistent.- In 2010 we saw an average revenue of \\$21.92- In 2011 we saw an average purchase of \\$22.65.- Average in 2010 and 2011 is \\$22.28.   Summarizing our Data Exploration: - Our revenue between 2010 and 2011 are fairly consistent of approximately \\$8 Million in sales. - We viewed seasonality in our customers purchasing behavior and this was seen across in Revenue, Quantity & total number of Customers. - In 2010 and 2011 we have served a total of 5790 customers.- The average revenue from our customers in 2010 and 2011 was \\$22.65  3 - Customer Analysis  The following analyses are to understand what type of customers we have and how are we retaining them.  New vs. Existing Customer Analysis<jupyter_code>df_first_purchase = df.groupby(['CustomerID'])['InvoiceDate'].min().reset_index()
df_first_purchase.columns = ['CustomerID','FirstPurchaseDate']
df_first_purchase.head()
#grouping df_First_Purchase with df to distinguish between New & Existing Users
df = pd.merge(df,df_first_purchase, on = 'CustomerID')
df['UserType'] = np.where(df['InvoiceDate']>df['FirstPurchaseDate'], 'Existing', 'New')
df.head()
df_usertypes = df.groupby(['InvoiceYear','UserType'])['CustomerID'].nunique().reset_index()
df_usertypes = pd.pivot_table(df_usertypes,values = 'CustomerID', index = 'InvoiceYear', columns = 'UserType',margins = True)
df_usertypes<jupyter_output><empty_output><jupyter_text>New Customer Ratio Based on our new column we are able to find how many new customers we have each month.<jupyter_code>df_user_ratio = ((df.query("UserType == 'New'").groupby(['InvoiceYearMonth'])['CustomerID'].nunique())/(df.query("UserType == 'Existing'").groupby(['InvoiceYearMonth'])['CustomerID'].nunique()))
df_user_ratio = df_user_ratio.reset_index()
df_user_ratio = df_user_ratio.dropna()
df_user_ratio.columns = ['InvoiceYearMonth','NewCustomerRatio']
plt.figure(figsize = (10,10))
sns.barplot(x = df_user_ratio.query('InvoiceYearMonth>201001')['InvoiceYearMonth'],y= 'NewCustomerRatio',data = df_user_ratio)
plt.title('New Customer Ratio', fontsize = 16)

plt.xticks(rotation = 90)
print('Average New Customer Ratio: {}'.format(df_user_ratio['NewCustomerRatio'].mean()))<jupyter_output>Average New Customer Ratio: 0.5553263135695328<jupyter_text>New Customer Ratio seems to be decreasing overtime.   Monthly Retention Rate We have customers, great! But how are we retaining them? Now that we understand how many new customers we are receiving it would be interesting to understand how many of our customers we are actually retaining from the previous month. Methodology: - Crosstab to show which customers are active each month. - Used a for loop to create an array of dictionary which kept **Retained** and **Total User Count** for each month. - Calculated **Monthly Retention Rate** = Retained Customers From Prev Month/Active Customers Total<jupyter_code>df_user_purchase = df.groupby(['CustomerID','InvoiceYearMonth'])['Revenue'].sum().reset_index() #identify which users are active by looking at their revenue per month
df_retention =pd.DataFrame(df[['InvoiceYearMonth','Country','CustomerID']]).sort_values('InvoiceYearMonth').reset_index(drop = True) # create a new retention dataframe
#create retention matrix with crosstab
df_retention = pd.crosstab(df_user_purchase['CustomerID'],df_user_purchase['InvoiceYearMonth'])#this show us which customer are active on each month --> 1 meaning they are active


# for loop to create an array of dictionary which keeps Retained & Total User count for each month

months = df_retention.columns[1:] # gives us all the months from df_retention dataframe above

retention_array = [] # where we will keep the dictionary data

for i in range(len(months)-1):
    retention_data = {} #dictionary used to retain total users from each month and how many are retained
    selected_month = months[i+1] # used to select the next month
    prev_month = months[i] # used to select the previous month
    retention_data['InvoiceYearMonth'] = int(selected_month)
    retention_data['TotalUserCount'] = df_retention[selected_month].sum() # total amount of users in the month we are looking at
    retention_data['RetainedUserCount'] = df_retention[(df_retention[selected_month]>0)&(df_retention[prev_month]>0)][selected_month].sum() # total amount of users retained
    retention_array.append(retention_data)
                                                       
#convert the array to dataframe and calculate Retention Rate

df_retention = pd.DataFrame(retention_array)
df_retention['RetentionRate'] = df_retention['RetainedUserCount']/df_retention['TotalUserCount']
df_retention['PercentChange'] = df_retention['RetentionRate'].pct_change()

#Plotting Retention Rate 
plt.figure(figsize = (10,10))
sns.barplot(x = 'InvoiceYearMonth',y = 'RetentionRate',data = df_retention)
plt.title('Monthly Retention Rate', fontsize = 16)
plt.xlabel('YearMonth',fontsize = 12)
plt.ylabel('RetentionRate', fontsize = 12)
plt.xticks(rotation=90)

print('The average retention rate for each month is {}'.format(round(df_retention['RetainedUserCount'].sum()/df_retention['TotalUserCount'].sum(),4)))<jupyter_output>The average retention rate for each month is 0.3791<jupyter_text>Findings: - There does not seem to be any true trend in regards to our retention rate. - On average we are retaining 38% of our customers from the previous month.   Cohort Retention Rate Now that we understand how many customer we retain on average each month it would be interesting to view how many customers we've retained in a cohort.  Cohort = Each month in the year 2010 & 2011<jupyter_code>#create our retention table again with crosstab()
df_retention_cohort = pd.crosstab(df_user_purchase['CustomerID'],df_user_purchase['InvoiceYearMonth'])
new_column_names = ['m_'+str(column) for column in df_retention_cohort.columns]
df_retention_cohort.columns = new_column_names
df_retention_cohort.head() # look at our newly named rentention matrix
#create the array of Retained Users for each cohort monthly
retention_array = []
for i in range(len(months)):
    retention_data = {}
    selected_month = months[i]
    prev_months = months[:i]
    next_months = months[i+1:]
    for prev_month in prev_months: 
        retention_data[prev_month] = np.nan
    
    total_user_count = retention_data['TotalUserCount'] = df_retention_cohort['m_'+str(selected_month)].sum()
    retention_data[selected_month]=1
    
    query = "{} > 0".format('m_' + str(selected_month))
    

    for next_month in next_months:
        query = query + " and {} > 0".format(str('m_' + str(next_month)))
        retention_data[next_month] = np.round(df_retention_cohort.query(query)['m_' + str(next_month)].sum()/total_user_count,2)
    retention_array.append(retention_data)
    
df_retention_cohort = pd.DataFrame(retention_array)
df_retention_cohort.index = months

#showing new cohort based retention table
df_retention_cohort
#slice the data inorder to leverage a heatmap
df_retention_cohort_heatmap = df_retention_cohort.iloc[0:,1:]

plt.figure(figsize = (20,10))
sns.heatmap(df_retention_cohort_heatmap,cmap="magma",vmax = .5,linewidths=.5,center = .2, annot = True)
plt.title('Cohort Retention Rate', fontsize = 18)
plt.xlabel('Year Month',fontsize = 12)
plt.ylabel('Year Month', fontsize = 12)
plt.xticks(rotation = 90)
plt.show()<jupyter_output><empty_output><jupyter_text>Findings: - Overall our cohort retention rate seems to be quite poor. - From February 2010 to February 2011 we are retaining approximately 5% of our customers. - From February 2010 to November 2011 we are only retaining 3% of our customers. - We are retaining on average 38% of our customers after each month, but in 12 months we are only retaining 5%.  In order to increase revenue, it is important to focus on our current customers and increase retention rates. This can be improved by reaching our to our top value customers more frequently or leveraging social media and email marketing tools to remarket to them.Currently, we have months where we have over **1600** customers so it is important to understand which customers to focus on.   4 -  RFM Analysis (Recency, Frequency, Monetary) Now that we know a bit about our customers and our business we would like to cluster our customers in order to find our: - High Value Customers- Mid Value Customers- Low Value CustomersIn order to conduct this analysis we will be completing an RFM Analysis which takes into account a customer's Recency, Frequency, and Monetary value and provides an overall score. This will be completed by using the clustering algorithm KMeans from Sklearn.This analysis will help overall marketing efforts and allow the marketing team to dedicate their efforts to our best customers.   Recency<jupyter_code>#creating a dataframe to keep UNIQUE CustomerIDs
df_user = pd.DataFrame(df['CustomerID'].unique())
df_user.columns = ['CustomerID']
df_last_purchase = df.groupby(['CustomerID'])['InvoiceDate'].max().reset_index()
df_last_purchase.columns = ['CustomerID','LastPurchaseDate']
df_last_purchase['Recency'] = (df_last_purchase['LastPurchaseDate'].max()-df_last_purchase['LastPurchaseDate']).dt.days # calculating the recency
df_last_purchase.head()
df_user = pd.merge(df_user,df_last_purchase[['CustomerID','Recency']]) # merging unique customer IDs with the last_purchase_dataframe
df_user.head()
df_user.describe()
#Look at the distribution of Recency
plt.figure(figsize = (10,10))
sns.distplot(df_user['Recency'])
plt.title('Recency Distribution Within the Customers', fontsize = 16)
plt.xlabel('Recency', fontsize = 12)
plt.ylabel('Customer Count', fontsize = 12)

# use KMeans Clustering for Recency Clustering
from sklearn.cluster import KMeans

#find out how many clusters are optimal using the elbow method
dic = {} #store the clusterin values in a dictionary
df_recency= df_user[['Recency']] #label what we are clustering
for k in range(1,10):
    kmeans = KMeans(n_clusters = k,max_iter = 1000).fit(df_recency)
    df_recency['clusters'] = kmeans.labels_
    dic[k] = kmeans.inertia_
plt.figure(figsize = (10,10))
plt.plot(list(dic.keys()),list(dic.values()))
plt.title('Elbow Method to Determine Cluster Size', fontsize = 16)
plt.show()<jupyter_output><empty_output><jupyter_text>Analyzing to view where there is a elbow created in the graph. From this analysis it seems it is between 3 and 4. We will be choosing 4 for our clustering sizes.<jupyter_code>#Cluster Customer based on Recency

kmodel_recency = KMeans(n_clusters = 4)
kmodel_recency.fit(df_user[['Recency']])
df_user['RecencyCluster'] = kmodel_recency.predict(df_user[['Recency']])

df_user.head()
# get statistical analysis for each cluster
df_user.groupby(['RecencyCluster'])['Recency'].describe()
#function for ordering cluster numbers

def order_cluster(cluster_field_name, target_field_name,df,ascending):
    new_cluster_field_name = 'new_' + cluster_field_name
    df_new = df.groupby(cluster_field_name)[target_field_name].mean().reset_index()
    df_new = df_new.sort_values(by=target_field_name,ascending=ascending).reset_index(drop=True)
    df_new['index'] = df_new.index
    df_final = pd.merge(df,df_new[[cluster_field_name,'index']], on=cluster_field_name)
    df_final = df_final.drop([cluster_field_name],axis=1)
    df_final = df_final.rename(columns={"index":cluster_field_name})
    return df_final

df_user = order_cluster('RecencyCluster','Recency',df_user,False)
df_user.groupby(['RecencyCluster'])['Recency'].describe()<jupyter_output><empty_output><jupyter_text>Running descriptive statistics on the recency cluster. **Findings**: - Group of very recent clusters in group 3. - Recency begins to diminish in group 1 and very little recency seen in group 0.   Frequency We will be taking the similar steps with frequency. **First** looking at our distribution:<jupyter_code>df_user.head()
#frequency of orders 
df_frequency = df.groupby(['CustomerID'])['InvoiceDate'].count().reset_index()
df_frequency.columns = ['CustomerID', 'Frequency']
df_user = pd.merge(df_user,df_frequency,on= 'CustomerID')
df_user.head()
plt.figure(figsize = (10,10))
sns.distplot(df_frequency.query('Frequency<1000')['Frequency'])
plt.title('Frequency Distribution',fontsize = 16)
plt.xlabel('Frequency' ,fontsize = 12)
plt.ylabel('Count', fontsize = 12)
k_model_frequency = KMeans(n_clusters = 4)
k_model_frequency.fit(df_frequency [['Frequency']])
k_model_frequency_predict = k_model_frequency.predict(df_frequency [['Frequency']])
df_user['FrequencyCluster'] = k_model_frequency_predict
df_user.head()
#Statistical Analysis of clusters based on frequency
df_user.groupby(['FrequencyCluster'])['Frequency'].describe()
#Ordering clusters based on frequency
df_user = order_cluster('FrequencyCluster','Frequency',df_user,True)
df_user.groupby(['FrequencyCluster'])['Frequency'].describe()<jupyter_output><empty_output><jupyter_text>**Findings**: - Very frequent customers in group 3. - Frequency seems to tail off quite rapidly in group 2,1,and 0.    Monetary<jupyter_code>df_customer_revenue=df.groupby(["CustomerID"])["Revenue"].sum().reset_index()
df_customer_revenue=pd.merge(df_user, df_customer_revenue, on="CustomerID")
df_customer_revenue.head()

# Revenue Distribution
plt.figure(figsize=(10,10))
sns.distplot(df_customer_revenue.query("Revenue < 10000")["Revenue"])
plt.title('Monetary Distribution',fontsize = 16)
plt.xlabel('Monetary' ,fontsize = 12)
plt.ylabel('Count', fontsize = 12)
df_monetary = df.groupby(['CustomerID'])['Revenue'].sum().reset_index()
df_monetary.columns = ['CustomerID', 'Monetary']
# Segmenting Customers Based on their Monetary Value
k_model_monetary=KMeans(n_clusters=4)
k_model_monetary.fit(df_monetary[['Monetary']])
k_model_monetary_predict=k_model_monetary.predict(df_monetary[['Monetary']])
df_user["MonetaryCluster"]=k_model_monetary_predict

df_user = pd.merge(df_user,df_monetary,on= 'CustomerID')
df_user.head()

#Ordering clusters based on monetary
df_user= order_cluster('MonetaryCluster','Monetary',df_user,True)
df_user.groupby(['MonetaryCluster'])['Monetary'].describe()<jupyter_output><empty_output><jupyter_text>Findings: - We have some very high value customers in Cluster 3.- Cluster 2 has the highest number of users and an admirable amount in regards to monetary value.   Overall Score<jupyter_code>df_user.head()
#Calculate overall score and use mean() to see details

df_user['OverallScore'] = df_user['RecencyCluster'] + df_user['FrequencyCluster'] + df_user['MonetaryCluster']
df_user.groupby('OverallScore')['Recency','Frequency','Monetary'].mean()<jupyter_output><empty_output><jupyter_text>The scoring above clearly shows that our customers with a score of 8 is our best customer and our customers with a socre of 0 is our worst.<jupyter_code>df_user['Segment'] = 'Low-Value'
df_user.loc[df_user['OverallScore']>4,'Segment'] = 'Mid-Value'
df_user.loc[df_user['OverallScore']>6,'Segment'] = 'High-Value'
df_user.head()
plt.figure(figsize = (15,10))
sns.scatterplot(x = df_user['Frequency'], y = df_user['Monetary'], hue = 'Segment', data = df_user)
plt.title('Frequency vs. Revenue Segments',fontsize = 16)
plt.xlabel('Frequency',fontsize = 12)
plt.ylabel('Revenue',fontsize = 12)

plt.figure(figsize = (15,10))
sns.scatterplot(x = df_user['Recency'], y = df_user['Monetary'], hue = 'Segment', data = df_user)
plt.title('Recency vs Revenue Segments',fontsize = 16)
plt.xlabel('Recency',fontsize = 12)
plt.ylabel('Revenue',fontsize = 12)
plt.figure(figsize = (15,10))
sns.scatterplot(x = df_user['Recency'], y = df_user['Frequency'], hue = 'Segment', data = df_user)
plt.title('Recency vs Frequency Segments',fontsize = 16)
plt.xlabel('Recency',fontsize = 12)
plt.ylabel('Frequency',fontsize = 12)
segment_grouping= df_user.groupby('Segment')['Segment'].count()
segment_grouping.reindex(['High-Value','Mid-Value','Low-Value'], axis=0)
df_user.head()
df_unique =df[['CustomerID','Country']]
df_unique = df_unique.drop_duplicates(keep = 'first')
df_user_country = pd.merge(df_user,df_unique, how = 'inner', on = 'CustomerID')
df_user_country.head()
df_user_country = pd.pivot_table(data =df_user_country, values = 'CustomerID', index = 'Country', columns = 'Segment', aggfunc = lambda x: len(x.unique()))
df_user_country = df_user_country.fillna(0)
df_user_country = df_user_country.sort_values(by = ['High-Value', 'Mid-Value'], ascending = False)
df_user_country= df_user_country.reindex(['High-Value','Mid-Value','Low-Value'], axis=1)
df_user_country.head()
df_user_country.tail()<jupyter_output><empty_output><jupyter_text>5 - Modeling: Classification<jupyter_code>#Import libraries
from sklearn.feature_extraction import text # use text processing tools
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import KFold,LeaveOneOut, cross_val_score, train_test_split
from sklearn.naive_bayes import MultinomialNB, GaussianNB, BernoulliNB
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn import tree, metrics
from sklearn import model_selection, metrics, linear_model, datasets, feature_selection
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import MinMaxScaler
from scipy import stats, special
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import average_precision_score



#do not show warnings
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Purpose: - Using a classification model to determine the segment of a customer: High-Value, Mid-Value, and Low-Value. - Classification models: RandomForestClassifier, DecisionTreeClassifier, and MultinomialNB. - Use of ColumnTransformer to include both numeric data and text data from Description using CountVectorizer.<jupyter_code>df_logreg = df[['CustomerID','Description', 'Country','Quantity', 'Price','Revenue','UserType']] # creating a new dataframe to query on with independent and dependent variables
df_logreg = pd.merge(df_logreg,df_user, how = 'inner', on = 'CustomerID') # merging the data with our clutered group analysis
df_logreg.head()
labelencoder = LabelEncoder()
# Assigning numerical values and storing in another column
df_logreg['SegmentNumeric'] = labelencoder.fit_transform(df_logreg['Segment'])
df_logreg['UserTypeNumeric'] = labelencoder.fit_transform(df_logreg['UserType'])
df_logreg['CountryNumeric'] = labelencoder.fit_transform(df_logreg['Country'])
df_logreg.head()
feature_cols = ['Description','Quantity', 'Price', 'Revenue', 'Recency', 'RecencyCluster', 'Frequency', 'FrequencyCluster', 'Monetary', 'MonetaryCluster', 'OverallScore','UserTypeNumeric','CountryNumeric']
X_train, X_test, y_train, y_test = train_test_split(df_logreg[feature_cols], df_logreg.SegmentNumeric, test_size=0.40, random_state=0)
# validate set shapes
print(X_train.shape, y_train.shape)
print(X_test.shape, y_test.shape)
numeric_features = ['Quantity', 'Price', 'Revenue', 'Recency', 'RecencyCluster', 'Frequency', 'FrequencyCluster', 'Monetary', 'MonetaryCluster', 'OverallScore','UserTypeNumeric']
numeric_transformer = Pipeline(steps=[
        ('scaler', MinMaxScaler())])
preprocessor = ColumnTransformer([
    ('description_countvec',CountVectorizer(stop_words = 'english'),'Description'),
    ('num', numeric_transformer,numeric_features)])
#create an array of models
models = []
models.append(("NB",MultinomialNB()))
models.append(("RF",RandomForestClassifier()))
models.append(("Dtree",DecisionTreeClassifier()))

for name, model in models:
    pipeline = Pipeline(steps = [('preprocessor', preprocessor),
                                ('classifier',model)])
    pipeline.fit(X_train,y_train)
    y_pred = pipeline.predict(X_test)
    print(name, "Accuracy:",metrics.accuracy_score(y_test, y_pred))
#measure the accuracy using KFold Cross Validation
for name,model in models:
    kfold = KFold(n_splits=10,shuffle = True)
    cv_result = cross_val_score(pipeline,X_train,y_train, cv = kfold,scoring = "accuracy")
    print(name, (cv_result))
#measure accuracy using shuffle split cross validation
for name,model in models:
    shufflesplit = model_selection.ShuffleSplit(n_splits=10, test_size=0.30, random_state=100)
    results_4 = model_selection.cross_val_score(pipeline, X_train, y_train, cv=shufflesplit)
    print(name, "Accuracy: %.2f%% (%.2f%%)" % (results_4.mean()*100.0, results_4.std()*100.0))
clf_rf = Pipeline(steps = [('preprocessor', preprocessor),
                                ('classifier',RandomForestClassifier())])
clf_rf.fit(X_train,y_train)
y_pred_rf = clf_rf.predict(X_test)
print( "Accuracy:",metrics.accuracy_score(y_test, y_pred_rf))
labels = {0:'High-Value',1:'Low-Value',2:'Mid-Value'}
cnf_matrix = metrics.confusion_matrix(y_test, y_pred_rf)
cnf_matrix
class_names=[0,1,2] # name  of classe
fig, ax = plt.subplots()
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names)
plt.yticks(tick_marks, class_names)
# create heatmap
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, cmap="YlGnBu" ,fmt='g')
ax.xaxis.set_label_position("top")
plt.tight_layout()
plt.title('Confusion matrix', y=1.1)
plt.ylabel('Actual label')
plt.xlabel('Predicted label')<jupyter_output><empty_output><jupyter_text>**Labels**High-Value = 0 Mid-Value = 2 Low-Value = 1<jupyter_code># Print the precision and recall, among other metrics
print(metrics.classification_report(y_test, y_pred_rf, digits=3))
def get_column_names_from_ColumnTransformer(column_transformer):    
    col_name = []
    for transformer_in_columns in column_transformer.transformers_[:-1]:#the last transformer is ColumnTransformer's 'remainder'
        raw_col_name = transformer_in_columns[2]
        if isinstance(transformer_in_columns[1],Pipeline): 
            transformer = transformer_in_columns[1].steps[-1][1]
        else:
            transformer = transformer_in_columns[1]
        try:
            names = transformer.get_feature_names()
        except AttributeError: # if no 'get_feature_names' function, use raw column name
            names = raw_col_name
        if isinstance(names,np.ndarray): # eg.
            col_name += names.tolist()
        elif isinstance(names,list):
            col_name += names    
        elif isinstance(names,str):
            col_name.append(names)
    return col_name
full_col_names = get_column_names_from_ColumnTransformer(preprocessor)<jupyter_output><empty_output><jupyter_text>Understaning Our Feature Importance<jupyter_code>print('Total number of features in the model: {}'.format(len(clf_rf.steps[1][1].feature_importances_)))
logReg_coeff = pd.DataFrame({'feature_name': full_col_names, 'model_coefficient': clf_rf.steps[1][1].feature_importances_.transpose().flatten()})
logReg_coeff = logReg_coeff.sort_values('model_coefficient',ascending=False)
logReg_coeff_top = logReg_coeff.head(30)
logReg_coeff_bottom = logReg_coeff.tail(20)
# Plot top 5 coefficients
plt.figure().set_size_inches(10, 6)
fg3 = sns.barplot(x='feature_name', y='model_coefficient',data=logReg_coeff_top, palette="Blues_d")
fg3.set_xticklabels(rotation=90, labels=logReg_coeff_top.feature_name)
# Plot bottom 5 coefficients
plt.figure().set_size_inches(10,6)
fg4 = sns.barplot(x='feature_name', y='model_coefficient',data=logReg_coeff_bottom, palette="GnBu_d")
fg4.set_xticklabels(rotation=90, labels=logReg_coeff_bottom.feature_name)
plt.xlabel('Feature')
plt.ylabel('Coefficient')
plt.subplots_adjust(bottom=0.4)<jupyter_output><empty_output><jupyter_text>6 - Product Description & Customer Segment Analysis  We'd like to answer our hypothesis to determine if the products have any importance based on the customer segment. In order to analyze this we will check  each  customer order for any of the words from the top 10 features in **Description**.<jupyter_code>description = logReg_coeff.sort_values('model_coefficient',ascending=False)
description_top = description.head(600)
description_top=list(description_top['feature_name'][11:21])
df_description_top = df_logreg[['CustomerID','Description','Country','Segment']]
df_description_top["Description"] = df_description_top["Description"].str.lower()
df_description_top.head()
print('The top 10 words in our feature extraction are: {}'.format(description_top))
df_description_top['TopFeature']=np.where(df_description_top['Description'].apply(lambda x: any([k in x for k in description_top])),'Yes','No')
df_description_top.head()
df_description_top['SegmentNumeric'] = labelencoder.fit_transform(df_description_top['Segment'])
df_description_top['TopFeatureNumeric'] = labelencoder.fit_transform(df_description_top['TopFeature'])
df_description_top.corr()
df_description_top_pivot =  pd.pivot_table(data =df_description_top, values = 'CustomerID', index = 'Segment', columns = 'TopFeature', aggfunc = lambda x: len(x))
df_description_top_pivot=df_description_top_pivot.reindex(['High-Value','Mid-Value','Low-Value'], axis=0)
df_description_top_pivot=df_description_top_pivot.reindex(['Yes','No'], axis=1)
df_description_top_pivot<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11556267806267806, 'fraction_numerical': 0.018925518925518924, 'mean_word_length': 6.7342120794806215, 'pattern_counts': {'":': 1, '<': 141, '<?xml version=': 0, '>': 150, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '211447', 'license': 'MIT', 'n_tokens_mistral': 12867, 'n_tokens_neox': 11873, 'n_words': 3360, 'path': 'OnlineRetailAnalysis_Nima-Aref.ipynb', 'repo_name': 'nimaaref/RetailCustomerAnalysis', 'size': '39312'}