<jupyter_start><jupyter_text>Product Pareto, Apriori, & Association  Table of Contents1. Introduction2. Install & Import Packages3. Exploration Countries Top 10 Products by Frequency, Quantity, Total Sales Low-Price Transactions4. Pareto5. Apriori (Pruning Item Sets)6. Association Rules Pruning Heatmap Parallel Plot7. References  1. IntroductionLet's work with the retail transactions data we [cleaned and processed](https://github.com/hrishipoola/Product_Pareto_Apriori_Association), focusing on customers from Germany. We'll explore Top 10 products and apply Pareto, or the 80/20 rule, in which roughly 80% of effects come from 20% of the causes (e.g., 20% of products generate 80% of sales). This can help us focus on the core group of customers and products that generate the overwhelming majority of sales. In the future, we'll explore time series trends with this data set in more depth. Next, we'll uncover products that are more likely to be purchased together in order to develop recommendations and bundles. If you want to learn about market basket concepts, including association metrics and pruning, in more depth, check out [Datacamp's Market Basket Analysis course](https://learn.datacamp.com/courses/market-basket-analysis-in-python). As a large set of unique items creates an unmanageable number of combinations and association rules, we'll first apply the Apriori algorithm to prune item sets. Apriori is based on the principle that subsets of frequent sets are also frequent and prunes large numbers of item sets by only focusing on items that have a minimum level of support (purchase frequency). We'll also visualize how different levels of minimum support affect the number of item sets created. Even a pruned frequent item set can generate an unruly number of association rules. Let's prune our association rules to a more manageable handful. For the pruning metric, let's choose lift, a measure of likelihood for items to co-occur. Lift is >1 for items that co-occur more than we'd expect if they were independently distributed across transactions. We'll also visualize how the number of association rules changes with increasing levels of lift criteria. We'll uncover products more likely to be purchased together that we can use to develop recommendations or bundle. Lastly, we'll visualize these associations through a heatmap and a parallel plot.    2. Install & Import Packages<jupyter_code>import pandas as pd
import numpy as np
import datetime as dt
from datetime import date, timedelta
from datetime import date

from mlxtend.frequent_patterns import apriori, association_rules
from pandas.plotting import parallel_coordinates

import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib import style
sns.set_style("darkgrid")
style.use('fivethirtyeight')

import io
from io import BytesIO
from zipfile import ZipFile
# Mount google drive
from google.colab import drive 
drive.mount('/content/drive')
from google.colab import files
uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>3. Exploration<jupyter_code>df = pd.read_csv('online_retail_processed.csv.gz', compression='gzip')
df.head()<jupyter_output><empty_output><jupyter_text>Let's take a look at the countries with the highest number of transactions. The overwhelming majority are in the UK.<jupyter_code>df.groupby('Country', as_index=False)['InvoiceNo'] \
  .count() \
  .sort_values('InvoiceNo', ascending = False) \
  .reset_index(drop=True) \
  .head(10)<jupyter_output><empty_output><jupyter_text>Germany Top 10Going forward, let's focus on the Germany market.  Frequency<jupyter_code>df[df.Country =='GERMANY'].Description.value_counts() \
                          .head(10) \
                          .sort_values(ascending=False) \
                          .iloc[::-1] \
                          .plot(kind='barh', 
                                x='Description', 
                                y='Frequency', 
                                legend=False, 
                                title='Germany - Top 10 Items by Frequency',
                                color='slategray',
                                figsize=(12,5))<jupyter_output><empty_output><jupyter_text>Quantity<jupyter_code>df[df.Country =='GERMANY'].groupby('Description', as_index=False)['Quantity','TotalPrice'] \
                          .sum() \
                          .sort_values(by='Quantity', ascending=False) \
                          .head(10) \
                          .iloc[::-1] \
                          .plot(kind='barh', 
                                x='Description', 
                                y='Quantity', 
                                legend=False, 
                                title='Germany - Top 10 Items by Quantity',
                                color='pink',
                                figsize=(12,5))<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.
  """Entry point for launching an IPython kernel.<jupyter_text>Total Sales<jupyter_code># Top 10 items by total price
df[df.Country =='GERMANY'].groupby('Description', as_index=False)['Quantity','TotalPrice'] \
                          .sum() \
                          .sort_values(by='TotalPrice', ascending=False) \
                          .head(10) \
                          .iloc[::-1] \
                          .plot(kind='barh', 
                                x='Description', 
                                y='TotalPrice', 
                                legend=False, 
                                title='Germany - Top 10 Items by Total Sales (£)',
                                color='turquoise',
                                figsize=(12,5))<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:2: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.<jupyter_text>Low-Price Transactions<jupyter_code>germany = df[df.Country =='GERMANY']
low_price_transactions = len(germany[germany.UnitPrice <= 5])/len(germany.UnitPrice)*100
print("{:.2f} % of transactions were for items 5 pounds or less.".format(low_price_transactions))
low_price_sales = germany[germany.UnitPrice<=5].TotalPrice.sum()/germany.TotalPrice.sum()*100
print("{:.2f} % of total sales come from items priced 5 pounds or less.".format(low_price_sales))<jupyter_output>77.53 % of total sales come from items priced 5 pounds or less.<jupyter_text>4. Pareto Let's also apply Pareto, or the 80/20 rule, in which roughly 80% of effects come from 20% of the causes (e.g., 20% of products generate 80% of sales). This can help the retailer understand and focus on the core group of customers or products that generate the vast majority of sales.<jupyter_code>def pareto(df, variable_20_percent, measure):
    '''
    # Returns contribution of 20% of a cause variable to 80% of sales
    df: dataset 
    variable_20_percent: column to be checked for 20% of its unique entries to affect 80% of sales
    measure: string - the measure that is conributing to 80%
    '''
    pareto_df = df.groupby(variable_20_percent, as_index=False)['TotalPrice'].sum().sort_values(by='TotalPrice', ascending=False)
    pareto_df['Cumulative_Percentage'] = round(pareto_df['TotalPrice'].cumsum() / pareto_df['TotalPrice'].sum()*100,2)

    # Number of products that make up 80% of sales
    percent_80 = pareto_df[pareto_df['Cumulative_Percentage'] <= 80]
    item_num = percent_80[variable_20_percent].nunique()
    item_percent = round(item_num / df[variable_20_percent].nunique()*100,2)

    print(item_percent, '% of', measure, 'generated 80% of sales') 
    print(item_percent,'% means', item_num, measure)
pareto(germany, 'Description', 'products')
pareto(germany, 'CustomerID', 'customers')<jupyter_output>32.98 % of customers generated 80% of sales
32.98 % means 31 customers<jupyter_text>In Germany, 80% of sales come from just 31 customers.   5. Apriori (Pruning Item Sets) A large set of unique items creates an unmanageable number of combinations and association rules. The apriori algorithm is based on the principle that subsets of frequent sets are also frequent and prunes large numbers of item sets by only focusing on items that have a minimum level of support (purchase frequency). Apriori requires onehot encoded items as input so let's create one<jupyter_code>def encode(x):
    if x <= 0:
        return 0
    else:
        return 1

def create_encoded_basket(df, country):
    
    # Returns one-hot-encoded basket of items. Inputs are df and country as all caps string in quotes
    global basket
    basket = df[df.Country == country].groupby(['InvoiceNo', 'Description'])['Quantity'] \
                                      .sum() \
                                      .unstack() \
                                      .fillna(0)

    basket = basket.applymap(encode)
    return basket

create_encoded_basket(df, 'GERMANY')<jupyter_output><empty_output><jupyter_text>Let's see how the number of itemsets is pruned by minimum level of support.<jupyter_code># Plot size of frequent itemsets vs minimum support level
supports = list(np.arange(0.01, 0.10, 0.01))
itemsets = []

for support in supports:
    frequent_itemsets = apriori(basket, min_support = support, max_len=3, use_colnames = True)
    num_itemsets = len(frequent_itemsets)
    itemsets.append(num_itemsets)

plot_itemsets = pd.DataFrame(list(zip(supports, itemsets)), columns=['support','itemsets'])
plt.figure(figsize=(10,5))
sns.lineplot(data=plot_itemsets, x='support', y='itemsets', color='turquoise')
plt.title('Number of Itemsets vs. Minimum Support')<jupyter_output><empty_output><jupyter_text>We can see the number of frequent itemsets drops dramatically with higher support levels. Let's choose 0.05 for our minimum level of support.<jupyter_code># Apriori to select the most important itemsets
itemsets = apriori(basket, min_support = 0.05, max_len=3, use_colnames = True)
itemsets.sort_values('support', ascending = False).head()
print('Number of frequent itemsets:', len(itemsets))<jupyter_output>Number of frequent itemsets: 60<jupyter_text>6. Association Rules  Pruning Even a pruned frequent item set can generate an unruly number of association rules. Let's prune our association rules to a more manageable handful. For the pruning metric, let's choose lift, a measure of likelihood for items to co-occur. Lift is >1 for items that co-occur more than we'd expect if they were independently distributed across transactions. We'll also visualize how the number of association rules changes with increasing levels of lift criteria.<jupyter_code># Plot number of association rules vs lift threshold
lifts = list(np.arange(1, 6, 0.2))
rules = []

for lift in lifts:
    associations = association_rules(itemsets, metric = 'lift', min_threshold = lift)
    num_rules = len(associations)
    rules.append(num_rules)

plot_rules = pd.DataFrame(list(zip(lifts, rules)), columns=['lift','rules'])
plt.figure(figsize=(10,5))
sns.lineplot(data=plot_rules, x='lift', y='rules', color='turquoise')
plt.title('Number of Association Rules vs. Lift Threshold')<jupyter_output><empty_output><jupyter_text>Let's choose a lift threshold of 2.5 to get to a manageable number of rules.<jupyter_code>associations = association_rules(itemsets, metric = 'lift', min_threshold = 2.5)
associations.sort_values('lift', ascending=False).reset_index(drop=True)
print('Number of association rules:', len(associations))<jupyter_output>Number of association rules: 12<jupyter_text>A few insights: - Customer is 6.5x as likely to buy WOODLAND CHARLOTTE BAG than an average customer (lift) if they buy RED RETROSPOT CHARLOTTE BAG. This rule is true in 84% of transactions (confidence) - Customer is 4.11x as likely to buy PLASTERS IN TIN WOODLAND ANIMALs than an average customer if they buy PLASTERS IN TIN CIRCUS PARADE. This rule is true in 58% of transactions. - Customer is 3.3x as likely to buy ROUND SNACK BOXES SET OF 4 WOODLAND than an average customer if they buy ROUND SNACK BOXES SET OF 4 FRUITS. This rule is true in 83% of transactions. We can use this to recommend or bundle these pairs of products together.   Heatmap A heatmap makes the association rules and lift metrics across antecedents and consequents easier to see.<jupyter_code># association_rules returns frozen sets, let's convert them to strings
associations['antecedents'] = associations['antecedents'].apply(lambda a: ','.join(list(a)))
associations['consequents'] = associations['consequents'].apply(lambda a: ','.join(list(a)))

# Pivot data on consequents and antecedents with lift as values
pivot = associations.pivot(index='consequents', columns='antecedents', values='lift')

plt.figure(figsize=(10,5))
sns.heatmap(pivot, annot = True, cbar = False, cmap='viridis_r')
plt.title('Lift between Antecedents & Consequents')
plt.yticks(rotation=0)
plt.xticks(rotation=65)<jupyter_output><empty_output><jupyter_text>Parallel Plot Let's plot association just the associations using a parallel plot. The consequents on the right are the same as the antecedents on the left.<jupyter_code>associations = association_rules(itemsets, metric = 'lift', min_threshold = 2.5)

# association_rules returns frozen sets, let's convert them to strings
associations['antecedents'] = associations['antecedents'].apply(lambda a: ','.join(list(a)))
associations['consequents'] = associations['consequents'].apply(lambda a: ','.join(list(a)))

# Create coordinates datafram from rules
associations['rule'] = associations.index
coords = associations[['antecedents', 'consequents', 'rule']]

plt.figure(figsize=(8,5))
parallel_coordinates(coords,'rule', colormap='viridis')
plt.title('Associations')
plt.legend('')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09218278488288548, 'fraction_numerical': 0.015375772381089238, 'mean_word_length': 3.70872801082544, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '460233', 'license': 'MIT', 'n_tokens_mistral': 4152, 'n_tokens_neox': 3852, 'n_words': 1358, 'path': 'Product_Focus_&_Recs_using_Pareto,_Apriori,_&_Association.ipynb', 'repo_name': 'hrishipoola/Product_Pareto_Apriori_Association', 'size': '13918'}