<jupyter_start><jupyter_text>Purchase History and Market Basket Analysis

Company XYZ is an online grocery store. In the current version of the website, they have manually grouped the items into a few categories based on their experience.

However, they now have a lot of data about user purchase history. Therefore, they would like to put the data into use!

This is what they asked you to do:

- The company founder wants to meet with some of the best customers to go through a focus group with them. You are asked to send the ID of the following customers to the founder:
 - the customer who bought the most items overall in her lifetime
 - for each item, the customer who bought that product the most
- Cluster items based on user co-purchase history. That is, create clusters of products that have the highest probability of being bought together. The goal of this is to replace the old/manually created categories with these new ones. Each item can belong to just one cluster.

While not requested, we'll also demonstrate basic market basket analysis and see how the rules it derives can be used to craft sales or recommendations to our users.  Index
1. [Initial Exploration](explore)
2. [Top Customers](topcustomers)
    1. [Overall](overall)
    2. [Each Item](eachitem)
3. [Clustering Products](cluster)
    1. [K-Means](kmeans)
    2. [Correlation](correlation)
4. [Market Basket Analysis](mba)
5. [Conclusions](conclusions) 
 Initial Exploration<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import normalize
from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules
item_ids = pd.read_csv('PRIVATE CSV')
print(item_ids.shape)
item_ids.head()
purchases = pd.read_csv('PRIVATE CSV')
print(purchases.shape)
purchases.head()<jupyter_output>(39474, 2)<jupyter_text>We need to split the item ID's into individual columns, then rename them to their item names, and combine the purchase history of individual users.<jupyter_code>onehots = purchases['id'].str.split('\s*,\s*',
                            expand=True).stack().str.get_dummies().sum(level=0)
purchases = purchases.drop('id', axis='columns').merge(onehots, 
                                            left_index=True, right_index=True)
item_map = {str(key): value for key, value in 
            item_ids.set_index('Item_id').to_dict()['Item_name'].items()}
purchases = purchases.rename(item_map, axis='columns').set_index('user_id')
purchases = purchases.groupby('user_id').sum()
purchases.head()<jupyter_output><empty_output><jupyter_text>The most common item by far is lettuce and the least common is flour. The items on the low end are likely going to be the hardest to group. Lettuce being so prevalent may also be a problem simply due to being in so many baskets (nearly 60%).<jupyter_code>purchases.mean().sort_values(ascending=False).head()
purchases.mean().sort_values(ascending=False).tail()<jupyter_output><empty_output><jupyter_text>Top Customers 
To get the customer who purchased the most items we simply need to sum all of the counts and sort in descending order.

We see that customer 269335 has purchased the most items with 72.<jupyter_code>purchases.T.sum().sort_values(ascending=False).reset_index().rename({0:'count'},
                                                             axis='columns')[:1]<jupyter_output><empty_output><jupyter_text>Gettting the customer who purchased each item the most is a simple call to `idxmax` which gives us the first occurence of the maximum value over the defined axis.<jupyter_code>pd.DataFrame(purchases.idxmax().T).rename({0:'customer'}, axis='columns')<jupyter_output><empty_output><jupyter_text>Clustering Products 
K-Means clustering is going to be where we start, allowing us to easily find groupings of products.

We'll start by creating an item similarity matrix and then use that to determine how many groupings is optimal with an elbow and silhoutte plot.<jupyter_code>normed_purchases = normalize(purchases.values, axis=0)
item_similarity = normed_purchases.T.dot(normed_purchases)
item_similarity = pd.DataFrame(item_similarity, index=purchases.columns,
                               columns=purchases.columns)
clusters = range(2, 48)
elbow = []
silhouette = []

for n in clusters:
    kmeans = KMeans(n_clusters=n, random_state=13, n_jobs=-1)
    kmeans = kmeans.fit(item_similarity.T)
    label = kmeans.predict(item_similarity.T)
    
    elbow.append(kmeans.inertia_)
    silhouette.append(silhouette_score(item_similarity.T, label))

plt.plot(clusters, elbow)
plt.show()
plt.plot(clusters, silhouette)
plt.show()<jupyter_output><empty_output><jupyter_text>There doesn't seem to be a clear point on either graph for optimal number of clusters. I tested multiple points and found 13 or 14 clusters to be the best based on the what items are in each group.

The resulting groups are pretty solid, not much needs to be moved around besides `pet_items`.<jupyter_code>GROUPS = 14

kmeans = KMeans(n_clusters=GROUPS, random_state=13, n_jobs=-1)
kmeans = kmeans.fit(item_similarity.T)
label = kmeans.predict(item_similarity.T)

df = pd.DataFrame({'Item Name': purchases.columns, 'Group': label})
group_sets = {}
for i in range(GROUPS):
    group_sets[i] = list(df[df['Group'] == i]['Item Name'])<jupyter_output><empty_output><jupyter_text>Now that we have the groups, we can pull out the most popular categories. Vegetables and Fruits take the top two spots, which makes sense since they have the largest item sets, are generally staple ingredients in many dishes, and have a low price point. We can also see our weakest categories, surprisingly cereal is at the bottom. Based on the fact that this is a grocery store, cereal being so low seems abnormal and something the products team might want to dig into further.<jupyter_code>def find_item_set(group):
    return group_sets[group]
item_counts = pd.DataFrame(purchases.sum()).reset_index()
item_counts.columns = ['Item Name', 'Count']

df = df.merge(item_counts)
group_counts = df.groupby('Group').sum().sort_values('Count', ascending=False).reset_index()
group_counts['items_in_set'] = group_counts['Group'].apply(find_item_set)

group_counts<jupyter_output><empty_output><jupyter_text>---



This section was created when before I tweaked k-means for better results; I'm leaving it in to demonstrate a different method.

The core of the method below is to group items by their correlation coeffecient. The higher the correlation the more likely the items are to show up together. The other portion of this method is to make sure each item is in only one group, something we can correct with the use of `set` and `frozenset`.<jupyter_code>def create_groups(items, df):
    # Create groups based off item correlations ordered high to low
    groups = dict()
    cats = set()

    for col in items:
        # groups[col] = set(df.corr()[col].sort_values(ascending=False)[0:5].index)
        groups[col] = set(df[col].sort_values(ascending=False)[:5].index)

    for col in items:
        for col2 in items:
            if (groups[col] == groups[col2]) and (col != col2):
                cats.add(frozenset(groups[col]))
    
    return cats
def filter_groups(groups):
    # Cycle through the created groups and remove dupes, output the clean groups
    # and the miscellaneous items that are groupless
    unique_groups = set()
    items = []

    for group in groups:
        temp = set(group)
        for group2 in groups:
            if group != group2:
                temp.difference_update(set(group2))
        unique_groups.add(frozenset(temp))
        items += list(temp)

    return unique_groups, items<jupyter_output><empty_output><jupyter_text>We'll perform two passes with the above functions. The first pass generates 12 ungrouped items. One of the main drawbacks of our k-means groups is a large catch-all group and a lot of single item groups so we want to find a better way to deal with those 12 items. The solution is to do another pass through the two functions. We still end up with some miscellaneous items but overall we end up with better groupings. At this point, you could do some manual corrections of some of the items/groups but overall the groups are fairly coherent.<jupyter_code>unique_groups, items = filter_groups(create_groups(purchases.columns, purchases))
misc = list(set(purchases.columns).difference(set(items)))

misc_groups, items2 = filter_groups(create_groups(misc, purchases[misc]))
misc = list(set(purchases.columns).difference(set(items + items2)))

for group in misc_groups:
    unique_groups.add(group)
unique_groups.add(frozenset(misc))

unique_groups<jupyter_output><empty_output><jupyter_text>Market Basket Analysis Although the prompt did not require it, the dataset seems ripe for association analysis and would be useful for a focus group, especially if there are some unintuitive associations.<jupyter_code>def encode_units(x):
    if x <= 0:
        return 0
    else:
        return 1

encoded_purchases = purchases.applymap(encode_units)

encoded_purchases.head()
# Use apriori to get our rulesets
frequent_itemsets = apriori(encoded_purchases, min_support=0.05,
                            use_colnames=True)
rules = association_rules(frequent_itemsets, metric="lift", min_threshold=1.25)

# Add features so we can filter on number of items in a rule
rules["antecedent_len"] = rules["antecedents"].apply(lambda x: len(x))
rules["consequent_len"] = rules["consequents"].apply(lambda x: len(x))

rules.head()
rules.describe()<jupyter_output><empty_output><jupyter_text>We don't see anything too surprising unfortunately. Support, the relative frequency that a rule shows up, is pretty low across the board. This is likely due to how sparse this data is. Filtering on Confidence and Lift we see some predictable rules along the lines of a couple items in a category leads to another item in the same category. Drinks beget more drinks, bread begets more bread, etc.<jupyter_code>rules[(rules['confidence'] >= .5)
    & (rules['lift'] >= 1.9)].sort_values('lift', ascending=False)[:5]<jupyter_output><empty_output><jupyter_text>Lift is the ratio of the observed support to that expected if the two rules were independent. A lift value close to 1 means the rules were completely independent; the higher the lift the more interesting the rule. Unfortunately we don't see too much lift and what we do see is fairly predictable. There is still value though, as we can use it to craft sales such as "buy any 2 items from a set and get a third item at discount" or simply have a "Frequently Bought Together" section.<jupyter_code>rules[rules['lift'] > 2.3]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08311926605504587, 'fraction_numerical': 0.007064220183486239, 'mean_word_length': 4.359390363815143, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '208265', 'license': 'MIT', 'n_tokens_mistral': 3239, 'n_tokens_neox': 3099, 'n_words': 1192, 'path': '09. Purchase_History_and_Market_Basket_Analysis.ipynb', 'repo_name': 'bundickm/Data-Masked-Case-Studies', 'size': '10900'}