<jupyter_start><jupyter_text>Practical B - Association Rule Mining - Groceries DataSet  Introduction In this practical we will perform a market basket analysis of transactional data from a small grocery store (9835 transactions, 169 products).  Import Libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("darkgrid")
sns.set_context("paper")

from itertools import combinations, groupby
from collections import Counter
import sys<jupyter_output><empty_output><jupyter_text>Import the data The data used here was adapted from the Groceries dataset in the Apriori R package, and has been cleaned and simplified already.Note:  * In a large grocery store, hint Insacart, there is a huge variety of items. There might be five brands of milk, a dozen different types of laundry detergent, and three brands of coffee.  * If we can assume that the retailer is not terribly concerned with finding rules that apply only to a specific brand of milk or detergent we could remove all brand names and merge products. This reducesthe number of groceries to a more manageable size, using broad categories such as chicken, frozen meals, margarine, and soda.<jupyter_code># download remote source if local copy is not available
import requests, os
url = "https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/groceries.csv"
dataFile = "data/groceries.csv"
if not os.path.isfile(dataFile):
    r = requests.get(url)
    with open(dataFile, 'wb') as f:  
        f.write(r.content)
# print out the first 5 rows of groceries.csv
for line in open(dataFile).read().split("\n")[:5]:
    print(line)<jupyter_output><empty_output><jupyter_text>These lines indicate five separate grocery store transactions. The first transactionincluded four items: citrus fruit, semi-finished bread, margarine, and ready soups.In comparison, the third transaction included only one item, whole milk.<jupyter_code># read in csv and convert to list of lists
# Note the [:-1] to drop the last empty line in the CSV file

transactions = [line.split(',') for line in open(dataFile).read().split("\n")][:-1]
# print out the first 5 transactions (to compare with first 5 lines of file)
from pprint import pprint
pprint(transactions[:5])<jupyter_output><empty_output><jupyter_text>Transcode transactions into a Data frame<jupyter_code>from mlxtend.preprocessing import TransactionEncoder

te = TransactionEncoder()
te_ary = te.fit(transactions).transform(transactions,sparse=False)

df = pd.DataFrame(te_ary, columns=te.columns_)
df.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>true_count = sum(sum(te_ary)) / 1662115
print ("Number of transactions = {:,} ".format(len(transactions)))
print ("Number of products = {:,} ".format(len(te.columns_)))
print ("Number of (non-unique) items sold = {:,}".format(te_ary.sum()))

print ("Sparseness of transaction database {:.3%}".format(te_ary.sum()/te_ary.size))
print ("Average number of items per transaction = {:.4}".format(te_ary.sum()/len(transactions)))<jupyter_output><empty_output><jupyter_text>Size of Transaction<jupyter_code># sum to get nubmer of True along each row (each transaction)
a = df.apply(lambda row: sum(row), axis=1)
a.head()
# Generate plot of count of the number of transactions of the same size
a.value_counts().plot.bar()
plt.title("Distribution of Tranasaction Size")
plt.xlabel("Number of items")
plt.ylabel("Frequency")
plt.show()
# or just output data   
print(a.value_counts())
# We can generate a set of statistics about the size of transactions. 
a.describe()<jupyter_output><empty_output><jupyter_text>So from above tables and bar plot we see: * A total of 2,159 transactions contained only a single item, while one transaction had 32 items.  * The first quartile and median purchase size are 2 and 3 items respectively, implying that 25 percent of transactions contained two or fewer items and about half contained more or less than three items.  * The mean of 4.409 matches the value we calculated earlier.  Association Rule Analysis  Frequent Itemset Generation<jupyter_code>from mlxtend.frequent_patterns import apriori
a = apriori(df, min_support=0.1,use_colnames=True).sort_values(by='support',ascending=False)
a<jupyter_output><empty_output><jupyter_text>Visualizing item support – item frequency plots<jupyter_code>names = [next(iter(n)) for n in a["itemsets"]]
a.plot(kind='bar', title ="Support for most frequent products")
plt.xticks(range(len(names)), names, rotation=20)
plt.show()
a = apriori(df, min_support=0.05, max_len=1, use_colnames=True) \
    .sort_values(by='support',ascending=False)
    
names = [next(iter(n)) for n in a["itemsets"]]
a.plot.barh(title ="Support for most frequent products")
plt.yticks(range(len(names)), names)
plt.gca().invert_yaxis()
plt.show()<jupyter_output><empty_output><jupyter_text>Visualization of the sparse matrix for the first $k$ transactions<jupyter_code>k = 100
sns.set_style("white")
plt.figure(figsize=(10,5)) 
plt.imshow(1-te_ary[0:k], interpolation='none', cmap='gray')
plt.xlabel("Items (Columns)")
plt.ylabel("Transactions (Rows)")
plt.title("Visualation of first %d transactions" % k)
plt.show()<jupyter_output><empty_output><jupyter_text>The above diagram depicts a matrix with $k$ rows and 169 columns, indicating the $k$ transactions and 169 possible items we requested. Cells in the matrix are filled with black for transactions (rows) where the item (column) was purchased.A few columns seem fairly heavily populated, indicating some very popular items at the store, but overall, the distribution of dots seems fairly random. Given nothing else of note.This visualization can be a useful tool for exploring the data. For one, it may help with the identification of potential data issues. Columns that are filled all the way down could indicate items that are purchased in every transaction-a problem that could arise, perhaps, if a retailer's name or identification number was inadvertently included in the transaction datase.Additionally, patterns in the diagram may help reveal interesting segments of transactions or items, particularly if the data is sorted in interesting ways. For example, if the transactions are sorted by date, patterns in the black dots could reveal seasonal effects in the number or types of items people purchase. Perhaps around Christmas or Hanukkah, toys are more common; around Halloween, perhaps candy becomes popular. This type of visualization could be especially powerful if the items were also sorted into categories. In most cases, however, the plot will look fairly random, like static on an old CRT television screen which is not tuned to a channel. Keep in mind that this visualization will not be as useful for extremely large transaction databases because the cells will be too small to discern. Still, by combining it with sampling, you can view the sparse matrix for a randomly sampled set of transactions.<jupyter_code>frequent_itemsets = apriori(df, min_support=0.1,use_colnames=True)
frequent_itemsets<jupyter_output><empty_output><jupyter_text>Rule Generation<jupyter_code>from mlxtend.frequent_patterns import association_rules

frequent_itemsets = apriori(df, min_support=0.006,use_colnames=True)
rules = association_rules(frequent_itemsets, metric="confidence", min_threshold=0.8)
rules.head()<jupyter_output><empty_output><jupyter_text>So we generated zero rules.If you think about it, this outcome should not have been terribly surprising. With the default support of 0.1, this means that in order to generate a rule, an item must have appeared in at least 0.1 * 9385 = 938.5 transactions. Since only eight items appeared this frequently in our data, it's no wonder we didn't find any rules.One way to approach the problem of setting support is to think about the minimum number of transactions you would need before you would consider a pattern interesting. For instance, you could argue that if an item is purchased twice a day (about 60 times) then it may be worth taking a look at. From there, it is possible tocalculate the support level needed to find only rules matching at least that many transactions. Since 60 out of 9,835 equals 0.006, we'll try setting the support there first.Setting the minimum confidence involves a tricky balance. On one hand, if confidence is too low, then we might be overwhelmed with a large number of unreliable rules—such as dozens of rules indicating items commonly purchased with batteries. How would we know where to target our advertising budget then? On the other hand, if we set confidence too high, then we will be limited to rules that are obvious or inevitable—like the fact that a smoke detector is always purchased in combination with batteries. In this case, moving the smoke detectors closer to the batteries is unlikely to generate additional revenue, since the two items were already almost always purchased together. We'll start with a confidence threshold of 0.25, which means that in order to be included in the results, the rule has to be correct at least 25 percent of the time. This will eliminate the most unreliable rules while allowing some room for us to modify behavior with targeted promotions.<jupyter_code>from mlxtend.frequent_patterns import association_rules

frequent_itemsets = apriori(df, min_support=0.006,use_colnames=True)
rules = association_rules(frequent_itemsets, metric="confidence", min_threshold=0.25)
print ("Generated {:,} rules".format(len(rules)))
rules.head()
# get stats on objective measures
rules[["support","confidence", "lift"]].describe()<jupyter_output><empty_output><jupyter_text>Above, we see summary statistics for the rule quality measures: support, confidence, and lift. Support and confidence should not be very surprising, since we used these as selection criteria for the rules. However, we might be alarmed if most or all of the rules were very near the minimum thresholds—not the case here.<jupyter_code># order rules by lift
rules.sort_values(by='lift',ascending=False).head()<jupyter_output><empty_output><jupyter_text>TODO: Take the result of learning association rules and divide them into three categories: * Actionable * Trivial * Inexplicable<jupyter_code>for col in rules.columns: 
    print(col)<jupyter_output><empty_output><jupyter_text>Taking subsets of association rules (by rule length)<jupyter_code># add new columnn storing the rule length
rules["rule_len"] = rules.apply(lambda row: len(row["antecedants"])+len(row["consequents"]), axis=1)
# get stats on rules groupt by rule length
rules[["rule_len","support", "lift"]].groupby("rule_len").agg(['mean', 'count']).reset_index()
# restrict analysis to rules of length 4 and order rules by lift
rules[rules["rule_len"]==4].sort_values(by='lift',ascending=False).head()<jupyter_output><empty_output><jupyter_text>Taking subsets of association rules (by content)<jupyter_code># restrict analysis to transactions involving berries
df2 = df[df["berries"]==True]
frequent_itemsets = apriori(df2, min_support=0.006,use_colnames=True)
rules = association_rules(frequent_itemsets, metric="confidence", min_threshold=0.25)
rules.head()
rr = rules.apply(lambda row: "berries" in row["antecedants"], axis=1)
a = apriori(df, min_support=0.003,use_colnames=True)
a["itemset_len"] = a.apply(lambda row: len(row["itemsets"]), axis=1)
a.groupby("itemset_len").size().reset_index(name="count")<jupyter_output><empty_output><jupyter_text>Appendix -<jupyter_code># effect of minsupport on distribution of frequent itemsets
sns.set_style("darkgrid")
for minsupport in [0.06, 0.006, 0.003, 0.002, 0.001]:
    a = apriori(df, min_support=minsupport,use_colnames=True)
    a["itemset_len"] = a.apply(lambda row: len(row["itemsets"]), axis=1)
    d = a.groupby("itemset_len").size().reset_index(name="count")
    d["count"].plot.line(label="minsupp = %s" % 0.01)
plt.xticks(range(7))
plt.legend()
plt.xlabel("Size of itemset")
plt.ylabel("Number of Frequent itemsets")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08918941342404381, 'fraction_numerical': 0.012113166846428275, 'mean_word_length': 6.119905493207324, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '162536', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3577, 'n_tokens_neox': 3364, 'n_words': 1368, 'path': 'topics/09-Association_Analysis/21-Lab_09_-_Association_Analysis/files/Practical_B_-_ARM_Groceries_DataSet.ipynb', 'repo_name': 'DataMining-2021S2/live', 'size': '12053'}