<jupyter_start><jupyter_text>Tutorial 2: Association MiningIn the first tutorial we looked at how Python can be used to do basic data exploration. I hope it was useful to you-- Python is one of the most in-demand skills for data scientists. There are some good reasons for this. The first one is that relying only on software will restrict you to the bounds of that software. By being able to program, you are not as restricted, and can do things that require complex logic.The second reason why a data scientist would want to learn Python is more theoretical. The Python programming language is designed to be highly portable, and as a result is one of the most versatile programming languages. Python hosts a large number of machine learning tools such as scikit-learn (machine learning), pandas (big data processing) and nltk (natural language processing), which do a lot of heavy lifting. Many of these features open new possibilities. We will explore one of them today by writing a simple association mining script using the same dataset as before. About the Aipriori AlgorithmThe apriori association mining algorithm is a simple frequency item set mining algorithm, most commonly used to find associations over a transactional database. In the 1990s and early 2000s, this algorithm was used to discover a lot of data mining "gems" -- unexpected associations between purchases and behaviours. Today it is still commonly used by retailers and e-commerce companies to look at purchasing patterns. The algorithm looks at all of the transactions in a dataset and returns the most common associations between the items. It uses three measures: support, confidence and lift. Support describes the number of occurrences in the dataset. Confidence is how often the association is found to be true. Lift is the ratio of observed support to the independent observations of the items in the association. The wikipedia page gives a more detailed explanation: https://en.wikipedia.org/wiki/Association_rule_learning Installing the Apyori LibraryBefore we can get started, we will need to install the apyroi library. Yes, you read that right... aPYori. Python libraries often have letters "py" instead of "pi", just so you know that it is a PYthon library. If you haven't already, please type the following command in your Anaconda shell: pip install apyoriThis will install the necessary library. After that, we can get started as we did before by loading the dataset. Once again, we will load the csv file content and save it into the list ec.<jupyter_code>import csv #import the csv library for loading csv type files
from apyori import apriori #import the apriori library

ec = [] # we will declare a list to store the data

with open('data.csv', newline='') as csvfile: #this is an expression used by the csv library to open the file
    datareader = csv.reader(csvfile, delimiter=',', quotechar='|') #use the reader to read each row of the csv file and save as a list
    for row in datareader: #this is a loop for iterating through each row
        ec.append(row) #we will append the content of each row as row entry to the ec list<jupyter_output><empty_output><jupyter_text>There are two main differences from last time. You will notice that we imported the apyori library at the top of the script. This is how we bring outside libraries into our Python environment. Also, unlike last time, we will use the regular reader, instead of a DictReader. The apyori library prefers lists, instead of dictionaries. Using the regular csv reader, we will get results that are lists of lists. The practical difference for us is that we will have to select list items by specifying their numbers, instead of their key like last time.As before, it's usually a good practice to see what the string will print. If we run the following cell, we will get the output for the dataset that we loaded.<jupyter_code>print(ec[4]) #prints the fifth item<jupyter_output><empty_output><jupyter_text>Using the apyori library is simple. We simply call then apriori function on the dataset and it will create a bunch of associations for us. Let's do that, while saving the associations as the "associations" list. Let's print it to see what the associations look like.<jupyter_code>associations = list(apriori(ec)) #use the apriori function to create associations

print(associations) #print the associations<jupyter_output><empty_output><jupyter_text>Wait! This is not readable at all!. By simply printing the list, we retrieved direct output from the associations. We can't use this on any sort of report! That said, we can sort of make out the details of the output. You can see the associations, support and confidence values. These are saved as lists within the list of associations. One way to manage this data is to only print the elements that we need, one by one. Let's loop through the results and only print the relevant data.<jupyter_code>for a in associations: #loop through each association directly
    #corresponds to the list values for association, the support and the confidence.
    print("Association: " + str(list(a[0])) + ", Support: " + str(a[1]) + ", Confidence: " + str(a[2][0][2]))<jupyter_output><empty_output><jupyter_text>That's better. However, it also revealed another problem. We only have 10 association rules.This was caused by the apriori functions default values. By default, it only collects support above 0.1. This is far too low for our dataset, as there is a lot of variance in the purchases. We can override the default by stating it in the function. Let's try lowering it to 0.002 ... patterns that happen at least 1000 times in our dataset (warning: this may take about a minute to process on some computers).<jupyter_code>associations = list(apriori(ec, min_support=0.002)) #min support changed to 0.002
print(len(associations))<jupyter_output><empty_output><jupyter_text>We can once again look through these associations by looping and printing. Let's do that.<jupyter_code>for a in associations:
    print("Association: " + str(list(a[0])) + ", Support: " + str(a[1]) + ", Confidence: " + str(a[2][0][2]))<jupyter_output><empty_output><jupyter_text>Better still. If you scroll down, we see some interesting patterns start to emerge, such as:    'United Kingdom', 'ASSORTED COLOUR BIRD ORNAMENT'    'United Kingdom', 'WHITE HANGING HEART T-LIGHT HOLDER'    'United Kingdom', '4.95', '47566', 'PARTY BUNTING'Though interesting that some products are purchased from the UK at a certain price, it is clear that there is something missing. Typically, basket analysis is conducted on baskets. We seem to have output that is improperly structured for this task.One option is to restructure the data. Fortunately, we are using a programming language, so this step is relatively simple. We can create a simple script that creates a new list of baskets for association mining. The script below does this and finishes by reporting the number of basket associations produced.<jupyter_code>i = 1
basket = [] #specify a list for basket items
baskets = [] #a list of baskets
invoice = ec[1][0] #a placeholder for the previous invoice number

while i < len(ec): #iterate through each item sequentially
    if ec[i][0] != invoice: #if this is a new invoice
        baskets.append(basket) #append the baskets
        basket = [] #start a new basket
    basket.append(ec[i][2]) #add this item to basket
    invoice = ec[i][0] #specify this current invoice as the new previous invoice
    i += 1 #iterate to the next item

len(baskets)<jupyter_output><empty_output><jupyter_text>As before, we can use the apriori function to create the associations and the print statement to print the results. Let's only print results that also have a confidence of greater than 50%, meaning that at least half of the time, these things are purchased together.<jupyter_code>associations = list(apriori(baskets, min_support=0.02))

for a in associations:
    if a[2][0][2] > 0.5:
        print("Association: " + str(list(a[0])) + ", Support: " + str(a[1]) + ", Confidence: " + str(a[2][0][2]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06206811451135242, 'fraction_numerical': 0.008637709772951628, 'mean_word_length': 4.74822695035461, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '557249', 'license': 'Unlicense', 'n_tokens_mistral': 2134, 'n_tokens_neox': 2032, 'n_words': 1174, 'path': 'tutorial-2-associate.ipynb', 'repo_name': 'cdconrad/ec-tutorials', 'size': '8104'}