<jupyter_start><jupyter_text>Since the dataset is huge(32.4M rows), currently working on small portion of dataset by sampling 50000 users from the base of 200,000 users to achieve computations.<jupyter_code>final_df = pd.merge(sampledprior,products, on = ['product_id','product_id'])
final_df = pd.merge(final_df,orders,on=['order_id','order_id'])
final_df = pd.merge(final_df,aisles,on=['aisle_id','aisle_id'])
final_df = pd.merge(final_df,departments, on=['department_id', 'department_id'])
print(final_df.shape)
final_df.head(10)<jupyter_output>(3183793, 15)<jupyter_text>Quick Overview on products sold and uniqueness.<jupyter_code>#The most sold product is banana followed by bag of organic bananas and so on
final_df['product_name'].value_counts()[0:5]
#Top 5 aisles categories
final_df['aisle'].value_counts()[0:5]
#Top 5 departments categories
final_df['department'].value_counts()[0:5]<jupyter_output><empty_output><jupyter_text>Clustering Want to find a possible clusters among the different customers and substitute single user_id with the cluster to which they are assumed to belong. Hence creating a dataframe with all the purchases made by each user<jupyter_code>customer_prod = pd.crosstab(final_df['user_id'], final_df['department'])
print(customer_prod.shape)
customer_prod.head(10)<jupyter_output>(20000, 21)<jupyter_text>Here we can see that there are approximately 67k unique users and 21 departments Model 1 --> Clustering by reducing the no of dimensions, i.e, PCA<jupyter_code>#Model 1
#Fitting it into 2 components
pca = PCA(n_components=2)
pca.fit(customer_prod)
sample_pca = pca.transform(customer_prod)
reduced_df = pd.DataFrame(sample_pca)
reduced_df.head()
tocluster = pd.DataFrame(reduced_df[[1,0]])
#print(tocluster.shape)
tocluster.head()
plt.figure(figsize=(8,8))
plt.plot(tocluster[1], tocluster[0], 'o', markersize=2, color='blue', alpha=0.5, label='class1')

plt.xlabel('x_values')
plt.ylabel('y_values')
plt.legend()
plt.show()
clusterer = KMeans(n_clusters=21,random_state=1).fit(tocluster)
centers = clusterer.cluster_centers_
c_preds = clusterer.predict(tocluster)

#CLuster centers
centers
c_preds[0:200]<jupyter_output><empty_output><jupyter_text>Here is how our clusters appear<jupyter_code>cluster_segregation = customer_prod.copy()
cluster_segregation['cluster'] = c_preds
cluster_segregation.head(10)<jupyter_output><empty_output><jupyter_text>We have formed the clusters based on departments, i.e, 21 departments. Now we can look into the clust_segregation crosstab for interesting pattern.<jupyter_code>#No of customers in each cluster
cluster_count = []
cluster_names = []
for number in range(len(departments)):
    cluster_count.append(len(cluster_segregation[cluster_segregation['cluster']==number]))
    cluster_names.append('C'+str(number))

cluster_info = pd.DataFrame(cluster_count, cluster_names)
cluster_info.columns = ['Count']
cluster_info.transpose()<jupyter_output><empty_output><jupyter_text>Top 5 departments for each cluster<jupyter_code>top_5 = []
colnames = []
for number in range(len(departments)):
    cluster = cluster_segregation[cluster_segregation['cluster']==number].drop('cluster',axis=1).mean()
    temp = cluster.sort_values(ascending=False)[0:5]
    top_5.append([temp.index[i] for i in range(5)])
    colnames.append('C'+str(number))
table = pd.DataFrame(top_5).transpose()
table.columns = colnames
table<jupyter_output><empty_output><jupyter_text>Analysis of the clusters confirm the initial hypothesis that the products from the departments mentioned below are most generically bought by majority of customers:1. Produce2. Dairy Eggs3. Snacks4. Beverages5. Frozen and pantry<jupyter_code>sample = cluster_segregation.loc[cluster_segregation['cluster'] == 0]
#sample['user_id'].head(5)
#list(sample.rows.values)
userlist = sample.index.tolist()
#row = sample[0:1]
#sample.get_value(1,'user_id')
#cluster_segregation.loc[cluster_segregation['cluster'] == 2].shape
sample1 = sample.reset_index()
sample1.index.name = None
sample1.head()
df1 = orders[orders['user_id'].isin(userlist)]
print(df1.shape)
df2 = df1.loc[df1['eval_set'] == "prior"]
print(df2.shape)
df2.head()
ap = pd.merge(sampledprior,df2, on = ['order_id','order_id'])
print(ap.shape[0])
ap.head()

ordprod = ap[['order_id','product_id']]
print(ordprod[:20])
#print(ordprod.get_value(0,'product_id'))
x = ordprod["order_id"].unique()
#type(temp)
temp = x.tolist()
#print(temp)
big = []
big1 = []
j=0
for i in temp:
    l = []
    k = []
    #print(j)
    while(ordprod.get_value(j,'order_id')==i and j<ap.shape[0]-1):
        l.append(str(ordprod.get_value(j,'product_id')))
        k.append(ordprod.get_value(j,'product_id'))
        j+=1
    #i+=1
    if(l):
        #big.append(tuple(l))
        big.append(l)
        big1.append(k)
big.pop(len(big)-1)
big1.pop(len(big1)-1)
#print(big)
f = open("nf","w")
f.write(str(big1))

f = open("myfile.txt", "w")
#f.write("hello")
#for a,b,c in l:
#    print(str(int(a))+","+str(int(b))+","+str(int(c))+",")
for i in big:
    #for j in i:
    #list1 = [1, 2, 3]
    str1 = ','.join(str(e) for e in i)
    str1 = str1+","
    #print(str1)
    f.write(str1+"\n")
from efficient_apriori import apriori
itemsets, rules = apriori(big, min_support=0.7,  min_confidence=1)
print(rules)
itemsets, rules = apriori(big, min_support=0.007,  min_confidence=0.1)
print(rules)
fp = open("nf","r")
import ast
contents = ast.literal_eval(fp.read())
#print(type(contents))
import pyfpgrowth
patterns = pyfpgrowth.find_frequent_patterns(contents, 0.008)
#print(patterns)
rules = pyfpgrowth.generate_association_rules(patterns, 0.1)
fn = open("rulesfp","w")
fn.write(str(rules))
#print(rules)
transactions1 = [['25931', '35163', '29662', '26283', '47766', '12036', '13497', '44953', '7969', '8518', '18689', '44299', '8381', '32205'], ['35951', '24852', '46206', '25890', '33120', '21903', '40545', '47766'], ['8571', '21292', '32303', '16283']]
transactions2 = [['25913','3563','29662'],['25913','3563','54661'],['29662','3563','54661']]
itemsets, rules = apriori(transactions1, min_support=0.05,  min_confidence=1)
#print(itemsets)
print(rules)
# Import the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

from apyori import apriori
rules = apriori(transactions1, min_support = 0.7, min_confidence = 0.2, min_lift = 3, min_length = 2)
print(list(rules))<jupyter_output><empty_output><jupyter_text>The above functions using built-in efficient apriori function and the classical apriori function don't print any rules because apriori takes exponential time to finish computation<jupyter_code>#FP tree algorithm
class treeNode:
    def __init__(self, nameValue, numOccur, parentNode):
        self.name = nameValue
        self.count = numOccur
        self.nodeLink = None
        self.parent = parentNode      #needs to be updated
        self.children = {} 
#increments the count variable with a given amount    
    def inc(self, numOccur):
        self.count += numOccur
#display tree in text. Useful for debugging        
    def disp(self, ind=1):
        print ('  '*ind, self.name, ' ', self.count)
        for child in self.children.values():
            child.disp(ind+1)  
rootNode = treeNode('pyramid',9,None)
rootNode.children['eye'] = treeNode('eye',13,None)
rootNode.disp()
def createTree(dataSet, minSup=1): #create FP-tree from dataset but don't mine
    headerTable = {}
    #go over dataSet twice
    for trans in dataSet:#first pass counts frequency of occurance
        for item in trans:
            headerTable[item] = headerTable.get(item, 0) + dataSet[trans]
    for k in list(headerTable):  #remove items not meeting minSup
        if headerTable[k] < minSup: 
            del(headerTable[k])
    freqItemSet = set(headerTable.keys())
    #print 'freqItemSet: ',freqItemSet
    if len(freqItemSet) == 0: return None, None  #if no items meet min support -->get out
    for k in headerTable:
        headerTable[k] = [headerTable[k], None] #reformat headerTable to use Node link 
    #print 'headerTable: ',headerTable
    retTree = treeNode('Null Set', 1, None) #create tree
    for tranSet, count in dataSet.items():  #go through dataset 2nd time
        localD = {}
        for item in tranSet:  #put transaction items in order
            if item in freqItemSet:
                localD[item] = headerTable[item][0]
        if len(localD) > 0:
            orderedItems = [v[0] for v in sorted(localD.items(), key=lambda p: p[1], reverse=True)]
            updateTree(orderedItems, retTree, headerTable, count)#populate tree with ordered freq itemset
    return retTree, headerTable #return tree and header table
def updateTree(items, inTree, headerTable, count):
    if items[0] in inTree.children:#check if orderedItems[0] in retTree.children
        inTree.children[items[0]].inc(count) #incrament count
    else:   #add items[0] to inTree.children
        inTree.children[items[0]] = treeNode(items[0], count, inTree)
        if headerTable[items[0]][1] == None: #update header table 
            headerTable[items[0]][1] = inTree.children[items[0]]
        else:
            updateHeader(headerTable[items[0]][1], inTree.children[items[0]])
    if len(items) > 1:#call updateTree() with remaining ordered items
        updateTree(items[1::], inTree.children[items[0]], headerTable, count)
def updateHeader(nodeToTest, targetNode):   #this version does not use recursion
    while (nodeToTest.nodeLink != None):    #Do not use recursion to traverse a linked list!
        nodeToTest = nodeToTest.nodeLink
    nodeToTest.nodeLink = targetNode
def createInitSet(dataSet):
    retDict = {}
    for trans in dataSet:
        retDict[frozenset(trans)] = 1
    return retDict

initSet = createInitSet(big)

initSet
#The FP-tree
#myFPtree, myHeaderTab = createTree(initSet, 1)
#myFPtree.disp()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1380255941499086, 'fraction_numerical': 0.036156814950233596, 'mean_word_length': 5.119950279676818, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '372930', 'license': 'MIT', 'n_tokens_mistral': 3582, 'n_tokens_neox': 3345, 'n_words': 801, 'path': 'Sampling,segmentationandfptree.ipynb', 'repo_name': 'OmkarMetri/Instacart-Market-Basket-Analysis', 'size': '9846'}