<jupyter_start><jupyter_text>Author - <NAME> Dataset - Instacart The Dataset Overview - Instacart is a same-day delivery and pick up service that allows consumers to shop through the company’s mobile app or website from selected grocery stores, such as Trader Joe’s, Costco and Fairway.The dataset is anonymized and contains a sample of over 3 million grocery orders from more than 200,000 Instacart users. For each user, the dataset provides between 4 and 100 of their orders, with the sequence of products purchased in each order. The dataset also provides the week and hour of day the order was placed and relative measure of time between orders.So let's start by importing the dependencies. NOTE  - : The Order product Dataframe is the train dataset which we will be using for training.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
# Importing the libraries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
# Loading the data-frame
aisles = pd.read_csv(r'/content/sample_data/aisles.csv')
department = pd.read_csv(r'/content/sample_data/departments.csv')
order_products = pd.read_csv(r'/content/sample_data/order_products_train.csv')
orders = pd.read_csv(r'/content/sample_data/orders.csv')
products = pd.read_csv(r'/content/sample_data/products.csv')
# Quick-over-view
aisles.head()

department.head()

order_products.head()

orders.head()

products.head()
#Check the null-values in each set of data
aisles.isnull().sum()
department.isnull().sum()
order_products.isnull().sum()
orders.isnull().sum()
products.isnull().sum()<jupyter_output><empty_output><jupyter_text>Hence, above following code shows that there's no null values. Hurray!<jupyter_code># Let's make the data informative by using info() method to each of the CSV File
aisles.info()
department.info()
order_products.info()
orders.info()
products.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 49688 entries, 0 to 49687
Data columns (total 4 columns):
 #   Column         Non-Null Count  Dtype 
---  ------         --------------  ----- 
 0   product_id     49688 non-null  int64 
 1   product_name   49688 non-null  object
 2   aisle_id       49688 non-null  int64 
 3   department_id  49688 non-null  int64 
dtypes: int64(3), object(1)
memory usage: 1.5+ MB<jupyter_text>Hence, we are lucky enough to get all non-null values and will segregate the columns nowWill now Merge Products and Departments on Common Department id. From the code below we will look deep into the orders dataframe thathow eval set is used for training, testing and prior.We know that the dataset has 4 to 100 orders of a customer are given and here we need to predict the products that are re-ordered.The column eval_set tells us that which category the order belongs to from the three sub-datasets (prior, train and test).<jupyter_code>cnt = orders.eval_set.value_counts()
plt.figure(figsize=(12,8))
sns.barplot(cnt.index, cnt.values, alpha=0.8, color='g')
plt.ylabel('Occurences', fontsize=12)
plt.xlabel('Eval set type', fontsize=12)
plt.title('Count of rows in each dataset', fontsize=15)
plt.xticks(rotation='vertical')
plt.show()
def unique_count(x):
    return len(np.unique(x))
cnt_srs = orders.groupby('eval_set')["user_id"].aggregate(unique_count)
cnt_srs<jupyter_output><empty_output><jupyter_text>Here we can see that there are 206209 customers in total out of which Train contains 131209 and Test contains 75000. Let's Explore more by Analysis now!<jupyter_code>cnt_srs = orders.groupby('user_id')["order_number"].aggregate(np.max).reset_index() # Order number is the name, we pass the name in such paramets
cnt_srs= cnt_srs.order_number.value_counts()
plt.figure(figsize=(12,8))
sns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8, color='r')
plt.ylabel('Occurences', fontsize=12)
plt.xlabel('Order Number', fontsize=12)
plt.xticks(rotation='vertical')
plt.show()<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Hence through the graph we can see that the minimum orders starts from 4 and ends to 100.Let's take a look how ordering changes in week.<jupyter_code>plt.figure(figsize=(12,8))
sns.countplot(x='order_dow', data=orders, color='blue')
plt.ylabel('Count', fontsize=12)
plt.xlabel('Days of Week', fontsize=12)
plt.xticks(rotation='vertical')
plt.title('Frequency of Orders in Week')
plt.show()<jupyter_output><empty_output><jupyter_text>Seems 0 and 1 are the Weekends as the orders are high as compared to others.Now let's explore how it changes with the times of the day<jupyter_code>plt.figure(figsize=(12,8))
sns.countplot(x='order_hour_of_day', data=orders, color='yellow')
plt.ylabel('Count', fontsize=12)
plt.xlabel('Hours of Day', fontsize=12)
plt.xticks(rotation='vertical')
plt.title('Frequency of Orders in Hour')
plt.show()<jupyter_output><empty_output><jupyter_text>Here we can explore that the orders went on increasing from morning 7am to and was on high till 8pm as this time the customers were active online.Now we will combine both to see the distribution. The Order for 24 x 7 days are shown below we can see 0th day 3pm to 4 pm was the max ordered.<jupyter_code>grouped = orders.groupby(['order_dow','order_hour_of_day'])['order_number'].aggregate('count').reset_index()
grouped = grouped.pivot('order_dow','order_hour_of_day','order_number')

plt.figure(figsize=(12,6))
sns.heatmap(grouped)
plt.title('Frequency of days of week vs Hour of days')
plt.show()<jupyter_output><empty_output><jupyter_text>We will check the time intervals between the orders now.<jupyter_code>plt.figure(figsize=(12,8))
sns.countplot(x='days_since_prior_order', data=orders, color='k')
plt.ylabel('Count', fontsize=12)
plt.xlabel('Days since prior order', fontsize=12)
plt.xticks(rotation='vertical')
plt.title('Frequency distribution by days since prior')
plt.show()<jupyter_output><empty_output><jupyter_text>By the above graph we can see that on 7th day and 30th day ordser were placed the most. The smaller peaks on 14 to 29 indicates that the order placed were less on those respective days.Our aim is to get the re-order and check out the re-order percentage.<jupyter_code>order_products.reordered.sum()/order_products.shape[0]<jupyter_output><empty_output><jupyter_text>Hence on average its like 60% of people re-ordered.There is also a situation where none of the products are reordered let's check it.<jupyter_code>order_products.head(2)
grouped = order_products.groupby('order_id')['reordered'].aggregate('sum').reset_index()
grouped['reordered'].iloc[grouped['reordered']>1] = 1
grouped.reordered.value_counts()/ grouped.shape[0]<jupyter_output><empty_output><jupyter_text>By this we can see that 93% were reordered and rest 6.5% were not reorderd.<jupyter_code>grouped = order_products.groupby('order_id')['add_to_cart_order'].aggregate('max').reset_index()
grouped = grouped.add_to_cart_order.value_counts()

plt.figure(figsize=(12,6))
sns.barplot(grouped.index, grouped.values, alpha=0.8)
plt.ylabel('Occurences', fontsize=12)
plt.xlabel('Number of Products in Given Order', fontsize=12)
plt.xticks(rotation='vertical')
plt.show()
data = pd.merge(order_products, products)
data.tail()
data = pd.merge(data, aisles, on= 'aisle_id')
data.head()
data = pd.merge(data, department, on= 'department_id')
data.head()
data.isnull().sum()
# No null values<jupyter_output><empty_output><jupyter_text>Let's Check the frequency count on the product name<jupyter_code>count = data['product_name'].value_counts().reset_index().head(20)
count.columns = ['product_name','frequebcy_count']
count<jupyter_output><empty_output><jupyter_text>By the above chart we can see how much count is present for each product, hence bana has the most and red onion the least.<jupyter_code>count= data['aisle'].value_counts().head(20)
plt.figure(figsize=(12,8))
sns.barplot(count.index, count.values, alpha =0.8, color = 'lightblue')
plt.ylabel('Occurence', fontsize=12)
plt.xlabel('Aisle', fontsize=12)
plt.xticks(rotation='vertical')
plt.show()<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>By the graph above we can see that how many occurence of items are present on the data which we have just created by merging.Most order Aisle's are fresh fruits and fresh vegetables.Now let's create a pie chart for the department level distribution Department Chart<jupyter_code>plt.figure(figsize=(10,10))
series = data['department'].value_counts()
labels = (np.array(series.index))
size = (np.array((series/series.sum())*100))
plt.pie(size,labels=labels, autopct='%1.1f%%', startangle=200)
plt.title('Department Distribution', fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>RFM Modelling (Recency, Frequency and Monetary)RFM is a method used for analyzing customer value. It is commonly used in database marketing and direct marketing and has received particular attention in retail and professional services industries.RFM stands for the three dimensions:Recency – How recently did the customer purchase?Frequency – How often do they purchase?Monetary Value – How much do they spend?Customer purchases may be represented by a table with columns for the customer name, date of purchase and purchase value. One approach to RFM is to assign a score for each dimension on a scale from 1 to 10.![Incontent_image.png](https://drive.google.com/uc?export=view&id=1TekkkLigNdISwrf-UOuHUB_JHMptGkqp)  Formulas Recency = Latest date - Last Invoice Date Frequency = Count of invoice number of transaction(s) Monetary = Sum of Total<jupyter_code>data = pd.merge(data, orders, on= 'order_id')
data.head()
index = data.index
index 
data = data[data.index<5999]
data.shape
invoice = pd.read_csv(r'/content/sample_data/InvoiceDates.csv')
invoice.head()
invoice.shape
invoice = invoice[invoice.index<5999]
invoice.shape
data = pd.merge(data, invoice, left_index=False, right_index= False)
data.head()
Amount = pd.read_csv(r'/content/sample_data/Amount.csv')
Amount.shape
data = pd.merge(data, Amount, left_index=True, right_index= True)
data.head()
data.shape
data['invoicedates'] = pd.to_datetime(data['invoicedates'])
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 5999 entries, 0 to 5998
Data columns (total 19 columns):
 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   order_id                5999 non-null   int64         
 1   product_id              5999 non-null   int64         
 2   add_to_cart_order       5999 non-null   int64         
 3   reordered               5999 non-null   int64         
 4   product_name            5999 non-null   object        
 5   aisle_id                5999 non-null   int64         
 6   department_id           5999 non-null   int64         
 7   aisle                   5999 non-null   object        
 8   department              5999 non-null   object        
 9   user_id                 5999 non-null   int64         
 10  eval_set                5999 non-null   object        
 11  order_number            5999 non-null   int64         
 12  order_dow               5999 non-null   float64 [...]<jupyter_text>We have completely preprocessed the data and added the Invoice dates column which was a major disadvantage of our dataset provided, Hence now we can create RFM Model. Hence we need to create a RFM model it will unfortunately require the date-time module.The date time module will help to set a threashold and help us to know that how recently the customer has placed his/her order. So let's get started.<jupyter_code>import datetime as dt
# Setting the threashold as latest date at the pont where we are intrested caltulating the RFM

Latest_Date = dt.datetime(2020,12,31).date()
#Latest_Date = Latest_Date.strftime('%d/%m/%Y')
#print(Latest_Date)
data = data.rename(columns={'user_id':'Customer_ID'})
# Creating the modelling

RFM = data.groupby('Customer_ID').agg({'invoicedates': lambda x :(Latest_Date - x.max().date()).days,'order_id': lambda x:len(x), 'Amount': lambda x : x.sum()})

# Converting Invoice Dates to integer time so we can do Mathematical Operations
RFM['invoicedates'] = RFM['invoicedates'].astype(int)

# renaming the columns
RFM.rename(columns = {
    'invoicedates' : 'Recency',
                   'order_id' : 'Frequency',
                     'Amount' : 'Monetary'}, inplace=True)

RFM.reset_index().head()<jupyter_output><empty_output><jupyter_text>We can see in the output of above code snippet that customer 146 has bought recently 683 products, we can say that there was a discount going arounnd that time.The Description of RFM is shown below, The statictics data of RFM Modelling.<jupyter_code>RFM.describe()<jupyter_output><empty_output><jupyter_text>Here we will create the Quantiles as to subdivide the data into 4 groups into 4 groups based on recency, frequency and monetary, We can choose any quantiles values as per the business requirements.<jupyter_code># Splitting into 4 Quantiles Segments
quantiles = RFM.quantile(q=[0.25,0.5,0.75])
quantiles = quantiles.to_dict()
quantiles<jupyter_output><empty_output><jupyter_text>Now we will create a Funtion to create the R F and M segmentsNote -: Low value of R, Better the performance of customerwhile for F and M Higher the value, better is the customer. Hence, we can see below has low values for low quantilesin RScore.For FnM Score low value for Quantiles will have high values.<jupyter_code>def RScore(x,p,d):
  if x <= d[p][0.25]:
    return 1
  elif x <= d[p][0.50]:
    return 2
  elif x<= d[p][0.75]:
    return 3
  else : 
    return 4

def FMScore(x,p,d):
  if x <= d[p][0.25]:
    return 4
  elif x <= d[p][0.50]:
    return 3
  elif x <= d[p][0.75]:
    return 2
  else:
    return 1<jupyter_output><empty_output><jupyter_text>Here we are giving the values of R F and M in terms of 1,2,3 and 4 based on their performances and merging them to the dataframe.<jupyter_code>RFM['R'] = RFM['Recency'].apply(RScore, args=('Recency', quantiles,))
RFM['F'] = RFM['Frequency'].apply(FMScore, args=('Frequency', quantiles,))
RFM['M'] = RFM['Monetary'].apply(FMScore, args=('Monetary', quantiles,))
RFM.head()<jupyter_output><empty_output><jupyter_text>Here, we are first creating a group of R F and M by combining them together by applying concat function of string.Secondly, we are adding all R, F and M Score to create a combined score of all these three segments.<jupyter_code># Calculating and adding RFM Group value of combined R F and M
RFM['RFMGroup'] = RFM.R.map(str) + RFM.F.map(str)+ RFM.M.map(str)

# Calculating combined Scores

RFM['RFMScore'] = RFM[['R','F','M']].sum(axis=1)
RFM.head()<jupyter_output><empty_output><jupyter_text>We will assign the loyalty level of the customers now using the below condditions.1.   Best by Platimun2.   Better by Gold3.   Good by Silver4.   Fine by BronzeThe pd.qcut() function has the first args as the dataframe, second is Q cut which  represents the bins with same number of records, for each bin, third is labels which in our case are the above labels.<jupyter_code>Loyalty = ['Platinum','Gold','Silver','Bronze']
Scorecut = pd.qcut(RFM.RFMScore, q=4, labels=Loyalty)
RFM['Loyalty'] = Scorecut.values
RFM.reset_index().head()
RFM.shape<jupyter_output><empty_output><jupyter_text>**Feature Selection to Apply on the Model**Here we are selecting the features for x and y which we are going to fit into our model and map the accuracy of our Model.Model which we are going to use are:1. K-NN2. K-Means3. SVMHence, we need to selectthe features and their mappings and convert it into arra, as array well-fit on the models.<jupyter_code>x = np.asarray(RFM[['Recency','Frequency','Monetary']])
y = np.asarray(RFM['Loyalty'])<jupyter_output><empty_output><jupyter_text>Using the Trqain Test Split to split the data into trqaining set and testing set. Using the Sklearn Library.<jupyter_code>from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(x,y, test_size=0.2, random_state=4)
print(xtrain.shape)
print(ytrain.shape)
print(xtest.shape)
print(ytest.shape)<jupyter_output>(208, 3)
(208,)
(53, 3)
(53,)<jupyter_text>**K-Nearest Neighbour**K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on Supervised Learning technique.K-NN algorithm assumes the similarity between the new case/data and available cases and put the new case into the category that is most similar to the available categories.K-NN algorithm stores all the available data and classifies a new data point based on the similarity. This means when new data appears then it can be easily classified into a well suite category by using K- NN algorithm.K-NN algorithm can be used for Regression as well as for Classification but mostly it is used for the Classification problems.![KNN.png](https://drive.google.com/uc?export=view&id=1AFUZAFU48AuZuQG0SvZCex0Gkd6YrOC5)Firstly, we will choose the number of neighbors, so we will choose the k=5.Next, we will calculate the Euclidean distance between the data points. The Euclidean distance is the distance between two points, which we have already studied in geometry.By calculating the Euclidean distance we got the nearest neighbors, as three nearest neighbors in category A and two nearest neighbors in category B.![distance.png](https://drive.google.com/uc?export=view&id=1TNg6gC6EfOrWNgVC-Mn9J7Dl4Aw6UI0_)**Fitting K-NN classifier to the Training data:**Now we will fit the K-NN classifier to the training data. To do this we will import the KNeighborsClassifier class of Sklearn Neighbors library. After importing the class, we will create the Classifier object of the class. The Parameter of this class will ben_neighbors: To define the required neighbors of the algorithm. Usually, it takes 5.metric='minkowski': This is the default parameter and it decides the distance between the points.p=2: It is equivalent to the standard Euclidean metric.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

KNN_Classifier = KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2)
KNN_Classifier.fit(xtrain,ytrain)
Knnypred = KNN_Classifier.predict(xtest)<jupyter_output><empty_output><jupyter_text>**Support Vector Machines**Support vector machines so called as SVM is a supervised learning algorithm which can be used for classification and regression problems as support vector classification (SVC) and support vector regression (SVR). It is used for smaller dataset as it takes too long to process. In this set, we will be focusing on SVC.The hyperplane is a function which is used to differentiate between features. In 2-D, the function used to classify between features is a line whereas, the function used to classify the features in a 3-D is called as a plane similarly the function which classifies the point in higher dimension is called as a hyperplane. ![SVM.png](https://drive.google.com/uc?export=view&id=15eJKaEg_wVrMZ1BUg5D9V1fB5Tjyay5J) Formula for Support Vector Machinesare being Shown Below -: ![Formula.png](https://drive.google.com/uc?export=view&id=1wgTGkzEPHosoWoAIBPbzzOQnJdOFg3HT)<jupyter_code>from sklearn import svm
classifier = svm.SVC(kernel = 'linear', gamma='auto', C=2)
classifier.fit(xtrain,ytrain)
ypredict = classifier.predict(xtest)<jupyter_output><empty_output><jupyter_text>Evaluation - :Classification report is already a function in sklearn inside metrics which predicts out theConfusion Matrix the Formulas Confusion Matrix is also shown below -:![Confusion Matrix.png](https://drive.google.com/uc?export=view&id=1wQFjihkZ7z1rLmWceNu59c9-if9yz_bu)<jupyter_code>print("Confusion Matrix For K-Nearest Neighbors")
from sklearn.metrics import classification_report
print(classification_report(ytest, Knnypred))
print("Confusion Matrix For Support Vector Machines")
from sklearn.metrics import classification_report
print(classification_report(ytest, ypredict))<jupyter_output>Confusion Matrix For Support Vector Machines
              precision    recall  f1-score   support

      Bronze       0.89      0.89      0.89         9
        Gold       0.58      0.88      0.70         8
    Platinum       1.00      0.82      0.90        17
      Silver       0.89      0.84      0.86        19

    accuracy                           0.85        53
   macro avg       0.84      0.86      0.84        53
weighted avg       0.88      0.85      0.86        53
{'fraction_non_alphanumeric': 0.09672796751850968, 'fraction_numerical': 0.024170050155242418, 'mean_word_length': 4.7170944838885855, 'pattern_counts': {'":': 0, '<': 114, '<?xml version=': 0, '>': 107, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '662336', 'license': 'MIT', 'n_tokens_mistral': 6885, 'n_tokens_neox': 6405, 'n_words': 2208, 'path': 'Instacart- RFM Modelling/Instacart_Market_Analysis.ipynb', 'repo_name': 'eaglewarrior/Machine-learning', 'size': '20943'}