<jupyter_start><jupyter_text><NAME> Data Analysis This is a notebook where unsupervised learning processes such as Dimensionality Reduction, Feature Selection, association analysis and anomaly detection will be implemented so as to help boost the sales of the company.   Load Required Libraries<jupyter_code>library(tidyverse)
# View available libraries
library()
# Load Tsne package
install.packages("Rtsne")
# Caret package
install.packages('caret')
#Corrplot package
install.packages('corrplot')
# Installing the wrapper packages
install.packages('clustvarsel')
install.packages('mclust')
# Arules package for association analysis
install.packages('arules')
# Anomalize package for anomaly detection
install.packages('anomalize')
install.packages("tibbletime")<jupyter_output>Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)<jupyter_text>Loading the data<jupyter_code># Load the first data set for dimension Reduction and feature selection
df1 <- read.csv('/content/Supermarket_Dataset_1 - Sales Data.csv')
head(df1)
# Check for no of rows vs columns

print(nrow(df1))
print(ncol(df1))<jupyter_output>[1] 1000
[1] 16<jupyter_text>The dataset has 1000 rows of data with 16 features * We will go right into Dimensionality Reduction, due to time constraints.  Dimensionality Reduction  Will use t-SNE to reduce the dimensions of the data to a low dimensional space. This is so as PCA cannot retain non-linear variance. Our data consistes of different datatypes and scales, most of which are non linear, and will hopefully be handled by t-SNE.<jupyter_code># Load the t-SNE library
library(Rtsne)
# Load the data
df_tsne <- df1

# Separate the label for plotting

label <- df_tsne$Branch
unique(label)

# Change the label to factor datatype and assign colors

label <- as.factor(label)

print(class(label))

# Assign colors
colors <- rainbow(length(unique(label)))

names(colors) <- unique(label)

print(colors)
# Specify features to be used and proceed in encoding

df_tsne2 <- df_tsne[, c(2:8, 11, 12, 14, 15, 16)]

head(df_tsne2)
# Will proceed to hot encode the categorical columns then apply tsne

# Will try to use caret
library(caret)
# We will use the DummyVars function

dummys <- dummyVars('~.', data=df_tsne2)
df_encoded <- data.frame(predict(dummys, newdata = df_tsne2))

# Load first 5 rows
head(df_encoded)
# Ok the data is encoded so lets reduce the features that were actually increased after hot encoding using dummyVars

# Will use the Rtsne library

tsne <- Rtsne(df_encoded, dims=2, perplexity = 30, verbose = TRUE, max_iter = 500)<jupyter_output>Performing PCA
Read the 1000 x 23 data matrix successfully!
OpenMP is working. 1 threads.
Using no_dims = 2, perplexity = 30.000000, and theta = 0.500000
Computing input similarities...
Building tree...
Done in 0.08 seconds (sparsity = 0.101260)!
Learning embedding...
Iteration 50: error is 59.528978 (50 iterations in 0.12 seconds)
Iteration 100: error is 52.508549 (50 iterations in 0.09 seconds)
Iteration 150: error is 51.352059 (50 iterations in 0.09 seconds)
Iteration 200: error is 50.964357 (50 iterations in 0.09 seconds)
Iteration 250: error is 50.658918 (50 iterations in 0.09 seconds)
Iteration 300: error is 0.561364 (50 iterations in 0.10 seconds)
Iteration 350: error is 0.394508 (50 iterations in 0.10 seconds)
Iteration 400: error is 0.363204 (50 iterations in 0.09 seconds)
Iteration 450: error is 0.353808 (50 iterations in 0.10 seconds)
Iteration 500: error is 0.344765 (50 iterations in 0.10 seconds)
Fitting performed in 0.96 seconds.<jupyter_text>The model is build on an execution time of 0.96 secods, thats way faster than when done on python<jupyter_code># Vizualize the dimensions
plot(tsne$Y, t='n', main='Low Dimension Data after t-SNE')
text(tsne$Y)<jupyter_output><empty_output><jupyter_text>Attempted to reduce the features and tSNE is successful. The 21 features have been reduced to two.However, clustering of the data is an issue as there is no outlined label. Attempted to use ranch of the supermarkets and here are the results.<jupyter_code># Attempting to cluster the dimensions according to the Branch of the supermarket
plot(tsne$Y, t='n', main='Low Dimension Data after t-SNE')
text(tsne$Y, labels = label, col = colors)<jupyter_output><empty_output><jupyter_text>Feature Selection When performing Machine Lerning or Data Analysis in General, some features have weight more than others, and feature selection techniques come in handy to aid us pick out those features for more accurate models and results.<jupyter_code># View our data

# Will use the encoded data created in the t-SNE section
df_fs <- df_encoded

head(df_fs)<jupyter_output><empty_output><jupyter_text>Will use filter method and wrapper method then compare the results  1. Filter Method<jupyter_code># Load corrplot library
library(corrplot)
## The data will be encoded via label encoding this time to avoid adding unneccessary features with hot encoding
# Branch

# Change the column to factor then as numeric
df_fs2 <- df_tsne2

df_fs2$Branch <- factor(df_fs2$Branch)
df_fs2$Branch <- as.numeric(df_fs2$Branch)

# Same for gender, customer type, product line and payment

# Customer type
df_fs2$Customer.type <- factor(df_fs2$Customer.type)
df_fs2$Customer.type <- as.numeric(df_fs2$Customer.type)

# Gender
df_fs2$Gender <- factor(df_fs2$Gender)
df_fs2$Gender <- as.numeric(df_fs2$Gender)

# Product line
df_fs2$Product.line <- factor(df_fs2$Product.line)
df_fs2$Product.line <- as.numeric(df_fs2$Product.line)

#Payment
df_fs2$Payment <- factor(df_fs2$Payment)
df_fs2$Payment <- as.numeric(df_fs2$Payment)

head(df_fs2)
# Find factors with a high correlation. Cutoff set to 0.7
corrmatrix <- cor(df_fs2)

highcorr <- findCorrelation(corrmatrix, cutoff = 0.7)
highcorr

names(df_fs2[, highcorr])<jupyter_output><empty_output><jupyter_text>The factors with high correlation include cogs, total and tax<jupyter_code># Dropping the highly correlated factors from the data

df_fs_clean <- df_fs2[-highcorr]

head(df_fs_clean)
# Original correlation
corrplot(cor(df_fs2), order = 'hclust')
# Vizualize the new correlations

corrplot(cor(df_fs_clean), order = 'hclust')<jupyter_output><empty_output><jupyter_text>The factors have been reduced to 8 from 12, which shows a great improvement in the correlation matrix.  2. Wrapper Method<jupyter_code># Will use the mclust module to have the function tell us what factors to drop. 
# It uses a Greedy Search type of search algorithm.

# Load the necessary libraries

library(clustvarsel)
library(mclust)
# Lets build a greedy search algo

greedy <- clustvarsel(df_fs2, G = 1 : 5)
greedy<jupyter_output><empty_output><jupyter_text>The algorithm has suggested that only two features , (Product Line and Branch) will be used in the model and the others to be rejected.Lets build the cluster model<jupyter_code># Build the mclust model

subset <- df_fs2[, greedy$subset]

model <- Mclust(subset, G = 1 : 5)

# summary of the built model
summary(model)
# The model plotted into clusters
plot(model, c('classification'))<jupyter_output><empty_output><jupyter_text>The clusters have been built, using the two features selected by the greedy algorithm, therefore our reduction is a success!  Association Analysis - Here is where things get interesting. Carrefour provided a separate dataset that had all transactions done by the customers.- The data will be used so as to find the most popular products and analyze the customers behaviour when picking certain products.- Vamos!<jupyter_code># Load the data

# Will use arules library
library(arules)
# Load the data

# Will use read.transactions

trs <- read.transactions('/content/customer purchases.csv', sep = ',')
trs<jupyter_output>Warning message in asMethod(object):
“removing duplicated items in transactions”<jupyter_text>We have 7500 transactions and 119 items<jupyter_code># Lets see 5 transactions
inspect(trs[1:5])
# summary of the transactions
summary(trs)<jupyter_output><empty_output><jupyter_text>We can see most popular item is mineral water, followed by eggs and spaghetti<jupyter_code># Viewing the items
items <- as.data.frame(itemLabels(trs))
colnames(items) <- "Items In the Supermarket"

#Preview 15
head(items, 15)<jupyter_output><empty_output><jupyter_text>Most popular items (support)<jupyter_code># Most 10 most popular items

itemFrequencyPlot(trs, col='blue', topN = 10)
# Items with 10% importance
itemFrequencyPlot(trs, support =0.1)<jupyter_output><empty_output><jupyter_text>These are the most important items for carrefour customers.  developing apriori Rules for comparison purposes<jupyter_code>rules <- apriori(trs, parameter = list(
  support = 0.001,
  confidence = 0.8
))<jupyter_output>Apriori

Parameter specification:
 confidence minval smax arem  aval originalSupport maxtime support minlen
        0.8    0.1    1 none FALSE            TRUE       5   0.001      1
 maxlen target  ext
     10  rules TRUE

Algorithmic control:
 filter tree heap memopt load sort verbose
    0.1 TRUE TRUE  FALSE TRUE    2    TRUE

Absolute minimum support count: 7 

set item appearances ...[0 item(s)] done [0.00s].
set transactions ...[119 item(s), 7501 transaction(s)] done [0.00s].
sorting and recoding items ... [116 item(s)] done [0.00s].
creating transaction tree ... done [0.00s].
checking subsets of size 1 2 3 4 5 6 done [0.01s].
writing ... [74 rule(s)] done [0.00s].
creating S4 object  ... done [0.00s].<jupyter_text>We have 74 rules<jupyter_code># For comparison, lets build another ruled module and compare
rules2 <- apriori(trs, parameter = list(
  support = 0.002,
  confidence = 0.7
))<jupyter_output>Apriori

Parameter specification:
 confidence minval smax arem  aval originalSupport maxtime support minlen
        0.7    0.1    1 none FALSE            TRUE       5   0.002      1
 maxlen target  ext
     10  rules TRUE

Algorithmic control:
 filter tree heap memopt load sort verbose
    0.1 TRUE TRUE  FALSE TRUE    2    TRUE

Absolute minimum support count: 15 

set item appearances ...[0 item(s)] done [0.00s].
set transactions ...[119 item(s), 7501 transaction(s)] done [0.00s].
sorting and recoding items ... [115 item(s)] done [0.00s].
creating transaction tree ... done [0.00s].
checking subsets of size 1 2 3 4 5 done [0.00s].
writing ... [11 rule(s)] done [0.00s].
creating S4 object  ... done [0.00s].<jupyter_text>Here there are 11 rules, therefore not efficient as the parameter's confidence is lowered and the support increased.We will use the first one<jupyter_code># Lets view our rules summary
summary(rules)
# Lets view first 5 rules
inspect(rules[1:5])<jupyter_output>lhs                              rhs             support     confidence
[1] {frozen smoothie, spinach}    => {mineral water} 0.001066524 0.8888889 
[2] {bacon, pancakes}             => {spaghetti}     0.001733102 0.8125000 
[3] {nonfat milk, turkey}         => {mineral water} 0.001199840 0.8181818 
[4] {ground beef, nonfat milk}    => {mineral water} 0.001599787 0.8571429 
[5] {mushroom cream sauce, pasta} => {escalope}      0.002532996 0.9500000 
    coverage    lift      count
[1] 0.001199840  3.729058  8   
[2] 0.002133049  4.666587 13   
[3] 0.001466471  3.432428  9   
[4] 0.001866418  3.595877 12   
[5] 0.002666311 11.976387 19<jupyter_text>In the first rule, if someone buys smoothie and spinach, then there is an 89% chance of the customer buying mineral water  Sorting the rules by confidence<jupyter_code>rules <- sort(rules, by = 'confidence', decreasing = TRUE)
# Lets view rules lower than 100% confidence
inspect(rules[5:9])<jupyter_output>lhs                        rhs                 support confidence    coverage      lift count
[1] {mushroom cream sauce,                                                                       
     pasta}                 => {escalope}      0.002532996  0.9500000 0.002666311 11.976387    19
[2] {red wine,                                                                                   
     soup}                  => {mineral water} 0.001866418  0.9333333 0.001999733  3.915511    14
[3] {eggs,                                                                                       
     mineral water,                                                                              
     pasta}                 => {shrimp}        0.001333156  0.9090909 0.001466471 12.722185    10
[4] {herb & pepper,                                                                              
     mineral water,                                                                              
     rice}          [...]<jupyter_text>There is a 95% chance of a customer buying escalope after buying pasta  Lets imagine Carrefour wants to boost their sales on milk as they have a deal with Brookside.<jupyter_code># Well see the most common items bought before a customer picks milk

milk <- subset(rules, subset = rhs %pin% 'milk')
# sort by confidence
milk <- sort(milk, by = 'confidence', decreasing = TRUE)

# Inspect first 5 commodities

inspect(milk[1 : 5])<jupyter_output>lhs                                    rhs    support     confidence
[1] {cake, meatballs, mineral water}    => {milk} 0.001066524 1.0000000 
[2] {escalope, hot dogs, mineral water} => {milk} 0.001066524 0.8888889 
[3] {meatballs, whole wheat pasta}      => {milk} 0.001333156 0.8333333 
[4] {black tea, frozen smoothie}        => {milk} 0.001199840 0.8181818 
[5] {burgers, ground beef, olive oil}   => {milk} 0.001066524 0.8000000 
    coverage    lift     count
[1] 0.001066524 7.717078  8   
[2] 0.001199840 6.859625  8   
[3] 0.001599787 6.430898 10   
[4] 0.001466471 6.313973  9   
[5] 0.001333156 6.173663  8<jupyter_text>We can see, customers buying cake, meatballs and water have the highest chances of picking milk, with a confidence of 100 %Milk should be placed around the said items. Other items to be considered includes black tea , hot dogs and ground beef<jupyter_code># Well see the most common items bought after a customer picks milk

milk2 <- subset(rules, subset = lhs %pin% 'milk')
# sort by confidence
milk2 <- sort(milk2, by = 'confidence', decreasing = TRUE)

# Inspect first 10 commodities

inspect(milk2[1 : 10])<jupyter_output>lhs                     rhs                     support confidence    coverage      lift count
[1]  {frozen vegetables,                                                                           
      milk,                                                                                        
      spaghetti,                                                                                   
      turkey}             => {mineral water}     0.001199840  0.9000000 0.001333156  3.775671     9
[2]  {cake,                                                                                        
      meatballs,                                                                                   
      milk}               => {mineral water}     0.001066524  0.8888889 0.001199840  3.729058     8
[3]  {burgers,                                                                                     
      milk,                                                                                        
[...]<jupyter_text>Mineral water, spaghetti and shrimp should be placed around milk, to boost the sales of both commodities This can be done on a variety of commodities so as to boost the sales of the items and bring substancial profits to carrefour Kenya.Very interesting.  Anomaly Detection As a Data Scientist, I will proceed to use some techniques so as to identify any inconsistencies within the company's day to day data, also known as anomalies<jupyter_code># Load anomalize library and tibbletime

library(anomalize)
library(tibbletime)
#Load data'
sales3 <- read.csv('/content/sales time series.csv') %>%

#Change date column to date time

mutate(Date = as.Date(Date, format = '%m/%d/%Y')) %>%
# Change to tibble
as_tbl_time(Date) %>%

#sort days per 'daily'
as_period('daily')
# output first 6 recorde
head(sales3)
# Run dataset on anomalizer 
sales3_out <- sales3 %>%
time_decompose(Sales, merge = TRUE) %>%
anomalize(remainder) %>%
time_recompose()

# Plotting the outliers
sales3_out %>%
plot_anomalies()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08338974706135763, 'fraction_numerical': 0.06431164994768909, 'mean_word_length': 3.052369077306733, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 98, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 47, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '502954', 'license': 'Unlicense', 'n_tokens_mistral': 5610, 'n_tokens_neox': 4791, 'n_words': 1768, 'path': 'Carrefour_Data_Analysis.ipynb', 'repo_name': 'mbururyan/Carrefour-KE-Data-Analysis', 'size': '16258'}