<jupyter_start><jupyter_text>Practice Case - Machine Learning PreprocessingName: <NAME>Data Fellowship Batch 6 by IYKRA<jupyter_code>#import data

import pandas as pd
import seaborn as sns

path = 'C:/Users/Alifia/Downloads/ML_2_Fellow/ML_2_Fellow.csv'

data = pd.read_csv(path, delimiter=',', encoding= 'unicode_escape')

print("Rows: %d and Columns: %d \n" %data.shape)
data.head(5)<jupyter_output>Rows: 541909 and Columns: 8<jupyter_text>Data Processing  Check data values in each columns Kode_Bayar<jupyter_code>#check data values
data.Kode_Bayar.value_counts()<jupyter_output><empty_output><jupyter_text>There was an error when importing the csv files in Kode_Bayer. They still had commas which make the values did not turn into columns. Therefore, I select this values and manualy change it in ms. excel to make it correct.<jupyter_code>#create new data except tagihan kolom dan No. kolom
error = data[data['Kode_Barang'].isnull()]
error = pd.DataFrame(error)
error.head()
#save the error data
error.to_csv(r'C:/Users/Alifia/Downloads/error_data.csv', index = False)
#drop error data
data = data.dropna(subset=['Kode_Barang'])
data
#import cleaned error data and concat it to the data
er_df = pd.read_csv(r'C:/Users/Alifia/Downloads/cleaned_er_data.csv', encoding= 'unicode_escape')
er_df.head()
df = pd.concat([data,er_df],ignore_index=True)
print("Rows: %d and Columns: %d \n" %df.shape)
df.sort_values('Kode_Bayar')
df.head()
#check data values
df.Kode_Bayar.value_counts()<jupyter_output><empty_output><jupyter_text>There was no more issues with the commas in Kode_Bayar column  Removed cancel transaction in the data, it was indicated by "C" in front of Kode_Bayar values<jupyter_code>df = df[~df.Kode_Bayar.str.contains("C", na=False)]
df['Kode_Bayar'].str.contains("C").value_counts()/df.shape[0]*100<jupyter_output><empty_output><jupyter_text>Kode_Barang<jupyter_code>#check data values
df.Kode_Barang.value_counts()<jupyter_output><empty_output><jupyter_text>Barang<jupyter_code>#check data values
df.Barang.value_counts()<jupyter_output><empty_output><jupyter_text>Jumlah<jupyter_code>#check data values
df.Jumlah.value_counts()<jupyter_output><empty_output><jupyter_text>Removed minus value in the Jumlah column, giving an assumption that it means damage/broken Barang<jupyter_code>df["Jumlah"] = df["Jumlah"].astype(str)
df = df[~df.Jumlah.str.contains("-", na=False)]
print(df['Jumlah'].str.contains("-").value_counts()/df.shape[0]*100)
df["Jumlah"] = df["Jumlah"].astype('float64')
#chang the data type into integer
df["Jumlah"] = df["Jumlah"].astype('int64')
df["Jumlah"].dtype<jupyter_output><empty_output><jupyter_text>Tanggal_Transaksi<jupyter_code>#check data values
df.Tanggal_Transaksi.value_counts()<jupyter_output><empty_output><jupyter_text>Harga_Satuan<jupyter_code>#check data values
df.Harga_Satuan.value_counts()<jupyter_output><empty_output><jupyter_text>Kode_Pelanggan<jupyter_code>#check data values
df.Kode_Pelanggan.value_counts()<jupyter_output><empty_output><jupyter_text>Negara<jupyter_code>#check data values
df.Negara.value_counts()<jupyter_output><empty_output><jupyter_text>check data types<jupyter_code>#check data types
df.dtypes<jupyter_output><empty_output><jupyter_text>- replace data type Kode_Pelanggan column into string.<jupyter_code>#replace into string/object
df["Kode_Pelanggan"] = df["Kode_Pelanggan"].astype(str)<jupyter_output><empty_output><jupyter_text>- replace data type Tanggal_Transaksi column into datetime.<jupyter_code>#replace into dateime
from datetime import datetime

df["Tanggal_Transaksi"] = pd.to_datetime(df["Tanggal_Transaksi"],errors='coerce')
print(df.dtypes)
df.head()<jupyter_output>Kode_Bayar                   object
Kode_Barang                  object
Barang                       object
Jumlah                        int64
Tanggal_Transaksi    datetime64[ns]
Harga_Satuan                float64
Kode_Pelanggan               object
Negara                       object
dtype: object<jupyter_text>create a new column of 'TotalPrice' that contains the total price of every purchase<jupyter_code>df['TotalHarga'] = df['Harga_Satuan'] * df['Jumlah']<jupyter_output><empty_output><jupyter_text>check null values<jupyter_code>#check the percentages of null values
df.isnull().sum() / data.shape[0] * 100<jupyter_output><empty_output><jupyter_text>check data distribution<jupyter_code>#check data
df.describe()<jupyter_output><empty_output><jupyter_text>save file for exploration<jupyter_code>df.to_csv(r'C:/Users/Alifia/Downloads/ml_data.csv', index = False)<jupyter_output><empty_output><jupyter_text>Exploration in TableauLink: https://public.tableau.com/app/profile/alifia.c.harmadi/viz/SupermarketData_16250626153460/Dashboard1  Customer segmentation  RFMRFM segmentation allows marketers to target specific clusters of customers with communications that are much more relevant for their particular behavior – and thus generate much higher rates of response, plus increased loyalty and customer lifetime value. Source: https://www.optimove.com/resources/learning-center/rfm-segmentation<jupyter_code># Calculate recency, relative recency and relative frequency
import numpy as np
from datetime import date

# Take the maximum tanggal transaksi as last
last = df['Tanggal_Transaksi'].max()
last<jupyter_output><empty_output><jupyter_text>Recency<jupyter_code>#get last purchased each customer
df_recency = df.groupby(['Kode_Pelanggan'],as_index=False)['Tanggal_Transaksi'].max()
df_recency.columns = ['Kode_Pelanggan','Transaksi_Terakhir']
df_recency['Recency'] = (last - df_recency['Transaksi_Terakhir'])/ np.timedelta64(1,'D')
df_recency.drop(columns=['Transaksi_Terakhir'],inplace=True)
df_recency['Recency'] = df_recency['Recency'].astype(int)
df_recency<jupyter_output><empty_output><jupyter_text>Frequency<jupyter_code>#get customer's buying frequency
df_freq = df.copy() 
df_freq.drop_duplicates(subset=['Kode_Pelanggan','Kode_Bayar'], keep="first", inplace=True)
df_freq = df_freq.groupby('Kode_Pelanggan',as_index=False)['Kode_Bayar'].count()
df_freq.columns = ['Kode_Pelanggan','Frequency']
df_freq.head()<jupyter_output><empty_output><jupyter_text>Monetary<jupyter_code>df_mon=df.groupby('Kode_Pelanggan',as_index=False)['TotalHarga'].sum()
df_mon.columns = ['Kode_Pelanggan','Monetary']
df_mon.head()<jupyter_output><empty_output><jupyter_text>Merge the data<jupyter_code># join the data

rf = df_recency.merge(df_freq,left_on='Kode_Pelanggan',right_on='Kode_Pelanggan')
rfm = rf.merge(df_mon,left_on='Kode_Pelanggan',right_on='Kode_Pelanggan')
rfm.set_index('Kode_Pelanggan',inplace=True)
rfm.head()


import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

#copy data for clustering
rfm_seg = rfm.copy()

#find the number of cluster with elbow
Num = range(1, 15)
kmeans = [KMeans(n_clusters=i) for i in Num]
score = [kmeans[i].fit(rfm_seg).score(rfm_seg) for i in range(len(kmeans))]
plt.plot(Num,score)
plt.title('Elbow Curve')
plt.show()
#we find that the optimal cluster is 3 from the plot

#fit the kmeans
kmeans = KMeans(n_clusters=3, random_state=0).fit(rfm_seg)
rfm_seg['cluster'] = kmeans.labels_
rfm_seg[rfm_seg.cluster == 0].head(10)


#create the function for applying score
def RScore(x,p,d):
    if x <= d[p][0.25]:
        return 1
    elif x <= d[p][0.50]:
        return 2
    elif x <= d[p][0.75]: 
        return 3
    else:
        return 4

def FMScore(x,p,d):
    if x <= d[p][0.25]:
        return 4
    elif x <= d[p][0.50]:
        return 3
    elif x <= d[p][0.75]: 
        return 2
    else:
        return 1
#get the quantile from rfm data
quantile = rfm.quantile(q=[0.25,0.5,0.75])
#apply the value in rfm_seg data
rfm_seg['R'] = rfm_seg['Recency'].apply(RScore,args=('Recency',quantile))
rfm_seg['F'] = rfm_seg['Frequency'].apply(FMScore, args=('Frequency',quantile))
rfm_seg['M'] = rfm_seg['Monetary'].apply(FMScore, args=('Monetary',quantile))
#create new column called RFM.
#The value from this column will determine the customers segmentation
rfm_seg['RFM'] = rfm_seg.R.map(str) + rfm_seg.F.map(str) + rfm_seg.M.map(str)
rfm_seg.head()<jupyter_output><empty_output><jupyter_text>RFM info- Best Customers = This group consists of those customers who are found in R-Tier-1, F-Tier-1 and M-Tier-1 (RFM = 111)- High-spending New Customers =  This group consists of those customers in 1-4-1 and 1-4-2- Lowest-Spending Active Loyal Customers = This group consists of those customers in segments 1-1-3 and 1-1-4 (they transacted recently and do so often, but spend the least).- Churned Best Customers = This segment consists of those customers in groups 4-1-1, 4-1-2, 4-2-1 and 4-2-2 (they transacted frequently and spent a lot, but it’s been a long time since they’ve transacted).reference: https://www.optimove.com/resources/learning-center/rfm-segmentation<jupyter_code>#see the number of each customer segmentation
print("Best Customers: ",len(rfm_seg[rfm_seg['RFM']=='111']))
print('High-spending Customers: ',len(rfm_seg[rfm_seg['M']==1]))
print("Lowest-Spending Customers: ",len(rfm_seg[rfm_seg['M']==4]))
print('Loyal Customers: ',len(rfm_seg[rfm_seg['F']==1]))
print("Potentialy Churned Customers: ",len(rfm_seg[rfm_seg['R']==4]))
print('Active Customers: ',len(rfm_seg[rfm_seg['F']==1]))
print("Spending Customers: ",len(rfm_seg[rfm_seg['M']==1]))<jupyter_output>Best Customers:  462
High-spending Customers:  1085
Lowest-Spending Customers:  1085
Loyal Customers:  873
Potentialy Churned Customers:  1084
Active Customers:  873
Spending Customers:  1085
{'fraction_non_alphanumeric': 0.12843457206021686, 'fraction_numerical': 0.01663332982419202, 'mean_word_length': 6.6, 'pattern_counts': {'":': 0, '<': 105, '<?xml version=': 0, '>': 99, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '331801', 'license': 'Unlicense', 'n_tokens_mistral': 3640, 'n_tokens_neox': 3491, 'n_words': 710, 'path': 'PracticeCase-MachineLearning.ipynb', 'repo_name': 'alifiaharmd/Python-EDA', 'size': '9516'}