<jupyter_start><jupyter_text>AmExpert 2021 – Machine Learning Hackathon  Problem Statement* A mid-sized private bank that includes a variety of banking products, such as savings accounts, current accounts, investment products, credit products, and home loans. * The task is to predict the next set of products (upto 3 products) for a set of customers (test data) based on their demographics and current product holdings.midm **Project Flow*** Importing the data* EDA* Feature Engineering / Data Preparation* Splitting the data to train and test* Feature Scaling* Model Building* Model Evaluation* Prediction on Given Test Set  Importing the libraries<jupyter_code># Required libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from ast import literal_eval

# Data Preprocessing
from sklearn.preprocessing import LabelEncoder, MultiLabelBinarizer
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.model_selection import train_test_split, KFold

# Model building
from sklearn.multiclass import OneVsRestClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier
from catboost import CatBoostClassifier
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from keras.layers import Dense
from keras.models import Sequential
from keras.callbacks import EarlyStopping

# Model Evaluation
from sklearn.metrics import multilabel_confusion_matrix, precision_score, average_precision_score
from sklearn.metrics import accuracy_score, f1_score, hamming_loss
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import classification_report

import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Data Exploration<jupyter_code># Loading the train & test datasets

# convert Target Variables to list while loading the csv using literal_eval
train_df = pd.read_csv('train_go05W65.csv', converters={"Product_Holding_B1": literal_eval, "Product_Holding_B2": literal_eval})
test_df = pd.read_csv('test_VkM91FT.csv', converters={"Product_Holding_B1": literal_eval})
sample_submission = pd.read_csv('sample_submission_kF044ur.csv')
# Training set
train_df.head()
# Testing set
test_df.head()
train_df.info() # There are no null values in the dataset
train_df.describe() # There are no outliers as the mean & median are almost similar 
# Analysis of categorical data
train_df.describe(include = 'object')<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code># Visualising the Gender Count
fig, ax = plt.subplots(1, 2, figsize=(10, 8))
sns.countplot(train_df['Gender'], ax=ax[0])
ax[1].pie(train_df['Gender'].value_counts(), labels=['Male', 'Female'], autopct="%.0f%%")
fig.show()
# Visualising the City_Category
fig, ax = plt.subplots(1,2, figsize = (10,8))
sns.countplot(train_df['City_Category'], ax=ax[0])
ax[1].pie(train_df['City_Category'].value_counts(), labels=['C1', 'C2'], autopct="%.0f%%")
fig.show()
# Visualising the Customer_Category
fig, ax = plt.subplots(1,2, figsize = (10,8))
sns.countplot(train_df['Customer_Category'], ax=ax[0])
ax[1].pie(train_df['Customer_Category'].value_counts(), labels=['S1', 'S2', 'S3'], autopct="%.0f%%")
fig.show()
# Visualising the Is_Active
fig, ax = plt.subplots(1,2, figsize = (10,8))
sns.countplot(train_df['Is_Active'], ax=ax[0])
ax[1].pie(train_df['Is_Active'].value_counts(), labels=['0','1'], autopct="%.0f%%")
fig.show()
# Plot the Age histogram
sns.histplot(data=train_df, x='Age', kde=True)
# Plot the Vintage histogram
sns.histplot(data=train_df, x='Vintage', kde=True)
plt.show()
# Group by Gender and plot the mean Age
gender_data = train_df.groupby('Gender').mean().reset_index()
sns.barplot(x=gender_data['Gender'], y=gender_data['Age'])<jupyter_output><empty_output><jupyter_text>**Both the genders have equal distribution in mean Age.**<jupyter_code># Group by Gender and plot the mean Vintage
sns.barplot(x=gender_data['Gender'], y=gender_data['Vintage'])<jupyter_output><empty_output><jupyter_text>**Both distributions have equal distribution of mean Vintage.**<jupyter_code># Insights across City Category
train_df.groupby('City_Category').mean()<jupyter_output><empty_output><jupyter_text>**Almost equal distributions of Age, Vintage and Is_Active observed across both the cities.**<jupyter_code># Insights across Customer_Category
train_df.groupby('Customer_Category').mean()<jupyter_output><empty_output><jupyter_text>**Almost equal distributions of Age, Vintage and Is_Active observed across both the Customer Cateogry.**<jupyter_code># Using Multi label Binarizer for converting Product_Holding_B1 column into numeric
mlb = MultiLabelBinarizer(sparse_output=False)
out_train = mlb.fit_transform(train_df['Product_Holding_B1'])
# Adding the converted columns to train dataset
train_df = train_df.join(pd.DataFrame(out_train, columns=mlb.classes_))
train_df.head()
# Products 13, 16 & 17 are bought more
plt.figure(figsize=(20,10))
sns.barplot(x=mlb.classes_, y=train_df[mlb.classes_].sum())
train_df.iloc[:, 9:].sum(axis=1).value_counts()
# We can see that combination of 1,2 & 3 products are bought more by the customers
plt.figure(figsize=(12, 5))
sns.barplot(x=[1, 2, 3, 4, 5, 6, 7, 8],y=train_df.iloc[:, 9:].sum(axis=1).value_counts())
# Target variable
# Converting Product_Holding_B2 column into numeric
mlb2 = MultiLabelBinarizer(sparse_output=False)
out_train_y = mlb2.fit_transform(train_df['Product_Holding_B2'])
# Creating different dataframe for target variables
target = pd.DataFrame(out_train_y, columns = mlb2.classes_)
target.head()
# Product 13, 16 & 17 are highly bought
plt.figure(figsize=(16,10))
sns.barplot(x=mlb2.classes_, y=target[mlb2.classes_].sum())<jupyter_output><empty_output><jupyter_text>Feature engineering/Data Preparation<jupyter_code># Columns with object datatype
columns_cat = train_df.select_dtypes(include='object')
columns_cat.head()
# Converting categorical columns to numeric
train_df = pd.get_dummies(train_df, columns=['Gender', 'City_Category', 'Customer_Category'], drop_first=True)
# Removing columns which are not required
train_df = train_df.drop('Customer_ID', axis=1)
test_df = test_df.drop('Customer_ID', axis=1)
# Removing Product_Holding_B1 & Product_Holding_B2 columns
train_df = train_df.drop('Product_Holding_B1', axis=1)
train_df = train_df.drop('Product_Holding_B2', axis=1)
train_df.head()
test_df.head()
# Preparing test dataset

# Converting categorical columns to numeric
test_df = pd.get_dummies(test_df, columns=['Gender','City_Category','Customer_Category'], drop_first= True)

# Converting Product_Holding_B1 column into numeric
mlb = MultiLabelBinarizer(sparse_output=False)
test_df = test_df.join(pd.DataFrame(mlb.fit_transform(test_df['Product_Holding_B1']), columns = mlb.classes_))

# Removing Product_Holding_B1 columns 
test_df = test_df.drop('Product_Holding_B1', axis=1) 
test_df.head()<jupyter_output><empty_output><jupyter_text>Spliting the data<jupyter_code># Creating X & Y variables
X= train_df
y = target
# Split the data into 80% train & 20% test 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=10)
# Checking shape of train & test 
print("X_train = {}".format(X_train.shape))
print("X_test = {}".format(X_test.shape))
print("y_train = {}".format(y_train.shape))
print("y_test = {}".format(y_test.shape))<jupyter_output>X_train = (30198, 29)
X_test = (7550, 29)
y_train = (30198, 20)
y_test = (7550, 20)<jupyter_text>Scaling the data<jupyter_code># Scaling continuous columns (Age & Vintage) of train & test by using Normalization
col_scale = ['Age', 'Vintage']

for col in col_scale:
    scale = MinMaxScaler()
    train_df[col] = scale.fit_transform(train_df[col].values.reshape(-1, 1))
    test_df[col] = scale.transform(test_df[col].values.reshape(-1, 1))<jupyter_output><empty_output><jupyter_text>Model Building  Logistic<jupyter_code># Model Building
lr = LogisticRegression()
model_lr = MultiOutputClassifier(lr)
model_lr.fit(X_train, y_train)
# Make Predictions and compute the precision at K
y_pred_prob_lr = model_lr.predict_proba(X_test)
y_pred_lr = model_lr.predict(X_test)
# Metrics
print('Accuracy : {}'.format(accuracy_score(y_test, y_pred_lr)))
print('Hamming loss : {}'.format(hamming_loss(y_test, y_pred_lr)))
print('Average precision score : {}'.format(average_precision_score(y_test, y_pred_lr, average='weighted')))<jupyter_output>Accuracy : 0.2691390728476821
Hamming loss : 0.051754966887417216
Average precision score : 0.37818203475532663<jupyter_text>XGBoost<jupyter_code># Model Building
xg = XGBClassifier(n_estimators=100,
                   max_depth=5,
                   random_state=10,
                   eta=0.01,
                   gamma=0.5,
                   learning_rate=0.1)
model_xg = MultiOutputClassifier(estimator=xg)
model_xg.fit(X_train, y_train)
# Make Predictions and compute the precision at K
y_pred_prob_xg = model_xg.predict_proba(X_test)
y_pred_xg = model_xg.predict(X_test)
# Metrics
print('Accuracy : {}'.format(accuracy_score(y_test, y_pred_xg)))
print('Hamming loss : {}'.format(hamming_loss(y_test, y_pred_xg)))
print('Average precision score : {}'.format(average_precision_score(y_test, y_pred_xg, average='weighted')))<jupyter_output>Accuracy : 0.27642384105960266
Hamming loss : 0.04990066225165563
Average precision score : 0.39504240909669647<jupyter_text>Catboost<jupyter_code># Model Building
cat = CatBoostClassifier(iterations=10, random_state=10)
model_cat = MultiOutputClassifier(estimator=cat)
model_cat.fit(X_train, y_train)
# Make Predictions and compute the precision at K
y_pred_prob_cat = model_cat.predict_proba(X_test)
y_pred_cat = model_cat.predict(X_test)
# Metrics
print('Accuracy : {}'.format(accuracy_score(y_test, y_pred_cat)))
print('Hamming loss : {}'.format(hamming_loss(y_test, y_pred_cat)))
print('Average precision score : {}'.format(average_precision_score(y_test, y_pred_cat, average='weighted')))<jupyter_output>Accuracy : 0.26543046357615896
Hamming loss : 0.05037086092715232
Average precision score : 0.38661856603872014<jupyter_text>ANN Model<jupyter_code># Neural Network (NN)

# Creating the Model
model = Sequential()
model.add(Dense(units=32, activation='relu',
                kernel_initializer='uniform', input_shape=[X_train.shape[1]]))
model.add(Dense(units=16, activation='relu', kernel_initializer='normal'))
model.add(Dense(units=8, activation='relu', kernel_initializer='normal'))
model.add(Dense(units=target.shape[1], activation='sigmoid'))
# Model optimization.
# Loss : It is more natural to use BinaryCrossEntropy loss for this multi category
# Optimizer : Using Adam for a faster learning.

# Compile the model and train it with the training data.
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
history = model.fit(X_train,
                    y_train,
                    validation_data=(X_test, y_test),
                    epochs=200,
                    verbose=0,
                    batch_size=32)
# Plot the results of training.
plt.plot(history.history['accuracy'], label='accuracy')
plt.plot(history.history['loss'], label='loss')
plt.xlabel('epoch number')
plt.ylabel('quantity')
plt.legend()
plt.show()

# Plot the results of testing
plt.plot(history.history['val_accuracy'], label='val_accuracy')
plt.plot(history.history['val_loss'], label='val_loss')
plt.xlabel('epoch number')
plt.ylabel('quantity')
plt.legend()
plt.show()
# Make Predictions and compute the precision at K
y_pred_prob_ANN = model.predict_proba(X_test)
y_pred_ANN = model.predict(X_test)
# Metrics
#print('Accuracy : {}'.format(accuracy_score(y_test, y_pred_ANN)))
#print('Hamming loss : {}'.format(hamming_loss(y_test, y_pred_ANN)))
print('Average precision score : {}'.format(average_precision_score(y_test, y_pred_ANN, average='weighted')))
#compare_score =<jupyter_output><empty_output><jupyter_text>**Going ahead with ANN model as it is giving highest Average precision score.**  Prediction   Predictions on training<jupyter_code># creation of dictionary for predicting Y variable
int_char = dict((i, c) for i, c in enumerate(y_train))

prediction = []
for i in range(len(X_test)):
    # highest probablity will have zero rank
    rank = np.argsort(y_pred_prob_ANN[i])

    # -> pic the top 3 Probabilities
    top_3_proba = ([i for i in rank])[-3:]

    pred = [int_char[x] for x in top_3_proba]
    prediction.append(pred)
prediction<jupyter_output><empty_output><jupyter_text>Predictions on test data<jupyter_code>#### Predictions on test data
pred_test = model.predict_proba(test_df)
prediction_test = []
for i in range(len(test_df)):

    # highest probablity will have zero rank
    rank = np.argsort(pred_test[i])
    
    # -> pic the top 3 Probabilities
    top_3_proba = ([i for i in rank])[-3:]  

    pred = [int_char[x] for x in top_3_proba]
    prediction_test.append(pred)
prediction_test<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11890524379024839, 'fraction_numerical': 0.027215578043544925, 'mean_word_length': 6.601981351981352, 'pattern_counts': {'":': 3, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '63406', 'license': 'MIT', 'n_tokens_mistral': 4717, 'n_tokens_neox': 4387, 'n_words': 1017, 'path': 'Hackathon - American Express - Final.ipynb', 'repo_name': 'DeeshaM/AmExpert_Hackathon_2021', 'size': '13044'}