<jupyter_start><jupyter_text>Kaggle Competition  Abstract It is very important for bank to identify which customers are satisfied or unstaisfied with their services. Santander Bank made some of the data public and asked us to help them identify dissatisfied customers early in their relationship. Doing so would allow Santander to take proactive steps to improve a customer's happiness before it's too late. **Competition: -** From frontline support teams to C-suites, customer satisfaction is a key measure of success. Unhappy customers don't stick around. What's more, unhappy customers rarely voice their dissatisfaction before leaving.Santander Bank is asking Kagglers to help them identify dissatisfied customers early in their relationship. Doing so would allow Santander to take proactive steps to improve a customer's happiness before it's too late.  Dependencies<jupyter_code># pip install numpy
# pip install pandas
# pip install matplotlib
# pip install seaborn
# pip install scikit-learn<jupyter_output><empty_output><jupyter_text>Importing Libraries And Datasets<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

import ssl
ssl._create_default_https_context = ssl._create_unverified_context
# Load the training data
train = pd.read_csv( "https://raw.githubusercontent.com/sumedhtelang/Kaggle_santander_customer_satisfaction/master/data/train.csv")

# Load the testing data
test = pd.read_csv( "https://raw.githubusercontent.com/sumedhtelang/Kaggle_santander_customer_satisfaction/master/data/test.csv")<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis **num_var4 :** how many bank account/loan/any other form of account customers have In below visualization we can see that majority of the customers have only 1 type of product with santander bank<jupyter_code>train.num_var4.hist(bins=100)
plt.xlabel('Number of bank account')
plt.ylabel('Number of customers')
plt.title('Customers vs no of product')
plt.show()<jupyter_output><empty_output><jupyter_text>Histogram of customers with no of products with santander bank Here we can see that majority of unhappy customers have less products with the bank<jupyter_code>import seaborn as sns
# Let's look at the density of the of happy/unhappy customers in function of the number of bank products
sns.FacetGrid(train, hue="TARGET", size=6) \
   .map(plt.hist, "num_var4") \
   .add_legend()
plt.title('Unhappy customers')
plt.show()<jupyter_output>/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/seaborn/axisgrid.py:316: UserWarning: The `size` parameter has been renamed to `height`; please update your code.
  warnings.warn(msg, UserWarning)<jupyter_text>Number of customers unhappy and number of products they have with bank<jupyter_code>train[train.TARGET==1].num_var4.hist(bins=6)
plt.title('Amount of unhappy customers in function of the number of accounts');<jupyter_output><empty_output><jupyter_text>One of the most important feature var15. According to a Kaggle discussion board var15 is the age of the customer. Average age of the customer in the bank is around 33<jupyter_code>train['var15'].describe()<jupyter_output><empty_output><jupyter_text>Histogram of distribution age of customers<jupyter_code>#Looks more normal, plot the histogram
train['var15'].hist(bins=100);<jupyter_output><empty_output><jupyter_text>Here in this kernel density plot we can see that majority of unhappy customers are older.<jupyter_code># Let's look at the density of the age of happy/unhappy customers
sns.FacetGrid(train, hue="TARGET", size=6) \
   .map(sns.kdeplot, "var15") \
   .add_legend()
plt.title('Kernel density plot of unhappy customers vs age');<jupyter_output>/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/seaborn/axisgrid.py:316: UserWarning: The `size` parameter has been renamed to `height`; please update your code.
  warnings.warn(msg, UserWarning)<jupyter_text>saldo_varX => saldo => balance Here we can see that most customers have accountbalance between 0 to 5,00,000<jupyter_code>train.saldo_var30.hist(bins=100)
plt.xlim(0, train.saldo_var30.max());<jupyter_output><empty_output><jupyter_text>var38:- mortgage value with bank Scatter plot of age of customer and mortgage value the have at bank<jupyter_code>sns.FacetGrid(train, hue="TARGET", size=10) \
   .map(plt.scatter, "var38", "var15") \
   .add_legend()<jupyter_output>/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/seaborn/axisgrid.py:316: UserWarning: The `size` parameter has been renamed to `height`; please update your code.
  warnings.warn(msg, UserWarning)<jupyter_text>- Above plot suggest we split up var38 into two variables- var38mc == 1 when var38 has the most common value and 0 otherwise- logvar38 is log transformed feature when var38mc is 0, zero otherwise<jupyter_code>train['var38mc'] = np.isclose(train.var38, 117310.979016)
train['logvar38'] = train.loc[~train['var38mc'], 'var38'].map(np.log)
train.loc[train['var38mc'], 'logvar38'] = 0
#Check for null values
print('Number of nan in var38mc', train['var38mc'].isnull().sum())
print('Number of nan in logvar38',train['logvar38'].isnull().sum())
sns.FacetGrid(train, hue="TARGET", size=10) \
   .map(plt.scatter, "logvar38", "var15") \
   .add_legend()
plt.ylim([0,120]); # Age must be positive ;-
# Exclude most common value for var38 
sns.FacetGrid(train[~train.var38mc], hue="TARGET", size=10) \
   .map(plt.scatter, "logvar38", "var15") \
   .add_legend()
plt.ylim([0,120]);
sns.FacetGrid(train[train.var38mc], hue="TARGET", size=6) \
   .map(sns.kdeplot, "var15") \
   .add_legend();
y_train = train.TARGET
X_train = train.drop(['ID','TARGET', 'var38mc', 'logvar38'], axis=1)

id_test = test['ID']
X_test_pred = test.drop(['ID'], axis=1)

# length of dataset
len_train = len(X_train)
len_test  = len(X_test_pred)
X_train.columns<jupyter_output><empty_output><jupyter_text>Feature Importance Chi Square test for most important features.<jupyter_code>from sklearn.feature_selection import SelectPercentile
from sklearn.feature_selection import f_classif,chi2
from sklearn.preprocessing import Binarizer, scale

p = 3

X_bin = Binarizer().fit_transform(scale(X_train))
selectChi2 = SelectPercentile(chi2, percentile=p).fit(X_bin, y_train)
selectF_classif = SelectPercentile(f_classif, percentile=p).fit(X_train, y_train)

chi2_selected = selectChi2.get_support()
chi2_selected_features = [ f for i,f in enumerate(X_train.columns) if chi2_selected[i]]
print('Chi2 selected {} features {}.'.format(chi2_selected.sum(),
   chi2_selected_features))

f_classif_selected = selectF_classif.get_support()
f_classif_selected_features = [ f for i,f in enumerate(X_train.columns) if f_classif_selected[i]]
print('F_classif selected {} features {}.'.format(f_classif_selected.sum(),
   f_classif_selected_features))

selected = chi2_selected & f_classif_selected
print('Chi2 & F_classif selected {} features'.format(selected.sum()))
features = [ f for f,s in zip(X_train.columns, selected) if s]
print (features)<jupyter_output>/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sklearn/preprocessing/_data.py:174: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.
  warnings.warn("Numerical issues were encountered "<jupyter_text>Random Forest Classifier to check most important features.<jupyter_code>y_train.value_counts() / float(y_train.size)
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100)
rf.fit(X_train, y_train)
feat_imp = pd.Series(rf.feature_importances_, index=X_train.columns)
feat_imp.sort_values(inplace=True)
ax = feat_imp.tail(20).plot(kind='barh', figsize=(10,7), title='Feature importance')
train.isnull().count()<jupyter_output><empty_output><jupyter_text>Correlation Matrix of Important Features Selected from chi-square test<jupyter_code>X_sel = train[features+['TARGET']]
cor_mat = X_sel.corr()
f, ax = plt.subplots()
sns.heatmap(cor_mat,linewidths=.5, ax=ax, annot=True);<jupyter_output><empty_output><jupyter_text>Model Building For Prediction<jupyter_code>clf = xgb.XGBClassifier(missing=np.nan, 
                        max_depth=5, 
                        n_estimators=560, 
                        learning_rate=0.02, 
                        nthread=4, 
                        subsample=0.7, 
                        colsample_bytree=0.7, 
                        seed=4242)
X_train, X_test, y_train, y_test= train_test_split(X_train, y_train, test_size=0.3)
clf.fit(X_train, y_train, early_stopping_rounds=20, eval_metric="auc", eval_set=[(X_test, y_test)])
print('Overall AUC:', roc_auc_score(y_train, clf.predict_proba(X_train)[:,1])) 
y_pred= clf.predict_proba(X_test_pred)[:,1]
submission = pd.DataFrame({"ID":id_test, "TARGET":y_pred})
submission.to_csv("result/submission1.csv", index=False)<jupyter_output><empty_output><jupyter_text>Attempt of randomize grid search<jupyter_code># from sklearn.model_selection import GridSearchCV
# parameters = {'max_depth': [6,7,8,9],
#               'n_estimators':[560, 580, 600],
#               'learning_rate':[0.02, 0.03, 0.04, 0.05], 
#               'nthread':[4,5,6,7]}

# xgb_grid = GridSearchCV(xgb.XGBClassifier(),parameters,cv = 3,n_jobs = 5,verbose=True)

# xgb_grid.fit(X_train,y_train)

# print(xgb_grid.best_score_)
# print(xgb_grid.best_params_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11060338560598193, 'fraction_numerical': 0.024613147782739642, 'mean_word_length': 5.80565371024735, 'pattern_counts': {'":': 2, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '345285', 'license': 'MIT', 'n_tokens_mistral': 3344, 'n_tokens_neox': 3136, 'n_words': 793, 'path': 'kaggle_santander_customer_satisfaction.ipynb', 'repo_name': 'sumedhtelang/Kaggle_santander_customer_satisfaction', 'size': '9629'}