<jupyter_start><jupyter_text>The DataThe data is related with direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict if the client will subscribe (1/0) a term deposit (variable y).This dataset provides the customer information. It includes 41188 records and 21 fields.<jupyter_code>data = pd.read_csv('banking.csv', header=0)
data = data.dropna()
print(data.shape)
print(list(data.columns))
data.head()<jupyter_output><empty_output><jupyter_text>Input variables1 - age (numeric)2 - job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')3 - marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)4 - education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')5 - default: has credit in default? (categorical: 'no','yes','unknown')6 - housing: has housing loan? (categorical: 'no','yes','unknown')7 - loan: has personal loan? (categorical: 'no','yes','unknown')8 - contact: contact communication type (categorical: 'cellular','telephone')9 - month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')10 - day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')11 - duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.12 - campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)13 - pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)14 - previous: number of contacts performed before this campaign and for this client (numeric)15 - poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')16 - emp.var.rate: employment variation rate - (numeric)17 - cons.price.idx: consumer price index - (numeric)18 - cons.conf.idx: consumer confidence index - (numeric)19 - euribor3m: euribor 3 month rate - (numeric)20 - nr.employed: number of employees - (numeric) Predict variable (desired target):y - has the client subscribed a term deposit? (binary: '1','0')The education column of the dataset has many categories and we need to reduce the categories for a better modelling. The education column has the following categories:<jupyter_code>data['education'].unique()<jupyter_output><empty_output><jupyter_text>Let us group "basic.4y", "basic.9y" and "basic.6y" together and call them "basic".<jupyter_code>data['education']=np.where(data['education'] =='basic.9y', 'Basic', data['education'])
data['education']=np.where(data['education'] =='basic.6y', 'Basic', data['education'])
data['education']=np.where(data['education'] =='basic.4y', 'Basic', data['education'])<jupyter_output><empty_output><jupyter_text>After grouping, this is the columns.<jupyter_code>data['education'].unique()<jupyter_output><empty_output><jupyter_text>Data exploration<jupyter_code>data['y'].value_counts()
sns.countplot(x='y',data=data, palette='hls')
plt.show()
count_no_sub = len(data[data['y']==0])
count_sub = len(data[data['y']==1])
pct_of_no_sub = count_no_sub/(count_no_sub+count_sub)
print("percentage of no subscription is", pct_of_no_sub*100)
pct_of_sub = count_sub/(count_no_sub+count_sub)
print("percentage of subscription", pct_of_sub*100)<jupyter_output>percentage of no subscription is 88.73458288821988
percentage of subscription 11.265417111780131<jupyter_text>Our classes are imbalanced, and the ratio of no-subscription to subscription instances is 89:11. Before we go ahead to balance the classes, Let's do some more exploration.<jupyter_code>data.groupby('y').mean()<jupyter_output><empty_output><jupyter_text>Observations:The average age of customers who bought the term deposit is higher than that of the customers who didn't. The pdays (days since the customer was last contacted) is understandably lower for the customers who bought it. The lower the pdays, the better the memory of the last call and hence the better chances of a sale. Surprisingly, campaigns (number of contacts or calls made during the current campaign) are lower for customers who bought the term deposit.We can calculate categorical means for other categorical variables such as education and marital status to get a more detailed sense of our data.<jupyter_code>data.groupby('job').mean()
data.groupby('marital').mean()
data.groupby('education').mean()<jupyter_output><empty_output><jupyter_text>Visualizations<jupyter_code>%matplotlib inline
pd.crosstab(data.job,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Job Title')
plt.xlabel('Job')
plt.ylabel('Frequency of Purchase')<jupyter_output><empty_output><jupyter_text>The frequency of purchase of the deposit depends a great deal on the job title. Thus, the job title can be a good predictor of the outcome variable.<jupyter_code>table=pd.crosstab(data.marital,data.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Marital Status vs Purchase')
plt.xlabel('Marital Status')
plt.ylabel('Proportion of Customers')<jupyter_output><empty_output><jupyter_text>Hard to see, but the marital status does not seem a strong predictor for the outcome variable.<jupyter_code>table=pd.crosstab(data.education,data.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Education vs Purchase')
plt.xlabel('Education')
plt.ylabel('Proportion of Customers')<jupyter_output><empty_output><jupyter_text>Education seems a good predictor of the outcome variable.<jupyter_code>pd.crosstab(data.day_of_week,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Day of Week')
plt.xlabel('Day of Week')
plt.ylabel('Frequency of Purchase')<jupyter_output><empty_output><jupyter_text>Day of week may not be a good predictor of the outcome.<jupyter_code>pd.crosstab(data.month,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Month')
plt.xlabel('Month')
plt.ylabel('Frequency of Purchase')<jupyter_output><empty_output><jupyter_text>Month might be a good predictor of the outcome variable.<jupyter_code>data.age.hist()
plt.title('Histogram of Age')
plt.xlabel('Age')
plt.ylabel('Frequency')<jupyter_output><empty_output><jupyter_text>Most customers of the bank in this dataset are in the age range of 30-40.<jupyter_code>pd.crosstab(data.poutcome,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Poutcome')
plt.xlabel('Poutcome')
plt.ylabel('Frequency of Purchase')<jupyter_output><empty_output><jupyter_text>Poutcome seems to be a good predictor of the outcome variable.  Create dummy variables<jupyter_code>cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
for var in cat_vars:
    cat_list='var'+'_'+var
    cat_list = pd.get_dummies(data[var], prefix=var)
    data1=data.join(cat_list)
    data=data1
    
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
data_vars=data.columns.values.tolist()
to_keep=[i for i in data_vars if i not in cat_vars]
data_final=data[to_keep]
data_final.columns.values<jupyter_output><empty_output><jupyter_text>Over-sampling using SMOTE<jupyter_code>X = data_final.loc[:, data_final.columns != 'y']
y = data_final.loc[:, data_final.columns == 'y']
from imblearn.over_sampling import SMOTE

os = SMOTE(random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
columns = X_train.columns

os_data_X,os_data_y=os.fit_sample(X_train, y_train)
os_data_X = pd.DataFrame(data=os_data_X,columns=columns )
os_data_y= pd.DataFrame(data=os_data_y,columns=['y'])
# we can Check the numbers of our data
print("length of oversampled data is ",len(os_data_X))
print("Number of no subscription in oversampled data",len(os_data_y[os_data_y['y']==0]))
print("Number of subscription",len(os_data_y[os_data_y['y']==1]))
print("Proportion of no subscription data in oversampled data is ",len(os_data_y[os_data_y['y']==0])/len(os_data_X))
print("Proportion of subscription data in oversampled data is ",len(os_data_y[os_data_y['y']==1])/len(os_data_X))<jupyter_output>C:\Users\SusanLi\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)<jupyter_text>Recursive feature elimination<jupyter_code>data_final_vars=data_final.columns.values.tolist()
y=['y']
X=[i for i in data_final_vars if i not in y]
from sklearn import datasets
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression()

rfe = RFE(logreg, 20)
rfe = rfe.fit(os_data_X, os_data_y.values.ravel())
print(rfe.support_)
print(rfe.ranking_)<jupyter_output>[False False False False False False False False  True False False  True
 False  True False False False False False False False False False False
 False  True False False False False  True False False False  True  True
 False False False False False False False  True  True  True  True  True
  True  True  True  True  True  True False False False False False False
  True False  True]
[39 38 26 42  9 12 24 36  1 35  8  1  7  1  5 32  2  4 31  3  6 10 23 21
 17  1 14 18 15 22  1 20 16 19  1  1 41 28 44 37 33 43 34  1  1  1  1  1
  1  1  1  1  1  1 29 30 11 27 40 25  1 13  1]<jupyter_text>The Recursive Feature Elimination (RFE) has helped us select the following features: "previous", "euribor3m", "job_blue-collar", "job_retired", "job_services", "job_student", "default_no", "month_aug", "month_dec", "month_jul", "month_nov", "month_oct", "month_sep", "day_of_week_fri", "day_of_week_wed", "poutcome_failure", "poutcome_nonexistent", "poutcome_success".<jupyter_code>cols=['euribor3m', 'job_blue-collar', 'job_housemaid', 'marital_unknown', 'education_illiterate', 'default_no', 'default_unknown', 
      'contact_cellular', 'contact_telephone', 'month_apr', 'month_aug', 'month_dec', 'month_jul', 'month_jun', 'month_mar', 
      'month_may', 'month_nov', 'month_oct', "poutcome_failure", "poutcome_success"] 
X=os_data_X[cols]
y=os_data_y['y']<jupyter_output><empty_output><jupyter_text>Implementing the model<jupyter_code>import statsmodels.api as sm
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary2())<jupyter_output>Warning: Maximum number of iterations has been exceeded.
         Current function value: 0.545891
         Iterations: 35
                                Results: Logit
==============================================================================
Model:                   Logit                No. Iterations:       35.0000   
Dependent Variable:      y                    Pseudo R-squared:     0.212     
Date:                    2018-09-10 12:16     AIC:                  55867.1778
No. Observations:        51134                BIC:                  56044.0219
Df Model:                19                   Log-Likelihood:       -27914.   
Df Residuals:            51114                LL-Null:              -35443.   
Converged:               0.0000               Scale:                1.0000    
------------------------------------------------------------------------------
                      Coef.    Std.Err.    z     P>|z|     [0.025     0.975]  
----------------------------------------[...]<jupyter_text>The p-values for four variables are very high, therefore, we will remove them.<jupyter_code>cols=['euribor3m', 'job_blue-collar', 'job_housemaid', 'marital_unknown', 'education_illiterate', 
      'month_apr', 'month_aug', 'month_dec', 'month_jul', 'month_jun', 'month_mar', 
      'month_may', 'month_nov', 'month_oct', "poutcome_failure", "poutcome_success"] 
X=os_data_X[cols]
y=os_data_y['y']
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary2())<jupyter_output>Optimization terminated successfully.
         Current function value: 0.555865
         Iterations 7
                           Results: Logit
=====================================================================
Model:                Logit             No. Iterations:    7.0000    
Dependent Variable:   y                 Pseudo R-squared:  0.198     
Date:                 2018-09-10 12:38  AIC:               56879.2425
No. Observations:     51134             BIC:               57020.7178
Df Model:             15                Log-Likelihood:    -28424.   
Df Residuals:         51118             LL-Null:           -35443.   
Converged:            1.0000            Scale:             1.0000    
---------------------------------------------------------------------
                      Coef.  Std.Err.    z     P>|z|   [0.025  0.975]
---------------------------------------------------------------------
euribor3m            -0.4488   0.0074 -60.6837 0.0000 -0.4633 -0.4343
job_blue-collar [...]<jupyter_text>Logistic Regression Model Fitting<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn import metrics

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))<jupyter_output>Accuracy of logistic regression classifier on test set: 0.74<jupyter_text>Confusion Matrix<jupyter_code>from sklearn.metrics import confusion_matrix
confusion_matrix = confusion_matrix(y_test, y_pred)
print(confusion_matrix)
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

          0       0.71      0.80      0.75      7666
          1       0.77      0.67      0.72      7675

avg / total       0.74      0.74      0.74     15341<jupyter_text>Interpretation:Of the entire test set, 74% of the promoted term deposit were the term deposit that the customers liked. Of the entire test set, 74% of the customer's preferred term deposit were promoted.<jupyter_code>from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
logit_roc_auc = roc_auc_score(y_test, logreg.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14115347018572824, 'fraction_numerical': 0.033235581622678395, 'mean_word_length': 4.358240223463687, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '322755', 'license': 'MIT', 'n_tokens_mistral': 5279, 'n_tokens_neox': 4808, 'n_words': 1317, 'path': 'Logistic Regression balanced.ipynb', 'repo_name': 'BeeChainFoundation/Brainstorming', 'size': '15345'}