<jupyter_start><jupyter_text>About DatasetThe dataset comes from the UCI Machine Learning repository, and it is related to direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict whether the client will subscribe (1/0) to a term deposit (variable y).<jupyter_code>import pandas as pd
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt 
plt.rc("font", size=14)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import seaborn as sns
sns.set(style="white")
sns.set(style="whitegrid", color_codes=True)
df=pd.read_csv('https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Input Variables1. age (numeric)2. job : type of job (categorical: “admin”, “blue-collar”, “entrepreneur”, “housemaid”, “management”, “retired”, “self-employed”, “services”, “student”, “technician”, “unemployed”, “unknown”)3. marital : marital status (categorical: “divorced”, “married”, “single”, “unknown”)4. education (categorical: “basic.4y”, “basic.6y”, “basic.9y”, “high.school”, “illiterate”, “professional.course”, “university.degree”, “unknown”)5. default: has credit in default? (categorical: “no”, “yes”, “unknown”)6. housing: has housing loan? (categorical: “no”, “yes”, “unknown”)7. loan: has personal loan? (categorical: “no”, “yes”, “unknown”)8. contact: contact communication type (categorical: “cellular”, “telephone”)9. month: last contact month of year (categorical: “jan”, “feb”, “mar”, …, “nov”, “dec”)10. day_of_week: last contact day of the week (categorical: “mon”, “tue”, “wed”, “thu”, “fri”)11. duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y=’no’). The duration is not known before a call is performed, also, after the end of the call, y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model12. campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)13. pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)14. previous: number of contacts performed before this campaign and for this client (numeric)15. poutcome: outcome of the previous marketing campaign (categorical: “failure”, “nonexistent”, “success”)16. emp.var.rate: employment variation rate — (numeric)17. cons.price.idx: consumer price index — (numeric)18. cons.conf.idx: consumer confidence index — (numeric)19. euribor3m: euribor 3 month rate — (numeric)20. nr.employed: number of employees — (numeric)  Predict Variablesy — has the client subscribed a term deposit? (binary: “1”, means “Yes”, “0” means “No”) (A term deposit is a type of deposit account held at a financial institution where money is locked up for some set period of time. )<jupyter_code># Checking whether any column contains null value
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 41188 entries, 0 to 41187
Data columns (total 21 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   age             41188 non-null  int64  
 1   job             41188 non-null  object 
 2   marital         41188 non-null  object 
 3   education       41188 non-null  object 
 4   default         41188 non-null  object 
 5   housing         41188 non-null  object 
 6   loan            41188 non-null  object 
 7   contact         41188 non-null  object 
 8   month           41188 non-null  object 
 9   day_of_week     41188 non-null  object 
 10  duration        41188 non-null  int64  
 11  campaign        41188 non-null  int64  
 12  pdays           41188 non-null  int64  
 13  previous        41188 non-null  int64  
 14  poutcome        41188 non-null  object 
 15  emp_var_rate    41188 non-null  float64
 16  cons_price_idx  41188 non-null  float64
 17  cons_conf_idx   41188 non-null [...]<jupyter_text>The Education column has many categories, we can possibly reduce some of them to have a better modelling.Let's see the categories of that column.<jupyter_code>df.education.unique()<jupyter_output><empty_output><jupyter_text>We can reduce the number of categories by adding 'basic.4y','basic.6y' and 'basic.9y' under a single category that is 'Basic'.<jupyter_code># Merging all basic category together
df['education']=np.where(df['education'] =='basic.9y', 'Basic', df['education'])
df['education']=np.where(df['education'] =='basic.6y', 'Basic', df['education'])
df['education']=np.where(df['education'] =='basic.4y', 'Basic', df['education'])
# After grouping
df['education'].unique()<jupyter_output><empty_output><jupyter_text>Data Exploration<jupyter_code>df['y'].value_counts()
sns.countplot(x='y',data=df)
plt.savefig('count_plot')
y_yes=len(df[df['y']==1])
y_no=len(df[df['y']==0])
print("The percantage of no_subscription is ",100*(y_no/(y_yes+y_no)))
print("The percantage of subscription is ",100*(y_yes/(y_yes+y_no)))<jupyter_output>The percantage of no_subscription is  88.73458288821988
The percantage of subscription is  11.265417111780131<jupyter_text>Here we can see that the percentage of no_subscription is 88.73 (approx) and the percentage of subscription is 11.26 (approx). So our classes are imbalanced.Note: An imbalanced classification problem is an example of a classification problem where the distribution of examples across the known classes is biased or skewed. Imbalanced classifications pose a challenge for predictive modeling as most of the machine learning algorithms used for classification were designed around the assumption of an equal number of examples for each class. Let's do some more exploration.<jupyter_code>df.groupby('y').mean()<jupyter_output><empty_output><jupyter_text>Here we can observe that the customer who have bought the term deposit* Average age was lower for * Average call duration was higher* The average campaign was lower Again we can observe other categorical variables like education,job,maritial status to get more detailes information of our data.<jupyter_code>df.groupby('job').mean()<jupyter_output><empty_output><jupyter_text>Here we can observe that* Students have the most average term deposit subscription followed by 'retired'<jupyter_code>df.groupby('marital').mean()<jupyter_output><empty_output><jupyter_text>Here we observe that more or categories under marital don't have any influence on the subcription of the term deposit<jupyter_code>df.groupby('education').mean()<jupyter_output><empty_output><jupyter_text>Here we can observe that 'illiterate' people tend to buy a term deposit more than other education background.  Visualizations<jupyter_code>pd.crosstab(df.job,df.y).plot(kind='bar')
plt.title('Purchase Frequency for Job Title')
plt.xlabel('Job')
plt.ylabel('Frequency of Purchase')
plt.savefig('purchase_fre_job')<jupyter_output><empty_output><jupyter_text>Here we see that the frequency of purchase of the deposit depends a great deal on the job title. So job title can be a good predictor of the outcome variable.<jupyter_code>table=pd.crosstab(df.marital,df.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Marital Status vs Purchase')
plt.xlabel('Marital Status')
plt.ylabel('Proportion of Customers')
plt.savefig('mariral_vs_pur_stack')<jupyter_output><empty_output><jupyter_text>Here we can say that the maritial status isn't a strong predictor of the outcome variable.<jupyter_code>table=pd.crosstab(df.education,df.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Education vs Purchase')
plt.xlabel('Education')
plt.ylabel('Proportion of Customers')
plt.savefig('edu_vs_pur_stack')<jupyter_output><empty_output><jupyter_text>Education might be a good predictor of the outcome.<jupyter_code>pd.crosstab(df.day_of_week,df.y).plot(kind='bar')
plt.title('Purchase Frequency for Day of Week')
plt.xlabel('Day of Week')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_dayofweek_bar')<jupyter_output><empty_output><jupyter_text>Day of week isn't a good predictor of the outcome variable.<jupyter_code>pd.crosstab(df.month,df.y).plot(kind='bar')
plt.title('Purchase Frequency for Month')
plt.xlabel('Month')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_fre_month_bar')<jupyter_output><empty_output><jupyter_text>Month might be a good predictor of the outcome variable.<jupyter_code>pd.crosstab(df.poutcome,df.y).plot(kind='bar')
plt.title('Purchase Frequency for Poutcome')
plt.xlabel('Poutcome')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_fre_pout_bar')<jupyter_output><empty_output><jupyter_text>Poutcome might be a good predictor for the outcome variable.<jupyter_code>pd.crosstab(df.default,df.y).plot(kind='bar')
plt.title('Purchase Frequency for default')
plt.xlabel('default')
plt.ylabel('Frequency of Purchase')
plt.savefig('default_bar')<jupyter_output><empty_output><jupyter_text>Default might be a good predictor for the target variable<jupyter_code>pd.crosstab(df.housing,df.y).plot(kind='bar')
plt.title('Purchase Frequency for housing')
plt.xlabel('housing')
plt.ylabel('Frequency of Purchase')
plt.savefig('housing_bar')<jupyter_output><empty_output><jupyter_text>Housing maybe a moderate predictor for the target variable<jupyter_code>pd.crosstab(df.loan,df.y).plot(kind='bar')
plt.title('Purchase Frequency for loan')
plt.xlabel('loan')
plt.ylabel('Frequency of Purchase')
plt.savefig('loan_bar')<jupyter_output><empty_output><jupyter_text>Loan is a strong predictor for the target variable<jupyter_code>pd.crosstab(df.contact,df.y).plot(kind='bar')
plt.title('Purchase Frequency for contact')
plt.xlabel('contact')
plt.ylabel('Frequency of Purchase')
plt.savefig('contact_bar')<jupyter_output><empty_output><jupyter_text>Therefore contact is also a strong predictor for the target variable<jupyter_code>df.age.hist(bins=20)
plt.title('Histogram of Age')
plt.xlabel('Age')
plt.ylabel('Frequency')
plt.savefig('hist_age')<jupyter_output><empty_output><jupyter_text>Here we can say that maximum bank customers are within 30-40 age group<jupyter_code># Hence dropping some of the categorical values
cols_to_drop=['marital','day_of_week']
df.drop(columns=cols_to_drop,axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>Create Dummy variables<jupyter_code>cat_vars=['job','education','default','housing','loan','contact','month','poutcome']
for var in cat_vars:
    cat_list = pd.get_dummies(df[var], prefix=var)
    data1=df.join(cat_list)
    df=data1
data_vars=df.columns.values.tolist()
to_keep=[i for i in data_vars if i not in cat_vars] #creating a list where only the dummies of categorical values exist not the actual variable
# Data to keep
data_final=df[to_keep]
data_final.columns.values
data_final<jupyter_output><empty_output><jupyter_text>Oversampling Using SMOTE As our training data is created we will up-sample or Oversample the 'subscription' or the 1's in the 'y' class by creating synthetic samples of the minority class(not a copy but a new observation created by knn technique) using SMOTE(Synthetic Minority Oversampling Technique)<jupyter_code># Separating the dataset into X,Y
X = data_final.loc[:, data_final.columns != 'y']
y = data_final['y']

# Implementing Smote
from imblearn.over_sampling import SMOTE
os=SMOTE(random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
print('before using SMOTE--------------')
print('X_train:',X_train.shape)
print('proportion of subscription:{:.2f}'.format(len(y_train[y_train==1])/len(y_train)))
print('proportion of no-subscription:{:.2f}'.format(len(y_train[y_train==0])/len(y_train)))
os_data_X,os_data_y=os.fit_resample(X_train,y_train) # Re-sampling the training data using SMOTE
print('after using SMOTE--------------')
print('os_data_X:',os_data_X.shape)
print('proportion of subscription:{:.2f}'.format(len(os_data_y[os_data_y==1])/len(os_data_y)))
print('proportion of no-subscription:{:.2f}'.format(len(os_data_y[os_data_y==0])/len(os_data_y)))<jupyter_output>before using SMOTE--------------
X_train: (28831, 52)
proportion of subscription:0.11
proportion of no-subscription:0.89
after using SMOTE--------------
os_data_X: (51134, 52)
proportion of subscription:0.50
proportion of no-subscription:0.50<jupyter_text>Recursive Feature Elimnination Recursive Feature eliminiation in short RFE is a feature selection algorithm.* It works by searching a subset of features in the training data set and successfully removing features until the desired number remains.* This is achieved by fitting the given machine learnong algorithm used in the core of the model, ranking features by importance,discarding the least important features and re-fitting the model.This process is repeated until a specified desired number remains.<jupyter_code>data_final_vars=data_final.columns.values.tolist()
y=['y']
X=[i for i in data_final_vars if i not in y]
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
# As we have binary output we are selecting logistic regression as our linear model

logreg = LogisticRegression(solver='liblinear',n_jobs=1)
rfe = RFE(estimator=logreg,n_features_to_select=20)
rfe = rfe.fit(os_data_X, os_data_y.values.ravel())
print(rfe.support_)
print(rfe.ranking_)
l=rfe.support_.tolist()
_=os_data_X.columns.values.tolist()
cols_to_select=[]
for i in range(51):
  if l[i]==True:
    cols_to_select.append(_[i])

print(cols_to_select)
# Now we put the selected features
X=os_data_X[cols_to_select]
y=os_data_y<jupyter_output><empty_output><jupyter_text>Implementing the model<jupyter_code>import statsmodels.api as sm
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary())
# We drop the columns which have nan or p-value>0.95
# Hence we drop 'housing_unknown' and 'loan_unknown'
X.drop(columns=['housing_unknown','loan_unknown'],inplace=True)
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary())
from sklearn import metrics

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
logreg.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Predicting test results and accuracy<jupyter_code>y_pred = logreg.predict(X_test)
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))<jupyter_output>Accuracy of logistic regression classifier on test set: 0.87<jupyter_text>Confusion Matrix<jupyter_code>from sklearn.metrics import confusion_matrix
confusion_matrix = confusion_matrix(y_test, y_pred)
print(confusion_matrix)<jupyter_output>[[7492  174]
 [1778 5897]]<jupyter_text>Compute precision, recall, F-measure and support The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier to not label a sample as positive if it is negative.The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.The F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0.The support is the number of occurrences of each class in y_test.<jupyter_code>from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.81      0.98      0.88      7666
           1       0.97      0.77      0.86      7675

    accuracy                           0.87     15341
   macro avg       0.89      0.87      0.87     15341
weighted avg       0.89      0.87      0.87     15341<jupyter_text>Interpretation: Of the entire test set, 87% of the promoted term deposit were the term deposit that the customers liked. Of the entire test set, 87% of the customer’s preferred term deposits that were promoted.  ROC Curve The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers. The dotted line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner).<jupyter_code>from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
logit_roc_auc = roc_auc_score(y_test, logreg.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)'% logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11026042572936605, 'fraction_numerical': 0.02395452699959399, 'mean_word_length': 5.722027290448343, 'pattern_counts': {'":': 0, '<': 112, '<?xml version=': 0, '>': 113, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '467736', 'license': 'MIT', 'n_tokens_mistral': 5861, 'n_tokens_neox': 5396, 'n_words': 1668, 'path': 'Notebook_1.ipynb', 'repo_name': 'Akash-Chak/google-colab', 'size': '17241'}