<jupyter_start><jupyter_text>DescriptionCar insurance offers protection against accidents, theft, natural calamities & other unforeseen circumstances that could harm your vehicle. Having a good motor insurance policy ensures that you and your car are protected from the financial losses of these unfortunate events.Government has also made it compulsory to have vehicle insurance to drive cars on roads.  Problem StatementImagine you are a data scientist at a bank in your city. Your bank besides usual services also provides car insurance services. The bank has potential customers data and bank employees call them for advertising available car insurance options.You are provided with general information about clients (age,job,etc.) as well as more specific information about the current insurance sell campaign (communication,last contact day) and previous campaign (attributes like previous attempts,outcome etc).You have data about 3100 customers who were contacted during the last campaign and for whom the results of the campaign (if the customer bought insurance or not) are known.  ObjectiveTo build a machine learning model to predict if these customers will buy the car insurance or not. Data DescriptionYou are provided 935 customers data who were contacted during the current campaign. There are **19** variables (including the target variable) in the train dataset. Below are some of the listed variables and their description: - age: age of the customer- job_type: what job does customer do- marital_status: if the customer is married or not- education_level: education level of the customer- default_or_not: if the customer was able to repay in time or not- balance_amt: balance amount of the customer in the bank- hoursehold_insurance: if the customer has bought household insurance or not- car_loan: if the customer has car loan- communication: mode of communication- last_contact_day: when the customer was last contacted (day)- last_contact_month: when the customer was last contacted (month)- no_of_contacts: How many times was the customer contacted- days_passed: days passed since the last contact (-1 means the data is missing)- prev_attempts: how many times was the customer previously contacted - outcome: what was the outcome of the last contact- call_start: call start time- call_end: call end time- **car_insurance: if the customer bought car insurance (the target variable, 1 - bought, 0 - not bought)**<jupyter_code># Import libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import auc, classification_report, roc_auc_score
from sklearn.model_selection import train_test_split
from lightgbm import LGBMClassifier
from sklearn.ensemble import RandomForestClassifier
pd.set_option('max_colwidth', 500)
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
# Load files
train = pd.read_csv('/content/Train_data.csv')
test = pd.read_csv('/content/Test_data.csv')
samplesubmission = pd.read_csv('/content/Sample_Submission.csv')

# Preview the first five rows of the train set
train.head()
# Preview the first five rows of the test set
test.head()
# Preview the first five rows of the sample submission file
samplesubmission.head()
# Check the shape of the train and test sets
print(f'The shape of the train set is: {train.shape}\nThe shape of the test set is: {test.shape}')
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3102 entries, 0 to 3101
Data columns (total 18 columns):
 #   Column               Non-Null Count  Dtype  
---  ------               --------------  -----  
 0   age                  3102 non-null   float64
 1   job_type             3091 non-null   object 
 2   marital_status       3102 non-null   object 
 3   education_level      2989 non-null   object 
 4   default_or_not       3102 non-null   int64  
 5   balance_amt          3102 non-null   float64
 6   household_insurance  3102 non-null   int64  
 7   car_loan             3102 non-null   int64  
 8   communication        2559 non-null   object 
 9   last_contact_day     3102 non-null   int64  
 10  last_contact_month   3102 non-null   object 
 11  no_of_contacts       3102 non-null   int64  
 12  days_passed          3102 non-null   int64  
 13  prev_attempts        3102 non-null   int64  
 14  Outcome              751 non-null    object 
 15  call_start           3102 non-null   o[...]<jupyter_text>Check for missing values<jupyter_code>print(train.isnull().sum())
print("*"*100)
print(test.isnull().sum())
# Check if there any missing values in train set
ax = train.isna().sum().sort_values().plot(kind = 'barh', figsize = (9, 10))
plt.title('Percentage of Missing Values Per Column in Train Set', fontdict={'size':15})
for p in ax.patches:
    percentage ='{:,.0f}%'.format((p.get_width()/train.shape[0])*100)
    width, height =p.get_width(),p.get_height()
    x=p.get_x()+width+0.02
    y=p.get_y()+height/2
    ax.annotate(percentage,(x,y))
# Check if there missing values in test set
ax = test.isna().sum().sort_values().plot(kind = 'barh', figsize = (9, 10))
plt.title('Percentage of Missing Values Per Column in Test Set', fontdict={'size':15})

for p in ax.patches:
    percentage ='{:,.1f}%'.format((p.get_width()/test.shape[0])*100)
    width, height =p.get_width(),p.get_height()
    x=p.get_x()+width+0.02
    y=p.get_y()+height/2
    ax.annotate(percentage,(x,y))<jupyter_output><empty_output><jupyter_text>1. The **outcome column** has quite a large number of missing values. We really can't fill the values since the outcome of the calls highly vary and are unique from one customer to the other.2. **Communication, Education level and Job type** columns can be filled with the mode of the column since it makes a more resonable assumption.   Combine train and test set for easy preprocessing<jupyter_code># Combine train and test set
ntrain = train.shape[0] # to be used to split train and test set from the combined dataframe

all_data = pd.concat((train, test)).reset_index(drop=True)
print(f'The shape of the combined dataframe is: {all_data.shape}')
# Check the column names and datatypes
all_data.info()
#Dropping the outcome column
all_data.drop('Outcome', axis=1, inplace=True)
# Convert Object to Category columns
cat_cols = all_data.select_dtypes(include=['object'],).columns
# Change columns to their respective datatypes
all_data[cat_cols] = all_data[cat_cols].astype('category')
# Confirm whether the changes have been successful
all_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 4037 entries, 0 to 4036
Data columns (total 17 columns):
 #   Column               Non-Null Count  Dtype   
---  ------               --------------  -----   
 0   age                  4037 non-null   float64 
 1   job_type             4021 non-null   category
 2   marital_status       4037 non-null   category
 3   education_level      3890 non-null   category
 4   default_or_not       4037 non-null   int64   
 5   balance_amt          4037 non-null   float64 
 6   household_insurance  4037 non-null   int64   
 7   car_loan             4037 non-null   int64   
 8   communication        3329 non-null   category
 9   last_contact_day     4037 non-null   int64   
 10  last_contact_month   4037 non-null   category
 11  no_of_contacts       4037 non-null   int64   
 12  days_passed          4037 non-null   int64   
 13  prev_attempts        4037 non-null   int64   
 14  call_start           4037 non-null   category
 15  call_end             [...]<jupyter_text>Fill in missing values<jupyter_code># Fill in missing values
# For cat cols and date cols fill in with mode and for num cols fill in with 9999
for col in all_data.columns:
  if col in cat_cols:
    all_data[col] = all_data[col].fillna(all_data[col].mode()[0])

# Confirm that there aren't any missing values
all_data[all_data.columns.difference(['car_insurance'])].isna().sum().any()<jupyter_output><empty_output><jupyter_text>Checking for Outliers<jupyter_code># Checking for Outliers
#

Q1 = all_data.quantile(0.25)
Q3 = all_data.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
#Plotting the box plot for columns with outliers

col_names = ['age','balance_amt','last_contact_day', 'days_passed', 'no_of_contacts']

fig,ax = plt.subplots(len(col_names),figsize=(8,40))
for i , col_val in enumerate(col_names):
  sns.boxplot(y=all_data[col_val],ax=ax[i])
  ax[i].set_title('Box plot - {}'.format(col_val), fontsize =10)
  ax[i].set_xlabel(col_val,fontsize=10)
plt.show()<jupyter_output><empty_output><jupyter_text>The outliers kinda make sense thus will leave them as they are.<jupyter_code>#creating a copy of our data
copy_1 = all_data.copy()
#Checking for duplicates
print("The duplicates are {} ".format(all_data.duplicated().sum()))
#dropping the duplicates
all_data.drop_duplicates(inplace = True)
#Checking for duplicates
print("Confirming the duplicates are {} ".format(all_data.duplicated().sum()))
all_data.drop(columns=['call_start', 'call_end'], axis=1,inplace=True)
all_data.info()
all_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 3737 entries, 0 to 4036
Data columns (total 17 columns):
 #   Column               Non-Null Count  Dtype   
---  ------               --------------  -----   
 0   age                  3737 non-null   float64 
 1   job_type             3737 non-null   category
 2   marital_status       3737 non-null   category
 3   education_level      3737 non-null   category
 4   default_or_not       3737 non-null   int64   
 5   balance_amt          3737 non-null   float64 
 6   household_insurance  3737 non-null   int64   
 7   car_loan             3737 non-null   int64   
 8   communication        3737 non-null   category
 9   last_contact_day     3737 non-null   int64   
 10  last_contact_month   3737 non-null   category
 11  no_of_contacts       3737 non-null   int64   
 12  days_passed          3737 non-null   int64   
 13  prev_attempts        3737 non-null   int64   
 14  call_start           3737 non-null   category
 15  call_end             [...]<jupyter_text>Feature Engineering1. Some columns don't make sense to be used as is. We'll modify them then preprocess2. We'll get dummies of the categorical columns<jupyter_code>all_data.head(3)
all_data.info()
# Shape of data before encoding
all_data.shape
#Correlation matrix.
#Plottting a heatmap to show correlation of different numerical variables
sns.heatmap(all_data.corr(), xticklabels = all_data.corr().columns, yticklabels = all_data.corr().columns, annot=True)
# Checking for multicollinearity on the independent variables only

F = all_data.drop('car_insurance', 1)
correlations = F.corr()

# Computing the Variance Inflation Factor(VIF)
# We create a new dataframe with the INVERSE of the correlations above.
pd.DataFrame(np.linalg.inv(correlations.values), index = correlations.index, columns=correlations.columns)<jupyter_output><empty_output><jupyter_text>There's neither a high correlation nor multicollinearity.<jupyter_code># Use one hot encoding to turn categorical features to numerical features
# Encode categorical features
all_data2 = all_data.copy()
all_data = pd.get_dummies(data = all_data)
all_data.head()
# Shape of data after encoding
all_data.shape
# Separate train and test data from the combined dataframe
train_df = all_data[:ntrain]
test_df = all_data[ntrain:]

# Check the shapes of the split dataset
train_df.shape, test_df.shape
# Select main columns to be used in training

y = train_df.car_insurance.astype(int)
X = train_df.drop('car_insurance', axis=1)
y = train_df.car_insurance.astype(int)

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3, random_state=42)

# Train model
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict_proba(X_test)[:, 1]

# Check the auc score of the model
print(f'RandomForest AUC score on the X_test is: {roc_auc_score(y_test, y_pred)}\n')

# print classification report
print(classification_report(y_test, [1 if x >= 0.5 else 0 for x in y_pred]))
# Import the model
from sklearn.ensemble import RandomForestClassifier

# The random_state variable is here to guarantee that we all see the same numbers.
forest = RandomForestClassifier(n_estimators = 100, random_state=42, min_samples_split = 10, max_depth=5)

forest = forest.fit(X_train, y_train)

# Predict based on the model we've trained
y_pred = forest.predict(X_test)


comparison_frame = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})

comparison_frame.head(5)
from sklearn.metrics import f1_score
from sklearn.metrics import r2_score, accuracy_score
print("R2 score : %.2f" % r2_score(y_test,y_pred))

print('accuracy_score:',accuracy_score (y_test , y_pred))

print('F1_score:',f1_score(y_test, y_pred))

# Visualizing tree number 25

from sklearn.tree import export_graphviz
from sklearn.externals.six import StringIO  
from IPython.display import Image  
import pydotplus

y_pred = forest.predict(X_test)
dot_data = StringIO()

# picking a specific tree from the forest; 

tree = forest.estimators_[25]

export_graphviz(tree, out_file=dot_data,  
                filled=True, rounded=True,
                special_characters=True,feature_names = X.columns)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
graph.write_png('Hypothyroid_Test.png')
Image(graph.create_png())
# Getting the feature importance

df = pd.DataFrame({'Features' : X.columns, 'Importance' : forest.feature_importances_})\
.sort_values(by = 'Importance', ascending = False).set_index('Features')
df
from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

params = {'n_estimators' : [250, 300, 450, 500, 600]}

ada_boost = AdaBoostClassifier()

search = GridSearchCV(ada_boost, scoring = 'accuracy', param_grid = params, cv = 10)

search.fit(X, y)

search.best_params_

ada_boost = AdaBoostClassifier(DecisionTreeClassifier(max_depth=4),
                          n_estimators=600, random_state=5)
X_train.drop('call_duration', axis=1)
ada_boost = ada_boost.fit(X_train, y_train)

y_pred_ada = ada_boost.predict(X_test)

print("Accuracy score:", accuracy_score(y_test, y_pred_ada))

print('F1_score:',f1_score(y_test, y_pred))
# The accuracy score is 99.5 which is a very high score<jupyter_output><empty_output><jupyter_text>Train different model and compare results<jupyter_code># Train model
model = LGBMClassifier()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict_proba(X_test)[:, 1]

# Check the auc score of the model
print(f'LGBM AUC score on the X_test is: {roc_auc_score(y_test, y_pred)}\n')

# print classification report
print(classification_report(y_test, [1 if x >= 0.5 else 0 for x in y_pred]))<jupyter_output>LGBM AUC score on the X_test is: 0.8065630642127252

              precision    recall  f1-score   support

           0       0.69      0.69      0.69       394
           1       0.77      0.77      0.77       537

    accuracy                           0.74       931
   macro avg       0.73      0.73      0.73       931
weighted avg       0.74      0.74      0.74       931<jupyter_text>- There is a light improvement when using LGBMClassifier[More on AUC score](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc:~:text=AUC%20represents%20the%20probability%20that,has%20an%20AUC%20of%201.0.)  Making predictions of the test set and creating a submission file<jupyter_code># test_df.columns = ['call_duration']
# test_df['call_duration'] = pd.to_datetime(test_df['call_duration'])
test_df['call_duration']=test_df['call_duration'].map(dt.datetime.toordinal)
# Make prediction on the test set
test_df = all_data[ntrain:]
test_df = test_df.drop(labels=['car_insurance'], axis=1)

predictions = ada_boost.predict(test_df)

# # Create a submission file
sub_file = samplesubmission.copy()
sub_file.target = predictions

# # Check the distribution of your predictions
sns.countplot([1 if x >= 0.5 else 0 for x in sub_file.target])
plt.title('Predicted Variable Distribution');
# Create a csv file and upload to zindi 
sub_file.to_csv('Baseline.csv', index = False)
sub_file.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09037990196078431, 'fraction_numerical': 0.035355392156862744, 'mean_word_length': 3.911525729762263, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '636629', 'license': 'MIT', 'n_tokens_mistral': 5493, 'n_tokens_neox': 5045, 'n_words': 1630, 'path': 'Car_Insurance_prediction.ipynb', 'repo_name': 'RuthNduta/Car-insurance-prediction', 'size': '16320'}