<jupyter_start><jupyter_text>Project description- Beta Bank customers are leaving: little by little, chipping away every month. The bankers figured out it’s cheaper to save the existing customers rather than to attract new ones.- We need to predict whether a customer will leave the bank soon. You have the data on clients’ past behavior and termination of contracts with the bank.- Build a model with the maximum possible F1 score. - To pass the project, you need an F1 score of at least 0.59. Check the F1 for the test set.- Additionally, measure the AUC-ROC metric and compare it with the F1.<jupyter_code>#Import all libraries
import pandas as pd
import numpy as np 

import matplotlib.pyplot as plt
import seaborn as sns#visualization
sns.set(style="ticks", color_codes=True)


from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import roc_curve 
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.utils import shuffle
from sklearn.metrics import precision_recall_fscore_support 
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix


#define most used variables
F1_REQUIRED=0.59  #F1 = 2 * (precision * recall) / (precision + recall)
RANDOM_STATE=12345
target_names = ['Exited 0', 'Exited 1']   #Used in printing classification report

import sys
print (sys.version)<jupyter_output>3.7.6 | packaged by conda-forge | (default, Jun  1 2020, 18:57:50) 
[GCC 7.5.0]<jupyter_text>STEP 1 - DATA PREPROCESSING- Download and prepare the data. - Explain the procedure. Data descriptionThe data can be found in /datasets/Churn.csv file. Download the dataset.**Features**- RowNumber — data string index- CustomerId — unique customer identifier- Surname — surname- CreditScore — credit score- Geography — country of residence- Gender — gender- Age — age- Tenure — period of maturation for a customer’s fixed deposit (years)- Balance — account balance- NumOfProducts — number of banking products used by the customer- HasCrCard — customer has a credit card- IsActiveMember — customer’s activeness- EstimatedSalary — estimated salary- Target- Exited — сustomer has left<jupyter_code>#Import the file and create the dataset
path = '/datasets'
df = pd.read_csv(path+'/Churn.csv')
def display_information(df):
    print('Head:')
    print()
    display(df.head())
    print ('-'*100)
    print('Info:')
    print()
    display(df.info())
    print ('-'*100)
    print('Describe:')
    print()
    display(df.describe())
    print ('-'*100)
    display(df.describe(include='object'))
    print()
    print('Columns with nulls:')
    display(get_precent_of_na_df(df,4))
    print ('-'*100)
    print('Shape:')
    print(df.shape)
    print ('-'*100)
    print('Duplicated:')
    print("\033[1m" + 'We have {} duplicated rows.'.format(df.duplicated().sum()) + "\033[0m")
    
def get_precent_of_na_df(df,num):
    df_nulls = pd.DataFrame(df.isna().sum(),columns=['Missing Values'])
    df_nulls['Percent of Nulls'] = round(df_nulls['Missing Values'] / df.shape[0],num) *100
    return df_nulls

def get_percent_of_na(df):
    count = 0
    df = df.copy()
    s = (df.isna().sum() / df.shape[0])
    for column, percent in zip(s.index,s.values):
        
        num_of_nulls = df[column].isna().sum()
        if num_of_nulls == 0:
            continue
        else:
            count += 1
        print('Column {} has {:.{}%} percent of Nulls, and {} of nulls'.format(column, percent,num,num_of_nulls))
        
    if count !=0:
        print('There are {} columns with NA!'.format(count))
    else:
        print()
        print('There are no columns with NA!')
display_information(df)<jupyter_output>Head:<jupyter_text>CONCLUSION:I don't see any outliers. The only data I see problematic is if we have all zeroes in Balance, HasCrCard, NumOfProducts, IsActiveMember.<jupyter_code>df_zerotest = df.query('Balance==0 & HasCrCard==0 & NumOfProducts==0')
df_zerotest.shape<jupyter_output><empty_output><jupyter_text>CONCLUSIONCustomers have atleast one of Balance, Credit Card or NumOfProducts.<jupyter_code>display(df.head(5),df.tail(5))
#Droping 3 columns which will have no use in the Supervised Learning
df=df.drop(['CustomerId','RowNumber','Surname'],axis=1)
df['Tenure'].value_counts()<jupyter_output><empty_output><jupyter_text>CONCLUSION- Dropped three columns: CustomerId, RowNumber, Surname. Not needed.- Checked 'Tenure' column has Nan values.- I need to identify if this column will have impact on the prediction of values.- I will check which columns have Nan Values for Tenure to see if they can be replaced.<jupyter_code>df1 = df[df.isna().any(axis=1)]
print(df1.shape)
display(df1.pivot_table(index=['Exited','Geography','CreditScore','Gender'],values='Balance',aggfunc=['mean','count']))
valueFrance = df[df['Geography']=='France']['Tenure'].median()
valueSpain =  df[df['Geography']=='Spain']['Tenure'].median()
print(valueFrance, valueSpain)
df['Tenure'].fillna(df['Tenure'].median(),inplace=True)
print(df['Tenure'].isnull().sum())<jupyter_output>0<jupyter_text>CONCLUSION- Replaced the null values in Tenure column with median value.- Reason: Tenure has Nan values for the customers in countries: France, Spain.- I calculated the median of Tenure for each of these countries, came out to 5- Hence I replaced the Nan values with the median value which is 5.- No more Nan values  STEP-2: EXPLORATORY DATA ANALYSIS<jupyter_code>heatmap = sns.heatmap(df.corr()[['Exited']].sort_values(by='Exited', ascending=False), vmin=-1, vmax=1, annot=True, cmap='BrBG')<jupyter_output><empty_output><jupyter_text>CONCLUSION- The heatmap depicts that Churn has positive correlation with Age, Balance and Estimated Salary. However, the correlation doesn't seem to be very strong.- Churn (Exited) also has negative correlation with HasCreditCard, tenure, CreditScore,NumOfProducts, IsActiveMember.<jupyter_code>plt.figure(figsize=(10,4))
df.corr()['Exited'].sort_values(ascending = False).plot(kind='bar')<jupyter_output><empty_output><jupyter_text>CONCLUSIONThe BAR Chart also displays that Age,Balance, IsActiveMember, NumOfProducts seem to have high correlation with the churn.<jupyter_code>#Categorical attirbutes churn rate
fig, axis = plt.subplots(2, 2, figsize=(12, 8))
sns.countplot(x= df.Geography, hue = 'Exited' ,data=df, ax =axis[0][0])
sns.countplot(x=df.Gender, hue = 'Exited' ,data=df, ax=axis[1][0])
sns.countplot(x=df.HasCrCard, hue = 'Exited' ,data=df, ax=axis[0][1])
sns.countplot(x=df.IsActiveMember, hue = 'Exited' ,data=df, ax=axis[1][1])
plt.ylabel('count')<jupyter_output><empty_output><jupyter_text>CONCLUSION- Geography: France show a huge number of customers with low churn.- CreditCard: %age of customers without credit card are churning more. There is a much bigger percentage of customers with credit card.- Gender: Female customers are churning more than male. More data is needed to confirm these exploratory findings. There are more male customers.- IsActiveMember: Inactive customers are churning more than active ones. There are more active members than inactive members.  STEP-3: FEATURE ENGINEERING- We have two categorical columns: Geography,Gender- We should OHE the columns in the dataset.<jupyter_code>#Define OHE columns for Supervised Learning
df_ohe=pd.get_dummies(df,drop_first=True)
print(df_ohe.shape)
display(df_ohe.head())<jupyter_output>(10000, 12)<jupyter_text>CONCLUSION- Geography and Gender columns are OHEd. STEP-4: DEFINE TRAINING, VALIDATION AND TEST SETS<jupyter_code>#Set the training, validation and test datasets (features and target)
target = df_ohe['Exited']
features = df_ohe.drop(['Exited'] , axis=1)


#FIRST SPLIT INTO TRAINING(60%) AND VALID_TEST (40%)
features_train, features_validtest, target_train, target_validtest = train_test_split(
    features, target, test_size=0.40, random_state=RANDOM_STATE)

#SPLIT VALID_TEST INTO VALIDATION and TEST (20% each)
features_test, features_valid, target_test, target_valid = train_test_split(
    features_validtest, target_validtest, test_size=0.50, random_state=RANDOM_STATE)

totsize = len(df_ohe)
print('training set  : {0:.0%}'.format(len(features_train) /totsize),features_train.shape, ', training target   :',target_train.shape)
print('validation set: {0:.0%}'.format(len(features_valid)/totsize),features_valid.shape,', validation target :',target_valid.shape)
print('test set      : {0:.0%}'.format(len(features_test)/totsize),features_test.shape,', test target       :',target_test.shape)<jupyter_output>training set  : 60% (6000, 11) , training target   : (6000,)
validation set: 20% (2000, 11) , validation target : (2000,)
test set      : 20% (2000, 11) , test target       : (2000,)<jupyter_text>CONCLUSIONTraining set is 60%, Validation and Test sets are 20% each STANDARDIZE THE NUMERICAL FEATURES<jupyter_code>#Standardize the numerical features.
#CreditScore	Age	Tenure	Balance	NumOfProducts	EstimatedSalary
numeric = ['CreditScore', 'Age', 'Tenure', 'Balance','NumOfProducts', 'EstimatedSalary']

scaler = StandardScaler()

# < transform feature set >
scaler.fit(features_train[numeric])
features_train.loc[:,numeric] = scaler.transform(features_train[numeric])

# < transform validation set >
features_valid.loc[:,numeric] = scaler.transform(features_valid[numeric])

# < transform test set >
features_test.loc[:,numeric] = scaler.transform(features_test[numeric])

print(features_train.shape)
print(features_valid.shape)
print(features_test.shape)

display('TRAINING SET',features_train.head(), 'VALIDATION SET',features_valid.head(),'TEST SET',features_test.head())<jupyter_output>(6000, 11)
(2000, 11)
(2000, 11)<jupyter_text>CONCLUSIONAll data sets are scaled correctly.  BUILD A RESULT DATA FRAME<jupyter_code>#Create a dataframe to store Model name, accuracy results - accuracy score, auc_roc, f1_score for valid, test, and if f1  and if the model met or exceeeded the required F1 score

column_names = ["method", "hyperparameters", "accuracy_score","auc_roc","f1_valid","f1_test",'f1_required','above_f1_threshold?']
df_results = pd.DataFrame(columns = column_names)<jupyter_output><empty_output><jupyter_text>STEP-5: TEST SUPERVISED LEARNING MODELS- LogisticRegression- DecisionTreeClassifier- RandomForestClassifierTry with these options of supervised learning:   - No Hyperparameters  - balanced weights  - Upsampled  - Downsampled<jupyter_code>#Function that accepts the model parameter and fits it to the sent data, then updates the results dataframe and returns the results dataframe
def supervisedModel(model,features_train, target_train,method,hyperparam):
      global df_results,predicted_test,target_test,predicted_valid, target_valid  #Using it in the function, global

      model.fit(features_train, target_train)
      predicted_valid = model.predict(features_valid)
      f1_valid = f1_score(target_valid,predicted_valid)
      
      predicted_test = model.predict(features_test)
      f1_test = f1_score(target_test,predicted_test)
      auc_roc = roc_auc_score(target_test, predicted_test)
      acc=accuracy_score(target_test,predicted_test)

      above_threshold = np.where( f1_test> F1_REQUIRED, True, False)
      resultRowStr= [method,hyperparam,acc,auc_roc,f1_valid,f1_test,F1_REQUIRED,above_threshold]
      rows = [pd.Series(resultRowStr, index=df_results.columns)]

      # append the rows
      df_results=df_results.append(rows,ignore_index=True).round(decimals=4)
      # check the rows 
      display(df_results)<jupyter_output><empty_output><jupyter_text>LogisticRegression (No Hyperparameter)<jupyter_code>method="LogisticRegression"
hyperparam = "none"
model = LogisticRegression(solver='liblinear',random_state=RANDOM_STATE) 
supervisedModel(model,features_train, target_train,method, hyperparam)<jupyter_output><empty_output><jupyter_text>DecisionTreeClassifier with Hyperparameter: Depth<jupyter_code>prevScore=0
foundDepth=0
for depth in range(1, 10, 1):
    model = DecisionTreeClassifier(max_depth=depth, random_state=RANDOM_STATE)
    model.fit(features_train, target_train)
    predicted_valid = model.predict(features_valid)
    scoreR = f1_score(target_valid,predicted_valid)
    
    print("scoreR:",scoreR,"Depth:",depth)
    if (scoreR > prevScore):
        prevScore=scoreR
        foundDepth=depth
print('maxDepth',foundDepth)

method="DecisionTreeClassifier"
hyperparam = "depth: "+str(foundDepth)
model = DecisionTreeClassifier(max_depth=foundDepth,random_state=RANDOM_STATE)
supervisedModel(model,features_train, target_train,method, hyperparam)<jupyter_output>/opt/conda/lib/python3.7/site-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
  'precision', 'predicted', average, warn_for)<jupyter_text>DecisionTreeClassifier with Hyperparameters: balanced weights,Depth<jupyter_code>prevScore=0
foundDepth=0
for depth in range(1, 10, 1):
    model = DecisionTreeClassifier(max_depth=depth, random_state=RANDOM_STATE,class_weight='balanced')
    model.fit(features_train, target_train)
    predicted_valid = model.predict(features_valid)
    scoreR = f1_score(target_valid,predicted_valid)
    
    print("scoreR:",scoreR,"Depth:",depth)
    if (scoreR > prevScore):
        prevScore=scoreR
        foundDepth=depth
print('maxDepth',foundDepth)

method="DecisionTreeClassifier"
hyperparam = "Balanced, depth: "+str(foundDepth)
model = DecisionTreeClassifier(max_depth=foundDepth,class_weight='balanced',random_state=RANDOM_STATE)
supervisedModel(model,features_train, target_train,method, hyperparam)<jupyter_output>scoreR: 0.48514851485148514 Depth: 1
scoreR: 0.5129682997118156 Depth: 2
scoreR: 0.5129682997118156 Depth: 3
scoreR: 0.5385239253852393 Depth: 4
scoreR: 0.5809128630705395 Depth: 5
scoreR: 0.5643835616438356 Depth: 6
scoreR: 0.5593869731800767 Depth: 7
scoreR: 0.5476635514018692 Depth: 8
scoreR: 0.5482625482625482 Depth: 9
maxDepth 5<jupyter_text>LogisticRegression with Hyperparameters: balanced weights<jupyter_code>model = LogisticRegression(solver='liblinear',class_weight='balanced',random_state=RANDOM_STATE)

method="LogisticRegression"
hyperparam="Balanced"
supervisedModel(model,features_train, target_train,method, hyperparam)<jupyter_output><empty_output><jupyter_text>RandomForest with Hyperparameters: n_estimators, depth, balanced weights<jupyter_code>prevScore=0.0
foundDepth=0
for depth in range(1, 16, 1):
    model = RandomForestClassifier(n_estimators=20, max_depth=depth, random_state=RANDOM_STATE,  class_weight='balanced')
    model.fit(features_train, target_train)
    predicted_valid = model.predict(features_valid)
    f1_scor = f1_score(target_valid,predicted_valid)
    print("f1Score:",f1_scor,"Depth:",depth)
    if (f1_scor > prevScore):
        prevScore=f1_scor
        foundDepth=depth
print('maxDepth',foundDepth)

model = RandomForestClassifier(n_estimators=100, max_depth=foundDepth, random_state=RANDOM_STATE,  class_weight='balanced')
method="RandomForestClassifier"
hyperparam = "Balanced, depth: "+str(foundDepth)
supervisedModel(model,features_train, target_train,method, hyperparam)<jupyter_output>f1Score: 0.5306122448979591 Depth: 1
f1Score: 0.558766859344894 Depth: 2
f1Score: 0.5532302092811647 Depth: 3
f1Score: 0.5685685685685686 Depth: 4
f1Score: 0.5897693079237714 Depth: 5
f1Score: 0.5916069600818832 Depth: 6
f1Score: 0.5851619644723092 Depth: 7
f1Score: 0.601536772777168 Depth: 8
f1Score: 0.6045454545454546 Depth: 9
f1Score: 0.5885167464114832 Depth: 10
f1Score: 0.5825977301387139 Depth: 11
f1Score: 0.5594594594594594 Depth: 12
f1Score: 0.5898491083676268 Depth: 13
f1Score: 0.5346820809248556 Depth: 14
f1Score: 0.542728635682159 Depth: 15
maxDepth 9<jupyter_text>CONFUSION MATRIX<jupyter_code>#Confusion Matrix

confusion_matrix = confusion_matrix(target_test, predicted_test)
print(confusion_matrix)<jupyter_output>[[1406  176]
 [ 144  274]]<jupyter_text>- The Confusion Matrix displays that the DecisionTree classifier gave too many False Negatives.- It also shows that the data has a lot of "True Negatives" which implies too many values are No, implying people didn't exit.- Issue is class imbalance. Hence we need to fix the imbalance in the data to improve the model quality.<jupyter_code>print(classification_report(target_test, predicted_test, target_names=target_names))<jupyter_output>precision    recall  f1-score   support

    Exited 0       0.91      0.89      0.90      1582
    Exited 1       0.61      0.66      0.63       418

    accuracy                           0.84      2000
   macro avg       0.76      0.77      0.76      2000
weighted avg       0.84      0.84      0.84      2000<jupyter_text>CONCLUSION- The Exited=0 values are predicted with f1_score of 79%, the Exited=1 values are predicted with f1_score of 48%- We should test models with upsampling of Exited=1, and downsampling of Exited=0  STEP-6: UpSampling, Downsampling<jupyter_code>features_zeros = features_train[target_train==0]
features_ones = features_train[target_train==1]
target_zeros = target_train[target_train==0]
target_ones = target_train[target_train==1]
print(features_zeros.shape)
print(features_ones.shape)
print(target_zeros.shape)
print(target_ones.shape)<jupyter_output>(4804, 11)
(1196, 11)
(4804,)
(1196,)<jupyter_text>- Positive observations are less that negative observations.- Less people have exited. (Exited=1 is less)- We need to do upsampling<jupyter_code>#Create Upsample
repeat = 2
features_upsampled = pd.concat([features_zeros] + [features_ones] * repeat)
target_upsampled = pd.concat([target_zeros] + [target_ones] * repeat)
print(features_upsampled.shape)
print(target_upsampled.shape)
features_upsampled, target_upsampled = shuffle(
        features_upsampled, target_upsampled, random_state=RANDOM_STATE)

#Test Upsample count
features_zeros = features_upsampled[target_train==0]
features_ones = features_upsampled[target_train==1]
target_zeros = target_upsampled[target_train==0]
target_ones = target_upsampled[target_train==1]
print(features_zeros.shape)
print(features_ones.shape)
print(target_zeros.shape)
print(target_ones.shape)
#TRYING DOWNSAMPLING, remove some (Exited=0)

print(features_train.shape)
fraction=0.8
features_downsampled = pd.concat(
        [features_zeros.sample(frac=fraction, random_state=RANDOM_STATE)] + [features_ones])
target_downsampled = pd.concat(
    [target_zeros.sample(frac=fraction,random_state=RANDOM_STATE)] + [target_ones])
features_downsampled, target_downsampled = shuffle(
        features_downsampled, target_downsampled, random_state=RANDOM_STATE)

print(features_downsampled.shape,target_downsampled.shape)
#Test Downsample count
features_zeros = features_downsampled[target_train==0]
features_ones = features_downsampled[target_train==1]
target_zeros = target_downsampled[target_train==0]
target_ones = target_downsampled[target_train==1]
print(features_zeros.shape)
print(features_ones.shape)
print(target_zeros.shape)
print(target_ones.shape)<jupyter_output>(3843, 11)
(2392, 11)
(3843,)
(2392,)<jupyter_text>CONCLUSIONTraining dataset: Upsample and Downsample created. I tested with various values of repeat, frac. The one I am showing above is the one with best f1_scores Improve:  It would be better if functions were used for upsampling/downsampling. Anyway your solution is acceptable.  STEP-7: Evaluating models with Upsampling, Downsampling  LogisticRegression with upsample, Hyperparameters: balanced weights<jupyter_code>#Train the LogisticRegression model with the new data
method="LogisticRegression"
hyperparam="Balanced, upsampled"
model = LogisticRegression(solver='liblinear',class_weight='balanced',random_state=RANDOM_STATE)
supervisedModel(model,features_upsampled, target_upsampled,method, hyperparam)<jupyter_output><empty_output><jupyter_text>DecisionTree with upsample, Hyperparameters: depth, balanced weights<jupyter_code>prevScore=0
foundDepth=0
for depth in range(1, 10, 1):
    model = DecisionTreeClassifier(max_depth=depth, random_state=RANDOM_STATE,class_weight='balanced')
    model.fit(features_upsampled, target_upsampled)
    predicted_valid = model.predict(features_valid)
    scoreR = f1_score(target_valid,predicted_valid)
    
    print("scoreR:",scoreR,"Depth:",depth)
    if (scoreR > prevScore):
        prevScore=scoreR
        foundDepth=depth
print('maxDepth',foundDepth)

method="DecisionTreeClassifier"
hyperparam = "Balanced, upsampled, depth: "+str(foundDepth)
model = DecisionTreeClassifier(max_depth=foundDepth,class_weight='balanced',random_state=RANDOM_STATE)
supervisedModel(model,features_upsampled, target_upsampled,method, hyperparam)<jupyter_output>scoreR: 0.48514851485148514 Depth: 1
scoreR: 0.5129682997118156 Depth: 2
scoreR: 0.5129682997118156 Depth: 3
scoreR: 0.5385239253852393 Depth: 4
scoreR: 0.5809128630705395 Depth: 5
scoreR: 0.5648994515539306 Depth: 6
scoreR: 0.5593869731800767 Depth: 7
scoreR: 0.5471521942110178 Depth: 8
scoreR: 0.5463320463320464 Depth: 9
maxDepth 5<jupyter_text>RandomForest with upsample, Hyperparameters: n_estimators, depth, balanced weights<jupyter_code>from sklearn.ensemble import RandomForestClassifier
prevScore=0.0
foundDepth=0
for depth in range(1, 16, 1):
    model = RandomForestClassifier(n_estimators=20, max_depth=depth, random_state=RANDOM_STATE,  class_weight='balanced')
    model.fit(features_upsampled, target_upsampled)
    predicted_valid = model.predict(features_valid)
    f1_scor = f1_score(target_valid,predicted_valid)
    print("f1Score:",f1_scor,"Depth:",depth)
    if (f1_scor > prevScore):
        prevScore=f1_scor
        foundDepth=depth
print('maxDepth',foundDepth,"f1Score:",f1_scor)

model = RandomForestClassifier(n_estimators=100, max_depth=foundDepth, random_state=RANDOM_STATE,  class_weight='balanced')
method="RandomForestClassifier"
hyperparam = "Balanced,upsampled, depth: "+str(foundDepth)
supervisedModel(model,features_upsampled, target_upsampled,method, hyperparam)<jupyter_output>f1Score: 0.5324298160697 Depth: 1
f1Score: 0.5503116651825468 Depth: 2
f1Score: 0.5672461116193962 Depth: 3
f1Score: 0.584070796460177 Depth: 4
f1Score: 0.5883514313919053 Depth: 5
f1Score: 0.5906432748538012 Depth: 6
f1Score: 0.6125385405960946 Depth: 7
f1Score: 0.5936842105263158 Depth: 8
f1Score: 0.5868644067796609 Depth: 9
f1Score: 0.5873192436040044 Depth: 10
f1Score: 0.6098964326812428 Depth: 11
f1Score: 0.575242718446602 Depth: 12
f1Score: 0.5844980940279542 Depth: 13
f1Score: 0.5876288659793815 Depth: 14
f1Score: 0.5638297872340426 Depth: 15
maxDepth 7 f1Score: 0.5638297872340426<jupyter_text>LogisticRegression with downsample, Hyperparameters: balanced weights<jupyter_code>#Train the LogisticRegression model with the new data
hyperparam="Balanced, downsampled"
supervisedModel(model,features_downsampled, target_downsampled,method, hyperparam)<jupyter_output><empty_output><jupyter_text>DecisionTree with downsample, Hyperparameters: depth, balanced weights<jupyter_code>prevScore=0
foundDepth=0
for depth in range(1, 10, 1):
    model = DecisionTreeClassifier(max_depth=depth, random_state=RANDOM_STATE,class_weight='balanced')
    model.fit(features_downsampled, target_downsampled)
    predicted_valid = model.predict(features_valid)
    scoreR = f1_score(target_valid,predicted_valid)
    
    print("scoreR:",scoreR,"Depth:",depth)
    if (scoreR > prevScore):
        prevScore=scoreR
        foundDepth=depth
print('maxDepth',foundDepth)

method="DecisionTreeClassifier"
hyperparam = "Balanced, downsampled, depth: "+str(foundDepth)
model = DecisionTreeClassifier(max_depth=foundDepth,class_weight='balanced',random_state=RANDOM_STATE)
supervisedModel(model,features_downsampled, target_downsampled,method, hyperparam)<jupyter_output>scoreR: 0.48514851485148514 Depth: 1
scoreR: 0.5129682997118156 Depth: 2
scoreR: 0.5129682997118156 Depth: 3
scoreR: 0.5388349514563108 Depth: 4
scoreR: 0.5794979079497907 Depth: 5
scoreR: 0.558303886925795 Depth: 6
scoreR: 0.5499557913351016 Depth: 7
scoreR: 0.5522935779816514 Depth: 8
scoreR: 0.5434782608695653 Depth: 9
maxDepth 5<jupyter_text>Examine the balance of classes. Train the model without taking into account the imbalance. Briefly describe your findings.Improve the quality of the model. Make sure you use at least two approaches to fixing class imbalance. Use the training set to pick the best parameters. Train different models on training and validation sets. Find the best one. Briefly describe your findings.Perform the final testing.  RandomForest with downsample, Hyperparameters: n_estimators, depth, balanced weights<jupyter_code>from sklearn.ensemble import RandomForestClassifier
prevScore=0.0
foundDepth=0
for depth in range(1, 16, 1):
    model = RandomForestClassifier(n_estimators=20, max_depth=depth, random_state=RANDOM_STATE,  class_weight='balanced')
    model.fit(features_downsampled, target_downsampled)
    predicted_valid = model.predict(features_valid)
    f1_scor = f1_score(target_valid,predicted_valid)
    print("f1Score:",f1_scor,"Depth:",depth)
    if (f1_scor > prevScore):
        prevScore=f1_scor
        foundDepth=depth
print('maxDepth',foundDepth)

model = RandomForestClassifier(n_estimators=100, max_depth=foundDepth, random_state=RANDOM_STATE,  class_weight='balanced')
method="RandomForestClassifier"
hyperparam = "Balanced, downsampled, depth: "+str(foundDepth)
supervisedModel(model,features_downsampled, target_downsampled,method, hyperparam)
#SORT all classifiers by the f1_score on test dataset
df_results = df_results.sort_values(by='f1_test',ascending=False)
df_results
df_results.plot(y=['f1_test','accuracy_score','auc_roc'], kind="bar", stacked=True,figsize=(12,6)).legend(loc='best')
plt.title("Methods and Scores")
plt.xlabel("Method")
plt.ylabel("Scores")<jupyter_output><empty_output><jupyter_text>STEP-8: OVERALL CONCLUSION- From the above table, the best classifier I found is   - RandomForestClassifier	Balanced, depth: 9  - Its f1_score is .6313, accuracy_score> 0.84, auc_roc: .77.  - I will print the auc_roc curve and precision_recall_curve for this classifier.  ROC curve, Precision-Recall curve for the BEST MODELRandomForestClassifier with n_estimators=100,max_depth=9, class_weight='balanced'<jupyter_code>model = RandomForestClassifier(n_estimators=100, max_depth=9, random_state=RANDOM_STATE,  class_weight='balanced')
model.fit(features_train, target_train)
probabilities_test = model.predict_proba(features_test)
probabilities_one_test = probabilities_test[:, 1]
print(probabilities_one_test[:5])
#roc curve
auc_roc = roc_auc_score(target_test, probabilities_one_test)

print(auc_roc)

fpr, tpr, thresholds = roc_curve(target_test,probabilities_one_test)

plt.figure()
# < plot the graph >
plt.plot(fpr,tpr)
# ROC curve for random model (looks like a straight line)
plt.plot([0, 1], [0, 1], linestyle='--')
plt.xlim([0.1,1.0])
plt.ylim([0.0,1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve')
plt.show()
#precision-recall curve
precision, recall, thresholds = precision_recall_curve(target_test, probabilities_test[:, 1])
plt.figure(figsize=(6, 6))
plt.step(recall, precision, where='post')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('Precision-Recall Curve')
plt.show()<jupyter_output><empty_output><jupyter_text>CONCLUSION- We have imbalanced data with more 0 Churn values than 1 values.- Using an ROC curve with an imbalanced dataset might be deceptive and lead to incorrect interpretations of the model skill. ROC graphs are based upon TP rate and FP rate, in which each dimension is a strict columnar ratio, so do not depend on class distributions.- The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.- Our ROC curve, and Precision_recall curve depict that our model is very good. - The ROC curve is much higher than the threshold, and the precision_recall curve is high.  SANITY CHECK- To perform sanity check, lets see if the model has better performance than random classification, or classification for just one class (the majority).- Lets find the larger of the two classes:- We can measure the **precision** of our model (predictions correct / total correct )  SANITY CHECK ON ENTIRE DATASET<jupyter_code>class_frequency = df['Exited'].value_counts(normalize=True)
print(class_frequency)
class_frequency.plot(kind='bar')<jupyter_output>0    0.7963
1    0.2037
Name: Exited, dtype: float64<jupyter_text>- Lets choose the retained clients since that is the larger of the two.- Our best model products a test accuracy of 81%, lets see if the model outperforms the random classification<jupyter_code>target = df['Exited']
features = df.drop('Exited', axis=1)

target_pred_constant = pd.Series([1 for x in range(len(target))], index=target.index)
print('Accuracy for the entire dataset of Churn Customers',accuracy_score(target,target_pred_constant))<jupyter_output>Accuracy for the entire dataset of Churn Customers 0.2037<jupyter_text>SANITY CHECK ON TEST DATASET<jupyter_code>stay_target = (target_test == 0)
exit_target = (target_test == 1)

print("Number of retained customer clients", stay_target.sum())
print("Number of churn clients", exit_target.sum())
accuracy_check = stay_target.sum() / target_test.shape[0]
print("Accuracy of the Retained customer classifier", accuracy_check )
print("Accuracy of the Churn customer classifier", exit_target.sum() / target_test.shape[0] )
print(classification_report(target_test, predicted_test, target_names=target_names))<jupyter_output>precision    recall  f1-score   support

    Exited 0       0.92      0.86      0.89      1582
    Exited 1       0.57      0.70      0.63       418

    accuracy                           0.82      2000
   macro avg       0.74      0.78      0.76      2000
weighted avg       0.84      0.82      0.83      2000
{'fraction_non_alphanumeric': 0.09983806470802076, 'fraction_numerical': 0.055256287385571236, 'mean_word_length': 6.194484070375654, 'pattern_counts': {'":': 0, '<': 114, '<?xml version=': 0, '>': 123, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '12097', 'license': 'CC0-1.0', 'n_tokens_mistral': 11064, 'n_tokens_neox': 9825, 'n_words': 2459, 'path': 'BankCustomerChurn/BankCustomerChurn_MachineLearningModel.ipynb', 'repo_name': 'dsdsgithub007/dsds007', 'size': '30259'}