<jupyter_start><jupyter_text>Finding Donors for *CharityML* In this project, I try to predict if a person makes more than 50k or not. This is basically a binary classification problem; ie. if the outcome is 0 or 1. This information is important for charities since they generally incur a cost of mailing to reach out potential donors. And in return, they may get donation from these people. The classification of if the person makes more than 50k or not effects the probability of getting donations. So charity organizations can make use of the model to predict the outcomes to their advantage to determine which people they mail their invitations and do not have to incur the cost of mailing with high probability of no donations.To be able to predict the outcome, some demographic variables collected from the 1994 Census Data will be used. I get the data from UCI Machine Learning Repository and clean it to be able to use for machine learning.  Data Preparation Data is contained in 2 seperate files on the repository. I combine them and drop the 'fnlwgt' column. There are 2809 missing values in 'workclass' and 'occupation' columns and I dropped rows with missing data.<jupyter_code># importing necessary libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display

%matplotlib inline
df = pd.read_csv('adult.csv', header = None, na_values = ' ?')
df2 = pd.read_csv('adult2.csv', header = None, na_values = ' ?')
df.drop(2, axis = 1, inplace = True)
df.head()
df2.drop(2, axis = 1, inplace = True)
df2.head()
cols = ['age', 'workclass', 'education_level', 'education_num', 'marital_status', 'occupation', 'relationship',
       'race', 'sex', 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country', 'income']
df.columns = cols
df2.columns = cols
data = pd.concat([df, df2], ignore_index = True)
display(data.head())
data.shape
data.isna().sum()<jupyter_output><empty_output><jupyter_text>I plot the histograms of missing data rows to see if there is any pattern in the data. But it seems no much different than the histograms of full data which is given below other than age variable. So I dropped all samples with missing data.<jupyter_code>data[data.workclass.isna()].hist(figsize = (8,6));
data = data.dropna()
data.shape
display(data.head())
data.income.unique()
# <=50K and <=50K. are the same things but dot in the second representation 
#makes the column unclean.

#changing <=50K. with <=50K and >50K. with  >50K

data.income = np.where(data.income == ' <=50K.', ' <=50K', data.income )
data.income = np.where(data.income == ' >50K.', ' >50K', data.income )
# now the column is clean
data.income.unique()
# coding the column as 0 and 1
data['income_bin'] = np.where(data.income == ' >50K', 1, 0)<jupyter_output><empty_output><jupyter_text>Data Exploration<jupyter_code>data.shape
n_records = data.shape[0]
n_greater_than_50k = data[data.income_bin == 1].shape[0]
n_at_most_50k = data[data.income_bin == 0].shape[0]
greater_percent = data.income_bin.mean()

print('Total number of records {}\n'.format(n_records),
     'Individuals making more than 50k : {}\n'.format(n_greater_than_50k),
     'Individuals making at most 50k : {}\n'.format(n_at_most_50k),
     'The percentage of people making more then 50k is {:.2f}%\n'.format(greater_percent*100))
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 45222 entries, 0 to 48841
Data columns (total 15 columns):
age                45222 non-null int64
workclass          45222 non-null object
education_level    45222 non-null object
education_num      45222 non-null int64
marital_status     45222 non-null object
occupation         45222 non-null object
relationship       45222 non-null object
race               45222 non-null object
sex                45222 non-null object
capital_gain       45222 non-null int64
capital_loss       45222 non-null int64
hours_per_week     45222 non-null int64
native_country     45222 non-null object
income             45222 non-null object
income_bin         45222 non-null int32
dtypes: int32(1), int64(5), object(9)
memory usage: 5.3+ MB<jupyter_text>There are 10 categorical variables and 5 continuous variables.<jupyter_code>sns.set_style('whitegrid');
data.hist(figsize = (12, 9), bins = 15);
sns.pairplot(data, hue = 'income_bin');<jupyter_output>C:\Users\ilknur\Anaconda3\lib\site-packages\statsmodels\nonparametric\kde.py:488: RuntimeWarning: invalid value encountered in true_divide
  binned = fast_linbin(X, a, b, gridsize) / (delta * nobs)
C:\Users\ilknur\Anaconda3\lib\site-packages\statsmodels\nonparametric\kdetools.py:34: RuntimeWarning: invalid value encountered in double_scalars
  FAC1 = 2*(np.pi*bw/RANGE)**2<jupyter_text>The distribution of age for people who make more than 50k is centered around a higher value and has a higher peak. It means that if the person is young, it is much more probable that he makes at most 50k whereas if the peron is old, it is much more probable that he makes more than 50k.The distribution of education_num variable including only people who make more than 50k seems to be bimodal. The centers are around 12.5 and 9 years. When we compare two distributions, it seems that the one for people who make more is shifted to the right meaning the range and clusters are more to the right. It basically says that this group consists of people with more education years.The capital_gain for people who make more than 50k is heavily right skewed. It is not the case for other distribution of capital_gain.People making more than 50k who work less than 40 hours a week is a small share in total. The distribution has a very thin tail before 40, whereas the hours_per_week distribution for people who make at most 50k has a considerable amount for values smaller than 40. It means that most people making more than 50k work 40 hours or more in a week<jupyter_code>fig, ax = plt.subplots(1, 2,figsize = (12,6))
sns.distplot(data['capital_gain'], ax = ax[0]);
sns.distplot(data['capital_loss'], ax = ax[1]);
plt.title('The Distribution of Capital Gain');
ax[0].set_title('The Distribution of Capital Gain');
ax[1].set_title('The Distribution of Capital Loss');<jupyter_output><empty_output><jupyter_text>capital_gain and capital_loss are right-skewed variables. I apply log-transformation to get more symmetrical distributions. Most of the people work in Private workclass and only around 20% of them make greater than 50k. Almost 60% of people in Self-emp-inc make more than 50k and the number of people in this category is a small share in total.When we look at education-level, most of the people have Hs-grad, some-college and bachelors degree. More than half of people who are in Prof-school, doctorate and masters education categories make more than 50k.Similarly, half of the people in exec-magerial and prof-specialty and around 30% of people in protective-service, tech-support and armed-forces occupation categories make more than 50k.There are small percentage of samples with race Asian-pac-islander, Amer-Indian-Eskimo and Other races. 25% of Asian people and a little more than 12% of people from Amer-Indian-Eskimo and Other races make more than 50k. Most of the people in our dataset are white. While 25% of white people make more than 50k, this number is around 12% for black people.The dataset has more males than females. 30% of males and only 10% of females make more than 50k.The relevant graphs are given below.<jupyter_code>sns.countplot(data = data, y = 'workclass', hue = 'income_bin');
base_color = sns.color_palette()[0]
sns.barplot(data = data, y = 'workclass', x = 'income_bin', ci = None, color = base_color);
sns.countplot(data = data, y = 'education_level', hue = 'income_bin');
order = data.groupby('education_level').income_bin.mean().sort_values(ascending = False).index
sns.barplot(data = data, y = 'education_level', x = 'income_bin', ci = None, color = base_color, order = order);
sns.countplot(data = data, y = 'occupation', hue = 'income_bin');
order = data.groupby('occupation').income_bin.mean().sort_values(ascending = False).index
sns.barplot(data = data, y = 'occupation', x = 'income_bin', ci = None, color = base_color, order = order);
sns.barplot(data = data, y= 'race', x = 'income_bin', color = base_color, ci = None);
sns.countplot(data = data, y= 'race', hue = 'income_bin');
plt.legend(loc = 'lower right');
sns.countplot(data = data, y= 'sex', hue = 'income_bin');
plt.legend(loc = 'lower right');
sns.barplot(data = data, x = 'sex', y = 'income_bin', ci = None);
plt.title('The percentage of people making more than 50k');
corr = data.corr()
mask = np.zeros_like(corr, dtype = np.bool)
mask[np.triu_indices_from(mask)] = True
plt.figure(figsize = (7,7));
cmap = sns.diverging_palette(220, 10, as_cmap = True)
sns.heatmap(corr, mask = mask, cmap = cmap, vmax = 0.3, center = 0, square = True, annot = True, fmt = '.2f',
            linewidths = 0.5, cbar_kws = {'shrink':0.5});
plt.yticks(fontsize = 12);
plt.xticks(fontsize = 12);
plt.title('Correlation of Variables', fontsize = 13);<jupyter_output><empty_output><jupyter_text>The correlation matrix of continuous variables is given above. Only the lower half is shown since the matrix is symmetric. Education_num has the highest correlation coefficient , 0.33 with income_bin variable which is 1 if the person makes more than 50k. Then comes age, hours_per_week and capital_gain that are 0.24, 0.23 and 0.22, respectively. The correlation coefficient of capital_loss is 0.15.  Data Preprocessing<jupyter_code># seperating the data as features and labels; X and y
y = data['income_bin']
data.drop(['income', 'income_bin'], axis = 1, inplace = True)
# getting dummies for categorical variables
df = pd.get_dummies(data)
display(df.head())
X = df.copy()<jupyter_output><empty_output><jupyter_text>Transforming Skewed Continuous Features<jupyter_code># log-transforming capital_gain and capital_loss
to_be_logtransformed = ['capital_gain', 'capital_loss']

X[['capital_gain_transformed', 'capital_loss_transformed']] = X[to_be_logtransformed].apply(lambda x: np.log10(x+1))
fig, ax = plt.subplots(1, 2,figsize = (12,6))
sns.distplot(X['capital_gain_transformed'], kde = False, ax = ax[0]);
sns.distplot(X['capital_loss_transformed'], kde = False, ax = ax[1]);
plt.title('The Distribution of Capital Gain');
ax[0].set_title('The Distribution of Capital Gain (Transformed)');
ax[1].set_title('The Distribution of Capital Loss (Transformed)');<jupyter_output><empty_output><jupyter_text>The graphs are not very symmetrical, but the range of values are now smaller on the x-axis. Around 41400 out of 45000 samples have capital gain equal to zero. So in such a case, we cannot expect much symmetry even after log-transformation.<jupyter_code># dropping capital_gain and capital_loss 
X.drop(to_be_logtransformed, axis = 1, inplace = True)
display(X.head())<jupyter_output><empty_output><jupyter_text>Normalizing Numerical Features<jupyter_code>#scaling numerical vaiables
from sklearn.preprocessing import MinMaxScaler
num_cols = ['age', 'education_num', 'hours_per_week', 'capital_gain_transformed', 'capital_loss_transformed']
scaler = MinMaxScaler()

X[num_cols] = scaler.fit_transform(X[num_cols])
display(X.head())
print('{} total features after one hot encoding'.format(X.shape[1]))<jupyter_output>103 total features after one hot encoding<jupyter_text>Shuffle and Split the Data<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 42)

print('Train set has {} samples\n'.format(X_train.shape[0]),
     'Test set has {} samples'.format(X_test.shape[0]))<jupyter_output>Train set has 36177 samples
 Test set has 9045 samples<jupyter_text>Naive Predictor Performance If we choose a model that always predicts outcome as 1, then the model catches all cases with people making more than 50k. It  means recall will be 100% for this model. We  know that 24.78% of total people make more than 50k, so the accuracy will be close to that number for test data. Out of all people, the people model predicts as making more than 50k, only around 25% of them actually make more than 50k. It is basically the definition of the precision score, so precision will be around 25%. In this case with 100% recall and 25% precision, fbeta score with beta equals 0.5 will be close to precision score, ie. 25%. The actual figures calculated below supports the reasoning here.<jupyter_code>tp, fp, tn, fn = (y == 1).sum(), (y == 0).sum(), 0, 0

accuracy = (tp+tn)/(tp+fp+tn+fn)
recall = tp/(tp+fn)
precision = tp/(tp+fp)

# calculating f-score for beta = 0.5
beta = 0.5
fscore = (1+ beta**2)*(precision * recall) / ((beta**2)*precision + recall)

print(' Accuracy: {:.2f}% \n'.format(accuracy*100),
     'Recall: {:.2f}% \n'.format(recall*100),
     'Precision: {:.2f}% \n'.format(precision*100),
     'Fbeta-score: {:.2f}% \n'.format(fscore*100))<jupyter_output>Accuracy: 24.78% 
 Recall: 100.00% 
 Precision: 24.78% 
 Fbeta-score: 29.17%<jupyter_text>Supervised Learning Models  Model Application I pick Logistic Regression, random forest and gradient boosting for this problem. Each model has its own strengths and weaknesses and characteristics that make it appropriate or not for some problems. Let's look at each model.  Logistic Regression Logistic Regresion can learn both linear and non-linear relationships between input and output variables. It is possible to avoid overfitting by Lasso l1 and Ridge l2 regularization. It does not require large data. Logistic regression is a probabilistic approach that give predicted probabilities of how likely a person's income is greater than or equal to 50K. It classifies classes as True if the probability is greater than a preset threshold, and False if it is smaller than the threshold. It also gives the relationship between labels and features. If you specifically want to look at and understand the relationship between labels and features so that you can take action in advance, logistic regression is a good model to use. It is used for credit risk scoring and lift curves. However, it has single linear decision surface that's why it is called linear model. So if it is best to use complex, non-linear decision surfaces for the data, logistic regression might perform poorly. It is a good candidate for this problem since first of all, the problem is a classification problem and we will have a chance to understand the relationships between features and labels if we want to use them to our advantage.  Random Forest Decision trees are simple yet powerful machine learning technique. It partitiones the space iteratively that gives highest information gain. Even though it partitioned linearly, it captures non-linear and complex relationships between labels and features nicely. Another feature that makes decision trees powerful tchnique is that they are very easy to understand; it consists of subsequent yes/no questions and keeps branching. One disadvantage of decision trees is that they are prone to overfitting. It keeps branching until it memorizes the data at the end the configuration of the tree cannot generalizes well to unseen data. To reduce variance, there are ensemble methods one of which is random forest. Random Forest is a tree based ensemble method. It uses a number of trees and combines the predictions of these trees by majority voting to label the data in classification problems. It introduces randomness by bootstrapping the data. It draws data points with replacement so while some points might be drawn several times some points are not drawn at all. It makes data each tree trained on different. It further randomizes the process by features bootstrapping for each tree to use in node splitting. How simple or complex these trees will be is a thing to decide when tuning hyperparameters of the model.One disadvantage of random forestmodel is lack of interpretebility. If you want to understand the explanation of prediction, random forest might not be a good choice. Even though it is possible to get feature importance from the model, feature importance does not give how (linear or non-linear) and how much each feature affects the dependent variable. Also we might say the fact that we need to tune a lot of hyprparameters for a better performance is a disadvantage. It also comes at a computational cost. It increases train and prediction times compared to Logistic Regression.In this dataset, we do not specifically try tto understand the relationship between dependent and independent variables. We care most about classifiying if the person earns more than 50K or not. We want high performance with less need for interpretation. So random forest is a good candidate for the problem.Random Forest is used in multi-class object detection in computer vision and gaming.   Gradient Boosting Gradient Boosting is like Random Forest a tree based ensemble method. In Gradient Boosting trees are trained sequentially and each tree is trained to predict the errors from the previous tree. And at the end, predictions from each tree are aggregated by weighting. It further increase randomization by bootstrap sampling. Each tree is trained to find the best split points and features. But the algorithm may lead to trees using the same split points and same features. Stochastic Gradient Boosting allows for bootstrap subsample of data and bootstrap features for each tree. This results in further ensemble diversity. Advantage of Stochastic Gradient Boosting Classifier is that it gives high performance.Disadvantages include longer training and prediction times because trees are trained sequentially and lack of interpretebility as in an other ensemble methods. It is a good fit for finding donor project since we care mostly about performance not interpretation.Gradient Boosting is used in anomaly detection for instance credit card fraud detection and cyber security. (link)  Implementation: Creating a Training and Predicting Pipeline<jupyter_code>from sklearn.metrics import accuracy_score, fbeta_score
from time import time

def train_predict(clf, sample_size, X_train, y_train, X_test, y_test):
    
    beta = 0.5
    results = {}
    time0 = time()
    clf.fit(X_train[:sample_size], y_train[:sample_size])
    time1 = time()
    
    results['train_time'] = time1 - time0
    
    time0 = time()
    pred = clf.predict(X_test)
    time1 = time()
    
    results['prediction_time'] = time1 - time0
    
    results['train_accuracy'] = accuracy_score(y_train[:300], clf.predict(X_train[:300]))
    results['train_fbeta_score'] = fbeta_score(y_train[:300], clf.predict(X_train[:300]), beta)
    
    results['test_accuracy'] = accuracy_score(y_test, pred)
    results['test_fbeta_score'] = fbeta_score(y_test, pred, beta)
    
    return results
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier

clf_A = LogisticRegression(random_state = 42)
clf_B = RandomForestClassifier(random_state = 42)
clf_C = GradientBoostingClassifier(random_state = 42)


sample_100 = len(X_train)
sample_10 = int(round(.10 * len(X_train)))
sample_1 = int(round(0.01 * len(X_train)))

results = {}

for clf in [clf_A, clf_B, clf_C]:
    
    clf_name = clf.__class__.__name__
    
    results[clf_name] = {}
    for i, sample_size in enumerate([sample_1, sample_10, sample_100]):
        
        results[clf_name][i] = train_predict(clf, sample_size, X_train, y_train, X_test, y_test)
def evaluate(results, accuracy, f1):
    """
    Visualization code to display results of various learners.
    
    inputs:
      - learners: a list of supervised learners
      - stats: a list of dictionaries of the statistic results from 'train_predict()'
      - accuracy: The score for the naive predictor
      - f1: The score for the naive predictor
    """
  
    # Create figure
    fig, ax = plt.subplots(2, 3, figsize = (11,7))

    # Constants
    bar_width = 0.3
    colors = ['red','blue','green']
    
    # Super loop to plot four panels of data
    for k, learner in enumerate(results.keys()):
        for j, metric in enumerate(['train_time', 'train_accuracy', 'train_fbeta_score', 'prediction_time', 'test_accuracy', 'test_fbeta_score']):
            for i in np.arange(3):
                
                # Creative plot code
                ax[j//3, j%3].bar(i+k*bar_width, results[learner][i][metric], width = bar_width, color = colors[k])
                ax[j//3, j%3].set_xticks([0.45, 1.45, 2.45])
                ax[j//3, j%3].set_xticklabels(["1%", "10%", "100%"])
                ax[j//3, j%3].set_xlabel("Training Set Size")
                ax[j//3, j%3].set_xlim((-0.1, 3.0))
    
    # Add unique y-labels
    ax[0, 0].set_ylabel("Time (in seconds)")
    ax[0, 1].set_ylabel("Accuracy Score")
    ax[0, 2].set_ylabel("F-score")
    ax[1, 0].set_ylabel("Time (in seconds)")
    ax[1, 1].set_ylabel("Accuracy Score")
    ax[1, 2].set_ylabel("F-score")
    
    # Add titles
    ax[0, 0].set_title("Model Training")
    ax[0, 1].set_title("Accuracy Score on Training Subset")
    ax[0, 2].set_title("F-score on Training Subset")
    ax[1, 0].set_title("Model Predicting")
    ax[1, 1].set_title("Accuracy Score on Testing Set")
    ax[1, 2].set_title("F-score on Testing Set")
    
    # Add horizontal lines for naive predictors
    ax[0, 1].axhline(y = accuracy, xmin = -0.1, xmax = 3.0, linewidth = 1, color = 'k', linestyle = 'dashed')
    ax[1, 1].axhline(y = accuracy, xmin = -0.1, xmax = 3.0, linewidth = 1, color = 'k', linestyle = 'dashed')
    ax[0, 2].axhline(y = f1, xmin = -0.1, xmax = 3.0, linewidth = 1, color = 'k', linestyle = 'dashed')
    ax[1, 2].axhline(y = f1, xmin = -0.1, xmax = 3.0, linewidth = 1, color = 'k', linestyle = 'dashed')
    
    # Set y-limits for score panels
    ax[0, 1].set_ylim((0, 1))
    ax[0, 2].set_ylim((0, 1))
    ax[1, 1].set_ylim((0, 1))
    ax[1, 2].set_ylim((0, 1))
  
    # Aesthetics
    plt.suptitle("Performance Metrics for Three Supervised Learning Models", fontsize = 16, y = 1.10)
    plt.tight_layout()
    plt.show()<jupyter_output><empty_output><jupyter_text>Choosing the Best Model Among the three models, Gradient Boosting algorithm has the highest test accuracy. GB identifies 86.4% of people if they make more than 50K or not correctly in the test data whereas it is 84.5% and 84.1% for Logistic Regression and Random Forest algorithms, respectively. Accuracy is not alone enough to evaluate performance of the models. In this problem, we want the people we predicted as making higher thn 50K to actually earn more than 50K since we will send a mail and pay a cost for that. So we would not want to send mail to people who make less than 50K so have a lower prbability of donation. At the same time, it is important to be able to identify a meaningful chunk of all people who make more than 50K as making more than 50K. If the people we labeled as 1 are in fact make more than 50K but if it is only a small portion of all people making more than 50K, then it means even though we minimize our cost by identifying people correctly, we reach out only a small percentage of all people who can make donations. So f score with more emphasis on precision is important in this problem. F-score for the test data is highest for Gradient Boosting. It is 75.7% when all data is used in training. The score is 70.9% and 69.4% for Logistic Regression and Random Forest respectively.Gradient Boosting takes much more time in training and predicting compared to the other two models. It takes around 12.4 seconds in GB whereas it takes around 0.46 and 0.74 seconds in LG and RF. But it is OK for 5% increase in f-score. All in all, high performance and suitability of the model to the data, out of three models, Gradient Boosting is the best model for this problem.<jupyter_code># red: Logistic Regresion
# blue: Random Forest
# green: Gradient Boosting
evaluate(results, accuracy, fscore)<jupyter_output><empty_output><jupyter_text>Describing the Model in Layman's Terms Gradient Boosting Classifier (GBC) is one of the boosting models. Boosting models basically use a number of weak learners that is a learner does a little better than random guessing and at the end do weighted averaging to get the final result. In GBC, there are a number of trees as weak learners. These learners do see and learn sequentially, not at the same time. First tree is trained on training data. Second tree uses the residual errors of first tree as labels in training. In other words, second tree tries to predict residual errors of the first tree. In a similar manner, third tree tries to predict the residual errors of the second tree. If we manage to predict the residual errors, it at the end gives us true labels. So in this model, each tree tries to correct its predecessor's error. The model does sequential correction of predecessor's errors.We can tweak the sample size in training and the number of features each tree can use at split points to introduce further randomization to overcome overfitting. In this case, each tree is trained in different bootstrap samples and use different bootstrap subset of features. These modifications decrease the correlation between trees and add further variance to the ensemble of trees. These steps help the model geneeralize well for the unseen data hence increase the performance.  Model Tuning<jupyter_code># Grid Search CV
# I picked Gradient Boosting Algorithm

from sklearn.metrics import make_scorer
from sklearn.model_selection import GridSearchCV

params_grid = {'n_estimators': [100, 200, 300],
              'subsample' : [0.8, 1.0],
              'max_depth': [1, 3, 4]}

scorer = make_scorer(fbeta_score, beta = 0.5)

clf = GradientBoostingClassifier(random_state = 42)

grid_obj = GridSearchCV(clf,
                        param_grid = params_grid, 
                        scoring = scorer, 
                        cv = 5)

grid_obj.fit(X_train, y_train)

best_clf = grid_obj.best_estimator_

# the results of unoptimized classifier
clf.fit(X_train, y_train)
predictions = clf.predict(X_test)
## results of optimized classifier
best_predictions = best_clf.predict(X_test)


## Before and after scores
print('Unoptimized Model: \n --------------')
print('Accuracy Score on Testing Data: {:.2f}'.format(accuracy_score(y_test, predictions)))
print('F-score on Testing Data: {} '.format(fbeta_score(y_test, predictions, beta = 0.5)))
print('\nOptimized Model: \n --------------')
print('Accuracy Score on Testing Data: {:.2f}'.format(accuracy_score(y_test, best_predictions)))
print('F-score on Testing Data: {}'.format(fbeta_score(y_test, best_predictions, beta = 0.5)))
## Before and after scores
print('Unoptimized Model: \n --------------')
print('Accuracy Score on Testing Data: {:.2f}'.format(accuracy_score(y_test, predictions)))
print('F-score on Testing Data: {} '.format(fbeta_score(y_test, predictions, beta = 0.5)))
print('\nOptimized Model: \n --------------')
print('Accuracy Score on Testing Data: {:.2f}'.format(accuracy_score(y_test, best_predictions)))
print('F-score on Testing Data: {}'.format(fbeta_score(y_test, best_predictions, beta = 0.5)))<jupyter_output>Unoptimized Model: 
 --------------
Accuracy Score on Testing Data: 0.86
F-score on Testing Data: 0.7576574633304572 

Optimized Model: 
 --------------
Accuracy Score on Testing Data: 0.87
F-score on Testing Data: 0.7728384991843392<jupyter_text>Final Model Evaluation |    Metric       |   Unoptimized Model   |   Optimized Model  || :-------------: | :-------------------: | :----------------: || Accuracy Score  |          0.86         |         0.87       || F-score         |          0.758        |         0.773      | The scores of optimized model are better than the unoptimized model. Optimization for hyper-parameters of the model increases accuracy score on testing data from 86% to 87% and increases Fbeta score from 75.8% to 77.3%. Even though there is improvement in the scores, they are not much. The accuracy score and F score for the naive predictor are 25.4% and 30% and it is related to how common positive labels in the total data. The performance of optimized are way better than naive predictor.   Feature Importance  Feature Relevance Observation I expect to see out of 13 variables, age, education_level, hours_per_week, occupation and capital_gain to be 5 most important ones in predicting the label. The ranking I believe to see is age, capital_gain, education_level, hours_per_week and occupation from the most important to the least important. As the person gets older and more senior at his job, she makes more and this increases the probability of making more than 50K. There is a relation between education level and the money the person makes. I put age first since 50K can be achieved in most types of jobs. If it was 100K, I would put education in front of age in ranking. From the pairplots, the distribution of capital_gain for people who make more than 50k is pretty distinguishable from the distribution for the people who make atmost 50k. For this reason, I expect to see this feature as important to differentiate between labels. For hours per week, if the person works full time, then this increases the odds of making more than 50k. The reasoning for occupation are similar.  Extracting Feature Importance<jupyter_code>importances = pd.Series(clf_C.feature_importances_, index = X_train.columns)
imp = importances.sort_values(ascending = False)[:20].sort_values()
plt.figure(figsize = (10, 7))
imp.plot(kind = 'barh', color = base_color);
plt.title('The Most Relevant 20 Features', fontsize = 15);
plt.yticks(fontsize = 12);
plt.xticks(fontsize = 12);
imp.index[-5:][::-1]
base_color = sns.color_palette()[0]
imp.sort_values(ascending = False).cumsum()[:10][::-1].plot(kind = 'barh', color = base_color, figsize = (7, 5));
plt.yticks(fontsize = 12);
plt.title('Cumulative Importance', fontsize = 13);<jupyter_output><empty_output><jupyter_text>The cumulative importance of the first 5 features is a little higher than 60%. The remaining 40% of importance is shared between the remaining 98 features.   Extracting Feature Importance For Stochastic Gradient Boosting, the most important features in predicting if a person makes more than 50K or not are capital_gain_transformed, capital_loss_transformed, age, marital_status_Married-civ-spouse and educaton_num. I expect to see capital_gain, age and education_num to be in the most important first 5 features. But i am surprised to see marital_status_Married-civ-spouse variable. But when I look at the average value of income_bin for people who are in Married-civ-spouse category, it is found out that 45% of these people make more than 50k. This number is only 6.8% for people who are not in Married-civ-spouse category. So it is a feature that can differentiate between people who can make more than 50k and who cannot. It totally makes sense.<jupyter_code>print('{:.1f}% of people in the Married-civ-spouse category make more than 50k'.format(data[data.marital_status == ' Married-civ-spouse'].income_bin.mean()*100))
print('{:.1f}% of people who are not in the Married-civ-spouse category make more than 50k'.format(data[data.marital_status != ' Married-civ-spouse'].income_bin.mean()*100))<jupyter_output>45.4% of people in the Married-civ-spouse category make more than 50k
6.8% of people who are not in the Married-civ-spouse category make more than 50k<jupyter_text>Effects of Feature Selection There are 13 independent variables (features) in the original dataset. We need to convert categorical variables into dummy variables to be able to feed the data into models. In the end, there are 103 fatures and 1 labeled variable. It is seen that capital_gain_transformed, capital_loss_transformed, age, marital_status_Married-civ-spouse and educaton_num are the most relevant variables in identifying labels. We will now look at how the model performs when we use only these most important 5 variables. It will definitely decrease training and predicting times and deteriorates model performance as there is some part of variance explained by omitted features. To be able to see the effects of feature selection, I create X_train_reduced and X_test_reduced including only 5 features and fit the model with new train and test data.<jupyter_code>from sklearn.base import clone
from time import time

indices = imp.sort_values(ascending = False).index
X_train_reduced = X_train[indices[:5]]
X_test_reduced = X_test[indices[:5]]

t1 = time()
clf = (clone(best_clf)).fit(X_train_reduced, y_train)
t2 = time()
reduced_predictions = clf.predict(X_test_reduced)
t3 = time()

t4 = time()
clf = (clone(best_clf)).fit(X_train, y_train)
t5 = time()
best_predictions = clf.predict(X_test)
t6 = time()

## scores on full data and reduced data
print('Final Model trained on full data: \n---------------------------------------------')
print('Accuracy Score on Testing Data: {:.2f}'.format(accuracy_score(y_test, best_predictions)))
print('F-score on Testing Data: {}'.format(fbeta_score(y_test, best_predictions, beta = 0.5)))
print('Time to train the model is {} seconds'.format(t5 - t4))
print('Prediction time is {} seconds'.format(t6 - t5))
print('\nFinal Model trained on reduced data: \n----------------------------------------')
print('Accuracy Score on Testing Data: {:.2f}'.format(accuracy_score(y_test, reduced_predictions)))
print('F-score on Testing Data: {}'.format(fbeta_score(y_test, reduced_predictions, beta = 0.5)))
print('Time to train the model is {} seconds'.format(t2 - t1))
print('Prediction time is {} seconds'.format(t3 - t2))<jupyter_output>Final Model trained on full data: 
---------------------------------------------
Accuracy Score on Testing Data: 0.87
F-score on Testing Data: 0.7728384991843392
Time to train the model is 37.85445737838745 seconds
Prediction time is 0.09508204460144043 seconds

Final Model trained on reduced data: 
----------------------------------------
Accuracy Score on Testing Data: 0.86
F-score on Testing Data: 0.7451923076923076
Time to train the model is 4.336373329162598 seconds
Prediction time is 0.04402971267700195 seconds
{'fraction_non_alphanumeric': 0.09283340136451106, 'fraction_numerical': 0.028048282698699634, 'mean_word_length': 4.34418822062948, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '330605', 'license': 'MIT', 'n_tokens_mistral': 10694, 'n_tokens_neox': 9670, 'n_words': 4221, 'path': 'Finding_Donors.ipynb', 'repo_name': 'ilknurCakir/FindingDonorswithCharityML', 'size': '34298'}