<jupyter_start><jupyter_text>US Adult Census data relating income to social factors such as Age, Education, race etc. Prepared by: <NAME> 27-02-2019The Us Adult income dataset was extracted by <NAME> from the 1994 US Census Database. The data set consists of anonymous information such as occupation, age, native country, race, capital gain, capital loss, education, work class and more. Each row is labelled as either having a salary greater than ">50K" or "<=50K".The goal here is to train a binary classifier on the training dataset to predict the column income_bracket which has two possible values ">50K" and "<=50K" and evaluate the accuracy of the classifier with the test dataset.Note that the dataset is made up of **categorical and continuous features.** It also contains missing values The **categorical columns are: workclass, education, marital_status, occupation, relationship, race, gender, native_country**The continuous columns are: age, education_num, capital_gain, capital_loss, hours_per_weekThis Dataset was obtained from the UCI repository, it can be found on [link](http://mlr.cs.umass.edu/ml/machine-learning-databases/adult/)USAGE This dataset is well suited to developing and testing wide linear classifiers, deep neutral network classifiers and a combination of both. For more info on Combined Deep and Wide Model classifiers, refer to the [Research Paper by Google](https://arxiv.org/abs/1606.07792)Refer to this kernel for sample usage : https://www.kaggle.com/johnolafenwa/wage-prediction Loading Packages<jupyter_code># access kaggle datasets
!pip install kaggle
# for data manipulation
import pandas as pd 
import numpy as np 

from sklearn.metrics import accuracy_score
import re<jupyter_output><empty_output><jupyter_text>Import Dataset<jupyter_code>from google.colab import files
uploaded = files.upload()
!ls
!unzip us-census-data.zip
train = pd.read_csv('adult-training.csv', names = columns)
test = pd.read_csv('adult-test.csv', names = columns)

columns = ['Age','Workclass','fnlgwt','Education','Education Num','Marital Status',
           'Occupation','Relationship','Race','Sex','Capital Gain','Capital Loss',
           'Hours/Week','Country','Above/Below 50K']<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code># to see features
train.head()
# defining function for estimating missing values in each columns
def missing_value(df):
    miss = []
    col_list = df.columns
    for i in col_list:
        missing = df[i].isnull().sum()
        miss.append(missing)
        list_of_missing = pd.DataFrame(list(zip(col_list,miss)))
    return list_of_missing
missing_value(test)
missing_value(train)
# categorical feature
train['Relationship'].value_counts()
# categorical feature
test['Occupation'].value_counts()<jupyter_output><empty_output><jupyter_text>Data Cleaning<jupyter_code>train.head()
# deleted the firt rows of the both dataframes since these are errorneous
train.drop(train.index[0]).head()
test.drop(test.index[0]).head()

print(train.shape)
print(test.shape)
# a swiss knife tool for separating string columns to separate list and numerical to separate list
all_data = [train, test]
str_list = []

for data in all_data:
    for colname, colvalue in data.iteritems(): 
        if type(colvalue[1]) == str:
            str_list.append(colname) 
num_list = data.columns.difference(str_list)
# no null values its good
print(test.isnull().sum())<jupyter_output>Age                0
Workclass          1
fnlgwt             1
Education          1
Education Num      1
Marital Status     1
Occupation         1
Relationship       1
Race               1
Sex                1
Capital Gain       1
Capital Loss       1
Hours/Week         1
Country            1
Above/Below 50K    1
dtype: int64<jupyter_text>Looking at the unique values for the categorical columns, it seems there are is special charatcers ' ?'inserted in the dataframe in place of null values we have to remove it and replace it wiyh nan and deleting the rows with Nan values..Picture is now clear. As you can see missing values in the WorkClass and Occupation are quite high. We will delete the missing value.<jupyter_code># the code will replace the special character to nan
for data in all_data:
    for i in data.columns:
        data[i].replace(' ?', np.nan, inplace = True)
    data.dropna(inplace = True)
test.isnull().sum()<jupyter_output><empty_output><jupyter_text>Feature Engineering Creating the target variable<jupyter_code># defining the target variable
for data in all_data:
    data['target'] = data['Above/Below 50K'].apply(lambda x: x.replace('.', ''))
    data['target'] = data['target'].apply(lambda x: x.strip())
    data['target'] = data['target'].apply(lambda x: 1 if x == '>50K' else 0)
    data.drop(['Above/Below 50K'], axis = 1, inplace = True)
train.target.sum()/len(train)<jupyter_output><empty_output><jupyter_text>Education and number of work hour/week looks to be a great variables deciding income. Lets create categories to further enhance its effect. We will create low, medium and high education. I will derive a generic function for creating bins<jupyter_code># data can be train or test
# var name is variable name: should be passed as strings within ('')
# bins is list of numeric values like [0,6,10,11]
# group names is list of groups you want to create in list form
def bin_var(data, var, bins, group_names):
    bin_value = bins
    group = group_names
    data[var + 'Cat'] = pd.cut(train[var], bin_value, labels = group)<jupyter_output><empty_output><jupyter_text>Education can be divided into three groups 0-6 low and 7-11 medium and above that high<jupyter_code>bin_var(train, 'Education Num', [0,6,11,16], ['Low', 'Medium', 'High'])
bin_var(test, 'Education Num', [0,6,11,16], ['Low', 'Medium', 'High'])
pd.crosstab(train['Education NumCat'], train['target'])<jupyter_output><empty_output><jupyter_text>The same way we can bin the Hours/Week variable. Initial exploation suggest that 40 hours is the most fequent value which make sense that it is 8hr/day. Hence we will bin this variable around this value.<jupyter_code>bin_var(train, 'Hours/Week', [0,35,40,60,100], ['Low', 'Medium', 'High','VeryHigh'])
bin_var(test, 'Hours/Week', [0,35,40,60,100], ['Low', 'Medium', 'High','VeryHigh'])
pd.crosstab(train['Hours/WeekCat'], train['target'], margins = True)<jupyter_output><empty_output><jupyter_text>Classifying the occupation into Highly Skilled and low Skilled<jupyter_code>occu = pd.crosstab(train['Occupation'], train['target'], margins = True).reset_index()
def occup(x):
    if re.search('managerial', x):
        return 'Highskill'
    elif re.search('specialty',x):
        return 'Highskill'
    else:
        return 'Lowskill'
train['Occupa_cat'] = train.Occupation.apply(lambda x: x.strip()).apply(lambda x: occup(x))
test['Occupa_cat'] = test.Occupation.apply(lambda x: x.strip()).apply(lambda x: occup(x))
train['Occupa_cat'].value_counts()<jupyter_output><empty_output><jupyter_text>Same way we can bin the Age variable. The minimum age in train is 17 and max is 90. We can categorize them as young, middle_aged and old<jupyter_code>bin_var(test, 'Age', [17,30,55,100], ['Young', 'Middle_aged', 'Old'])
bin_var(train, 'Age', [17,30,55,100], ['Young', 'Middle_aged', 'Old'])<jupyter_output><empty_output><jupyter_text>Marital status can also be binned<jupyter_code>train['Marital Status_cat']=train['Marital Status'].apply(lambda x: 'married' if x.startswith('Married',1) else 'Single')
test['Marital Status_cat']=test['Marital Status'].apply(lambda x: 'married' if x.startswith('Married',1) else 'Single')<jupyter_output><empty_output><jupyter_text>Race has been binned into White and others<jupyter_code>pd.crosstab(train['Race'], train['target'], margins = True)
train['Race_cat'] = train['Race'].apply(lambda x: x.strip())
train['Race_cat'] = train['Race_cat'].apply(lambda x: 'White' if x == 'White' else 'Other')
test['Race_cat'] = test['Race'].apply(lambda x: x.strip())
test['Race_cat'] = test['Race_cat'].apply(lambda x: 'White' if x == 'White' else 'Other')<jupyter_output><empty_output><jupyter_text>Work Class is divided into three categories Private, Selfemployed, gov and others<jupyter_code>train['Workclass'].value_counts()
def workclas(x):
    if re.search('Private', x):
        return 'Private'
    elif re.search('Self', x):
        return 'selfempl'
    elif re.search('gov', x):
        return 'gov'
    else:
        return 'others'
train['WorfClass_cat'] = train.Workclass.apply(lambda x: x.strip()).apply(lambda x: workclas(x))
test['WorfClass_cat'] = test.Workclass.apply(lambda x: x.strip()).apply(lambda x: workclas(x))
train['WorfClass_cat'].value_counts()<jupyter_output><empty_output><jupyter_text>* Assigning the targte to Y variable<jupyter_code># assigning the target to Y variable
Y_tr = train['target']
Y_te = test['target']
# since target is already assigned I Will drop the target from the train and test along with other unnecessary variables
train.drop(['Education','Occupation','Race','Education Num','Age', 'Hours/Week', 'Marital Status','target','fnlgwt','Workclass', 'Capital Gain','Capital Loss', 'Country'], axis = 1, inplace = True)
test.drop(['Education','Occupation','Race','Education Num','Age', 'Hours/Week', 'Marital Status','Workclass','target','fnlgwt', 'Capital Gain','Capital Loss', 'Country'], axis = 1, inplace = True)<jupyter_output><empty_output><jupyter_text>I will now create dummies for the categorical variables<jupyter_code>str_list = ['WorfClass_cat','Education NumCat', 'AgeCat', 'Race_cat',
            'Hours/WeekCat',
            'Marital Status_cat',
            'Occupa_cat',
            'Relationship',
            
            'Sex']

train_set = pd.get_dummies(train, columns=str_list)
test_set = pd.get_dummies(test, columns=str_list)<jupyter_output><empty_output><jupyter_text>Feature Selection Using Variance Threshold Variance Threshold is a univariate  approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.As an example, suppose that we have a dataset with boolean features, and we want to remove all features that are either one or zero (on or off) in more than 80% of the samples. Boolean features are Bernoulli random variables, and the variance of such variables is given byThe below approach removes variable which have more than 80% values are either 0 or 1<jupyter_code>train_set.columns
from sklearn.feature_selection import VarianceThreshold
def variance_threshold_select(df, thresh = 0.0, na_replacement = -999):
    
    # Make a deep copy of the dataframe
    df1 = df.copy(deep = True)
    
    # passing Threshold
    selector = VarianceThreshold(thresh)
    
    # fill NA values as VarianceThreshold cannot deal with those
    selector.fit(df1.fillna(na_replacement))
    
    # get new df with columns deleted that have NA values
    df2 = df.loc[:,selector.get_support(indices = False)] 
    return df2
df2 = variance_threshold_select(train_set, thresh = .8* (1 - .8))
print(df2.columns)<jupyter_output>Index(['WorfClass_cat_Private', 'Education NumCat_Medium',
       'Education NumCat_High', 'AgeCat_Young', 'AgeCat_Middle_aged',
       'Hours/WeekCat_Medium', 'Hours/WeekCat_High',
       'Marital Status_cat_Single', 'Marital Status_cat_married',
       'Occupa_cat_Highskill', 'Occupa_cat_Lowskill', 'Relationship_ Husband',
       'Relationship_ Not-in-family', 'Sex_ Female', 'Sex_ Male'],
      dtype='object')<jupyter_text>As you can see below the number of columns have been reduced to 15 because of the the variance threshold. The removed columns have the same value in 80% of the observations<jupyter_code># creates list of columns
col_tr = df2.columns

# creates list of columns for test
col_te = test_set.columns 

# creates array of values of features
X_tr=df2.values 

# subseting the test dataset to get the same variable as train and
X_te = test_set[col_tr].values
len(col_tr)<jupyter_output><empty_output><jupyter_text>Modelling Process<jupyter_code>import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np 
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import confusion_matrix, classification_report, roc_curve, roc_auc_score
import itertools

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
#    else:
#        print('Confusion matrix, without normalization')

#    print(cm)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.show()

def show_data(cm, print_res = 0):
    tp = cm[1,1]
    fn = cm[1,0]
    fp = cm[0,1]
    tn = cm[0,0]
    if print_res == 1:
        print('Precision =     {:.3f}'.format(tp/(tp+fp)))
        print('Recall (TPR) =  {:.3f}'.format(tp/(tp+fn)))
        print('Fallout (FPR) = {:.3e}'.format(fp/(fp+tn)))
    return tp/(tp+fp), tp/(tp+fn), fp/(fp+tn)
lrn = LogisticRegression(penalty = 'l1', C = .001, class_weight = 'balanced')

lrn.fit(X_tr, Y_tr)
y_pred = lrn.predict(X_te)
cm = confusion_matrix(Y_te, y_pred)
if lrn.classes_[0] == 1:
    cm = np.array([[cm[1,1], cm[1,0]], [cm[0,1], cm[0,0]]])

plot_confusion_matrix(cm, ['0', '1'], )
pr, tpr, fpr = show_data(cm, print_res = 1)
from sklearn.metrics import precision_score, \
    recall_score, confusion_matrix, classification_report, \
    accuracy_score, f1_score
    
print('Accuracy:', accuracy_score(Y_te, y_pred))
print('F1 score:', f1_score(Y_te,y_pred))<jupyter_output>Accuracy: 0.701261620185923
F1 score: 0.5866029587429936<jupyter_text>I achieved the accuracy of 72% with recall of 74% on test which is quite good.  Understanding Important features for High and Low Paying Jobs<jupyter_code># understanding the coefficients
coff = pd.DataFrame(lrn.coef_).T 
col = pd.DataFrame(col_tr).T 

print(coff)
print(col)<jupyter_output>0
0   0.000000
1   0.000000
2   0.560629
3  -0.275564
4   0.001234
5   0.000000
6   0.078825
7  -0.934614
8   0.582120
9   0.069669
10 -0.295364
11  0.000000
12  0.000000
13  0.000000
14  0.000000
                      0                        1                      2   \
0  WorfClass_cat_Private  Education NumCat_Medium  Education NumCat_High   

             3                   4                     5                   6   \
0  AgeCat_Young  AgeCat_Middle_aged  Hours/WeekCat_Medium  Hours/WeekCat_High   

                          7                           8   \
0  Marital Status_cat_Single  Marital Status_cat_married   

                     9                    10                     11  \
0  Occupa_cat_Highskill  Occupa_cat_Lowskill  Relationship_ Husband   

                            12           13         14  
0  Relationship_ Not-in-family  Sex_ Female  Sex_ Male<jupyter_text>As you can see above positive coefficients are for high paying and neagtive for low paying.As you can see High number of years of education, high workhour/week are importand for getting high salary and vice versa<jupyter_code>from sklearn.feature_selection import RFE, f_regression

# stop the search when only the last feature is left
rfe = RFE(lrn, n_features_to_select=10, verbose = 3)
rfe.fit(X_tr,Y_tr)
list(zip(map(lambda x: round(x, 4), rfe.ranking_), col_tr))<jupyter_output>Fitting estimator with 15 features.
Fitting estimator with 14 features.
Fitting estimator with 13 features.
{'fraction_non_alphanumeric': 0.11183848271642705, 'fraction_numerical': 0.024349954114408077, 'mean_word_length': 4.169512966476914, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 86, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '509822', 'license': 'MIT', 'n_tokens_mistral': 5488, 'n_tokens_neox': 5115, 'n_words': 1565, 'path': 'Data Science Lit/US_Adult_Income_(Logistic_Regression).ipynb', 'repo_name': '2series/DataScience-Courses', 'size': '16358'}